Sparse tree-based Initialization for Neural Networks
Authors: Patrick Lutz
Ludovic Arnould
Claire Boyer
Erwan Scornet
Venue: ICLR 2023 poster (2023)

Abstract: Dedicated neural network (NN) architectures have been designed to handle specific data types (such as CNN for images or RNN for text), which ranks them among state-of-the-art methods for dealing with these data. Unfortunately, no architecture has been found for dealing with tabular data yet, for which tree ensemble methods (tree boosting, random forests) usually show the best predictive performances. In this work, we propose a new sparse initialization technique for (potentially deep) multilayer perceptrons (MLP):  we first train a tree-based procedure to detect feature interactions and use the resulting information to initialize the network, which is subsequently trained via standard gradient descent (GD) strategies. Numerical experiments on several tabular data sets showthe benefits of this new, simple and easy-to-use method, both in terms of generalization capacity and computation time, compared to default MLP initialization and even to existing complex deep learning solutions. In fact, this wise MLP initialization raises the performances of the resulting NN methods to that of gradient boosting on tabular data. Besides, such initializations are able to preserve the sparsity of weights introduced in the first layers of the network throughout the training, which emphasizes that the first layers act as a sparse feature extractor (like convolutional layers in CNN).

Published as a conference paper at ICLR 2023
SPARSE TREE-BASED INITIALIZATION FOR NEURAL NETWORKS
Patrick Lutz
Ludovic Arnould LLPM, Sorbonne University 3CMAP, Ecole Polytechnique
Claire Boyer²
Erwan Scornet
|Botonn University
ABSTRACT
Dedicated neural network (NN) architectures have been designed to han- dle specific data types (such as CNN for images or RNN for text), which ranks them among state-of-the-art methods for dealing with these data. Unfortunately, no architecture has been found for dealing with tabular data yet, for which tree ensemble methods (tree boosting, random forests) usu- ally show the best predictive performances. In this work, propose a new sparse initialization technique for (potentially deep) multilayer per- ceptrons (MLP): we first train tree-based procedure detect feature in- teractions and use the resulting information to initialize the network, which is subsequently trained via standard gradient descent (GD) strategies. Nu- merical experiments on several tabular data sets show the benefits of this new, simple and easy-to-use method, both in terms of generalization ca- pacity and computation time, compared to default MLP initialization and even to existing complex deep learning solutions. initialization raises the performances of the resulting NN methods to that of gradient boosting on tabular data. Besides, such initializations are able to preserve the sparsity of weights introduced in the first layers of the net- work throughout the training, which emphasizes that the first layers act as sparse feature extractor (like convolutional layers in CNN).
fact, this wise MLP
INTRODUCTION
Neural networks are now widely used in many domains of machine learning, in particular when dealing with very structured data. They indeed provide state-of-thh-aa performances for applications with images or text. However, neural networks still perform poorly on tabular inputs, for which tree ensemble methods remain the gold standards (Grinsztajn let al. 2022). The goal of this paper is to improve the performances of the former by using the strengths of the latter.
Tree ensemble methods Tree-based methods are widely used in the ML community, es- pecially for processing tabular data. Two main approaches exist depending whether the tree building process is parallel (e.g. Random Forest, RF, see Breiman 2001b) or sequential (e.g. Gradient Boosting Decision Trees, GBDT, see Friedman 2001P. In these tree ensem- ble procedures, the final prediction relies on averaging predictions of randomized decision trees, coding for particular partitions of the input space. The two most successful and most widely used implementations of these methods are XGBoost and LightGBM (see Chen & Ke al. 2017) which both rely on the sequential GBDT approach.
Guestrin 2016
Neural networks Neural Networks (NN) are efficient methods to unveil the patterns of spatial or temporal data, such images (Krizhevsky et al. 2012) texts (Liu et al.) 2016). Their performance results notably from the fact that several architectures directly encode relevant structures in the input: convolutional neural networks (CNN, LeCun al.) 1995) use convolutions to detect spatially-invariant patterns in images, and recurrent neural net- works (RNN, Rumelhart al. 1985) use a hidden temporal state to leverage the natural or- der of a text. However, a dedicated natural architecture has yet to be introduced to deal with
"corresponding author: plutz@bu.edu

---

Published as a conference paper at ICLR 2023
tabular data. Indeed, designing such architecture would require to detect and leverage structuee of the relations between variables, which is much easier for images or text (spa- tial or temporal correlation) than for tabular data (unconstrained covariance structure).
NN initialization and training In the absence of a suitable architecture for handling tab- ular data, the Multi-Layer Perceptron (MLP) architecture (Rumelhart et al. 1986) remains the obvious choice due to its generalist nature. Apart from the large number of parameters, one difficulty of MLP training arises from the non-convexity of the loss function (see, e.g., Sun, 2020). In such situations, the initialization of the network parameters (weights and biases) are of the utmost importance, since it can influence both the optimization stability and the quality of the minimum found. Typically, such initializations are drawn according to independent uniform distributions with variance decreasing w.r.t. the size of the layer (He et al. 2015). Therefore, one may wonder how to capitalize on methods that are inher- ently capable of recognizing patterns in tabular data (e.g, tree-based methods) to propose new NN architecture suitable for tabular data and an initialization procedure that leads to faster convergence and better generalization performance.
1.1 RELATED WORKS
How MLP can be used to handle tabular data remains unclear, especially since a corre- sponding prior in the MLP architecture adapted to the correlations of the input is not obvi- ous, to say the least. Indeed, none of the existing NN architectures can consistently match the performance of state-of-theeart tree-based predictors on tabular data (Shwartz-Ziv &
Armon 2022 Gorishniy et al. 2021 and in particular Table 2 in Borisov et al. 2021
Self-attention architectures Specific NN architectures have been proposed to deal with tabular data. For example, TabNet (Arik & Pfister 2021) uses sequential self-attention structure to detect relevant features and then applies several networks for prediction. SAINT (Somepalli et al.. 2021), on the other hand, uses two-dimensional attention struc- ture (on both features and samples) organized in several layers to extract relevant informa- tion which is then fed to classical MLP. These methods typically require large amount of data, since the self-attention layers and the output network involve numerous MLP.
Trees and neural networks Several solutions have been proposed to leverage the corre- spondence between tree-based methods and NN, in order to develop more efficient models for processing tabular data. For example, TabNN (Ke et al. 2018) first trains GBDT on the available data, then extracts group of features per individual free, compresses the result- ing groups, and uses a tailored Recursive Encoder based on the structure of these groups (with an initialization based on the tree leaves). Therefore, TabNN employs pre-trained tree-based methods to design more efficient NN. Conversely, Sethi (1990) Brent (1991), and later Welbl (2014), Richmond et al. (2015) and Biau et al. (2019) propose to translate deci- sion trees into very specific MLP (made of layers) and use GD training to improve upon the original tree-based method. Such procedures can be seen as way to relax and gener- alize the partition geometry produced by trees and their aggregation. To our knowledge, such translations have not been used to boost the training of general NN architectures.
1.2 CONTRIBUTIONS
In this work, we propose a new method to initialize a potentially deep MLP for learning tasks with tabular data. Our method consists in first training tree-based predictor (RF, GBDT or Deep Forest, see Section 2.1) and then using its translation into an MLP as initial- ization for the first two layers, the deeper ones being randomly initialized. With subsequent standard GD training, this procedure is shown to outperform the widely used uniform ini- tialization of MLP (default initialization in Pytorch
(aatzc/ Paszke et al. 2019) follows.
1. Improved performances. For tabular data, the predictive performances of the MLP after training improved compared to MLP that use random initialization. Our procedure also outperforms more complex deep learning procedures based on self- attention and is on par with classical tree-based methods (such as XGBoost).
2

---

Published as a conference paper at ICLR 2023
2. Faster optimization. The optimization following tree-based initialization is boosted in the sense that it enjoys faster convergence towards a (better) empirical tree-based initialization results in faster training of the MLP.
minimum:
Initializing the first few layers of the MLP with the translation of the tree-based method and initializing randomly the deeper layers the most successful initialization scheme that we experimented. This supports the idea that in our method, the (first) tree-based initialized layers act as relevant feature extractors that allow the MLP to detect correlations in the puts. In this context, our approach is dedicated on improving the performance of standard MLP models; therefore conceptually different from pre-existing procedures also rely- ing on the translation of tree-based models into NN: (Biau et al., 2019) aim at fine-tuning tree-based methods using a very specific neural network framework (made of only lay- ers). We, on the other hand, use tree-based methods to carefully initialize certain layers of a generic MLP, which is then substantially trained using standard GD strategies.
Outline In Section methods can be translated into MLP. The core of our analysis is contained in Section 3 where we describe in detail the MLP initialization process and provide extensive numerica. evaluations showing the benefits of this method.
introduce the predictors in play and describe how tree-based
EQUIVALENCE BETWEEN TREES AND MLP
Consider the classical setting of supervised learning in which we are given a set of in- put/output samples {X,,, "") drawn i.idd from some unknown joint distribution. Our goal is to construct (MLP) function to predict the output from the input. To do so, we leverage the translation of tree-based methods into MLP.
2.1 PRESENTATION OF THE PREDICTORS IN PLAY
Tree-based methods We consider three different tree ensemble methods: Random Forests (RF), Gradient Boosting Decision Trees (GBDT) and Deep Forests (DF). They all share the same base component: the Decision Tree (DT, for details see Breiman et al., 1984). We call its terminal nodes leaf nodes, which correspond to the cells of the final tree partition. RF (Breiman 2001a) is predictor consisting of collection of independently trained and ran- domized trees. Its final prediction is made by averaging the predictions of all its DT in regression by majority vote in classification. GBDT (Friedman) 2001) aims at minimiz- ing prediction loss function by successively aggregating DI that approximate the oppo- site gradient of that loss function (see Chen Guestrin 2016 for details on XGBoost). DF (Zhou & Feng (219) is hybrid learning procedure in which random forests are used as elementary components (neurons) of neural-networkklikk architecture (see Figure 5 and Appendix A for details).
Multilayer Perceptron (MLP) The multilayer perceptron is a predictor consisting of a composition of multiple affine functions, with (potentially different) nonlinear activation functions between them. Standard activation functions include, for instance, the rectified linear unit the hyperbolic tangent. Deep MLP are much richer class of predictors than tree-based methods which build simple partitions of the space and output piecewise con- stant predictions. Therefore, any of the tree-based models presented above can be approx- imated and in fact exactly rewritten as an MLP as follows.
2.2 AN EXACT TRANSLATION TREE-BASED METHODS INTO MLP
From decision tree to 3-layer MLP Recall that a decision tree codes for a partition of the input space in as many parts as there are leaf nodes in the tree. Given input x, we can identify the leaf where x falls by examining for each hyperplane of the partition whether æ falls on the right left side of the hyperplane. The prediction is then made by averaging the outputs of all the training samples falling into the leaf z. A DT can be thus translated into highly sparse 3-layer MI.P:
3

---

Published as a conference paper at ICLR 2023
leper laper
Oetpen loye
Figure 1: (from Biau et al. 2019) Illustaation of decision tree, its induced feature space partition and its corresponding MLP translation on problem with 2 input variables.
1. The first layer contains number of neurons equal the number of hyperplanes in the partition, each neuron encoding by =11 whether æ falls on the left or right side the hyperplane.
2. The second layer contains number of neurons equal to the number of leaves in the DT. Based on the first layer, it identifies in which leaf x falls and outputs a vector with a single 1 at the leaf position and - everywhere else.
3. The last layer contains single output neuron that returns the tree prediction. Its weights encode the average output of all training samples for each leaf of the tree.
This procedure is explained in detail and formally Biau et al. 2019 and in Appendix
From RF/GBDT to 3-layer MLP Although RF and GBDT are constructed in different ways, they both average multiple DT predictions to give the final result. Thus, to trans- late a RF or GBDT into MLP, we simply turn each tree into a 3-layer MLP as described above, and concatenate all the obtained networks to form wider 3-layer MLP. When con- catenating, set all weights between the MLP translations of the different trees to 0, since the trees do not interact with each other in predicting the target value for a new feature vector. The step in which the responses of the different trees are averaged can be combined with the third layer of the individual tree translations, resulting in a final MLP translation with total of three layers.
From Deep Forests to deeper MLP A Deep Forest is a cascade of Random Forests. As such, can be translated into an MLP containing the MLP translations of the different RF in cas- cade, resulting in a deeper and wider MLP (note that the obtained MLP has number of layers that is multiple 3) Furthermore, is concatenated to the output of each intermediate layer. To mimic these skip connections in the MLP,, we add additional neurons to each layer, except for the last three, which en- code an identity mapping. Appendix A gives more insights into DF and their MLP trans- lations. In particular, perfect translation of DF suffers from numerical instabilities due to the replication of catastrophic cancellations (the deeper the DE, the greater their ampli- tude, cf Appendix D). This does not impact the sequel of the study, which relies on MLP approximations introduced in Section 2.2
the Deep Forest architecture, the input vector
2.3 RELAXING TREE-BASED TRANSLATION ALLOW GRADIENT DESCENT TRAINING
As shown in the previous section, one can construct an MLP that exactly reproduces tree-based predictor. However this translation involves (i) piecewise constant activation functions (sign) and (ii) different activation functions in a same layer (sign and identity when translating DF). These constraints can hinder the MLP training, which relies on GD strategies (requiring differentiability) ), as well as efficient implementation tricks, given that automatic differentiation libraries only support one activation function per layer. Therefore, given a pre-trained tree-based predictor (RF, GBDT or DF), we aim at relaxing its transla- tion into a MLP, mimicking its behavior as closely as possible but in a compatible way with standard NN training.
From tree-based methods to differentiable MLP To do so, Welbl 20 Biau et al. (0019) consider the differentiable tanh activation, well suited for approximating both the sign and
4

---

Published as a conference paper at ICLR 2023
identity functions. Indeed, this can be achieved by multiplying or dividing the output of a neuron by a large constant before applying the function tanh and rescaling the result ac- cordingly if necessary, ie. for large enough a, c > 0, sign(z) & tanh(ax) and x R etanh (#).
However, we cannot choose arbitrarily large this would make gradients vanish during the network optimization (the function being flat on most of the space), and hinder train- ing. We therefore introduce 4 hyper-parameters for the MLP encoding of any tree-based method that regulate the degree of approximation for the activation functions after the first, second and third layers of decision tree translation, as well as for the identity mapping, respectively denoted by st rength01, strength1 22, st strength2 and st st.rengthh.d.
Hyperparameter choice The use of the tanh activation function involves extra hyper- parameters. study the influence of each one, by making them vary in some range (keep- ing the others fixed to 10t0, resulting in an almost perfect approximation of the sign and identity functions), see Appendix D.1 for details. Our analysis shows that increasing the hyperparameters beyond some limit value is no longer beneficial (as the activation func- tions are already perfectly approximated) and, across multiple data sets, these limit values are similar. We also exhibit relevant search spaces that will allow us to find optimal HP values for each application.
3 A NEW INITIALIZATION METHOD FOR MLP TRAINING
In this section, we study the impact of tree-based initialization methods for MLP train- ing when dealing with tabular data. The latter empirically proves to be always prefer- able to standard random initialization and makes MLP competitive predictor for tab- ular data. Our code is publically available at SparseTreeBBseecnn.
3-1 OUR PROPOSAL
Random initialization is the most common technique for initializing MLP prior to stochastic gradient training. consists in setting all layer parameters to random values of small mag- nitude centered at O. More precisely, all parameter values of the j-th layer are uniformly drawn in /d,, 1/va,) where d, the layer input dimension; this is the default behaviour of most MLP implementations such as nn. Linear in PyTorch (Paszke et al., 2019).
We introduce new ways of initializing an MLP for learning with tabular data, by leveraging the recasting of tree-based methods in neural network fashion:
RF/GBDT initialization. First, a RF/GBDT is fitted to the training data and trans- formed into a 3-layer neural network, following the procedure described in Sec- tion 2 The first two layers of this network are used to initialize the first two layers of the network
interest. Thus, upon initialization, these first two layers encode the RF/GBDT partition. The parameters of the third and all subsequent layers are randomly initialized as described above. See Figure in Appendix C for an illus- tration.
DF initialization. Similarly as above, Deep Forest (DF) using forest layers is first fitted to the training data. The first layers of the MLP then initialized using the first 3/ layers of the MLP encoding of this pre-trained DF. The parameters of the 36-th and all subsequent layers are randomly initialized as explained above.
These tree-based initialization techniques may seem far-fetched at first glance, but they are actually consistent with recent approaches to adapting Deep Learning models for tabular data. The key to interpreting them is to think of the first (tree-based initialized) layers of the MLP as a feature extractor that produces an abstract representation of the input data (in fact, this is vector encoding the tree-based predictor's space partition in which the observation lies). The subsequent randomly initialized layers, once trained, then perform the prediction task based on this abstract representation.
5

---

Published as a conference paper at ICLR 2023
3.2 EXPERIMENTAL SETUP
Datasets & learning tasks We compare prediction performances total of 10 datasets:
3 regression datasets (Airbnb, Diamonds and Housing), 5 binary classification datasets (Adult, Bank, Blastchar, Heloc, Higgs) and 2 multi-class classification datasets (Covertype and Volkert). We mostly chose data sets that are used for benchmarking in relevent liter- ature: Adult, Heloc, Housing, Higgs and Covertype are used by [Borisov et al. (2021) and Bank, Blastchar and Volkert are used by Somepalli et al. (2021). Moreover, we add Airbnb and Diamonds to balance the different types of prediction tasks. The considered datasets are all medium-sized (10-60k observations) except for Covertype and Higgs (approx. 500k observations). Details about the datasets are given Appendix E.1
Random Forest (RF), Deep Forest (DF, Zhou & Feng 2017) and XGBoost (denoted by GBDT, Chen Guestrin 2016)
Predictors We consider the following tree-based predictors:
The latter usually achieves state-of-the-art performances on tabular data sets (see, e.g.r
Shwartz-Ziv & Armon 2022; Gorishniy et al. 2021 Borisov et al. 2021). We also consider
deep learning approaches: MLP with detault uniform initialization (MLP rand. init.) or
tree-based initialization (resp. MLP RF init., MLP GBDT init. and MLP DF init.); and transformer architecture SAANT[/Sompp ically designed for applications on tabular data and includes self-attention and inter-sample attention layers that extract feature correlations that are then passed on to an MLP. For re- gression and classification tasks, we use the mean-squared error (MSE) and cross-entropy loss for NN training, respectively. We choose SAINT as baseline model it is reported to outperform all other NN predictors on most of our data sets (all except Airbnb and Dia- monds, see Borisov et al. 2021 Somepalli al. 2021)
et al. (2021). This complex architecture is specif-
(a) Housing
(b) Airbnb
Parameter optimization All NN are trained using the Adam optimizer Kingmma & Ba 2014). All hyper-parameters (HI') are deter- mined empirically using the optuna library (Ak-
iba et al. 2019) for Bayesian optimization.
For most HP, we use the default search spaces of /Borisov et al. (2021). For all HP tuning the tree-to-MLP translation, we have identified relevant search spaces (see Appendix D.1). An overview of all search spaces used for each method and the HP selected for experimental protocol P2 can be found in Appendix [E.5 The quantity minimized during HP tuning the model's validation loss, and the smallest validation loss that occurred during training for MLP-based models.
(c) Adult
(d) Bank
(e) Covertype**
(f) Volkert
3-3 A BETTER
MLP INITIALIZATION FOR A BETTER OPTIMIZATION
In this subsection, the optimization of standard MLP is shown to benefit from the proposed ini- tialization technique. Experiments have been conducted on out of the 10 data sets.
Figure 2: Optimization behaviour of ran- domly, RF and GBDT initialized MLP evalu- ated over 5 times repeated (stratified) 5-fold of each data set, according to Protocol P1. The lines and shaded areas report the mean and standard deviation. *evaluation on a single 5- fold cross validation.
Experimental protocol 1 (P1) To obtain com- parable optimization processes, we ensure that all MLP-related hyper-parameters (width, depth, learning rate), are identical for all the MLP regardless of the initialization scheme. These HP are chosen to maximize the pre- dictive performance of the standard randomly initialized MLP. All HP related to the ini-

---

Published as a conference paper at ICLR 2023
tialization technique (HP for the tree-based predictors and their translation) are optimized independently for each tree-based initializations.
Results Figure 2 shows that for most data sets, the use of tree-based initialization meth- ods for MLP training provides a faster convergence towards a better minimum (in terms of generalization) than random initialization. This is all the more remarkable since Proto- col P1 has been calibrated in favor of random initialization. Among tree-based initializers, GBDT initializations outperform or are on par with RF initializations in terms of the opti- mization behavior on all regression and binary classification problems. However, for multi- class classification problems, the advantages of tree-based initialization seem to be limited. This is probably due to the fact that the MLP architecture at play is tailored for random initialization, being thus too restrictive for tree-based initializers. Experiments presented in Appendix E.3 with fixed arbitrary widths corroborate this idea: in this case, the RF ini- tialization is beneficial for the optimization process. For the Adult, Bank, and Volkert data sets, Figure 2 also shows the performance of each method at initialization. None of these procedures leads to a better MLP performance at initialization (due to both the non-exact translation from trees to MLP and the additional randomly initialized layers), but rather help guiding the MLP in its learning process,
3-4 A BETTER MLP INITIALIZATION FOR A BETTER CENERALIZATION
In this subsection, tree-based initialization methods are shown to systematically improve the predictive power of neural networks compared to random initialization. We compare our procedure to the predictors described in Section 3.2 but also to 3 other NN techniques: one close to the default uniform initialization (Xavier init., see Glorot & Bengio 2010), one using random orthonal matrices (LUSV init., see Mishkin & Matas 2015) and the winning ticket lottery strategy (WT prun., see Frankle & Carbin 2018), which a pruning method used during training to end up with sparse NN. The reader may refer to Appendix E.4.1 for more details about these three techniques.
Experimental protocol 2 (P2) Each MLP is trained on 100 epochs, but with HP tuned depending on the initialization technique. For maximum comparability, the optimization budget is strictly the same for all methods (100 "optuna" iterations each, where one optuna iteration consists of one hold-out validation). In particular, when using a tree-based initial- izer, we use 25 HP optimization iterations to find optimal HP for the tree-based predictor, fix these HPs, and then use the remaining 75 iterations to determine optimal HP for the MLP. For all NN approaches, the model with the best performance on the validation set during training is kept (using the classical early-stopping procedure). Performances are measured via the MSE for regression, the AUROC score (AUC) for binary classification and the accu- racy (Acc.) for multi-class classification, averaging 5 runs of 5-fold cross-ver coossvalidation.
Results I shows that RF or GBDT initialization strictly outperform random initial- ization, in terms of final generalization performance, for all data sets except Covertype (for which performances are similar). They also systematically achieve better results than the LUSV and Xavier init, and are better on all but 2 datasets than the WT pruning procedure which is a more refined procedure. Additionally, the MLP using both RE and GBDT initial- ization techniques outperform SAINT on all medium-sized data sets and fall short on large data sets (Higgs and Covertype).
Despite its simplicity the proposed method (based on RF or GBDT) is on par with GBDT on half of the data sets, ranking MLP as relevant predictors for tabular data. Note that the GBDT used for initialization of the MIP way less powerful than the best one found here (see details in Tables 10 and 11). This shows that our procedure produces, with rela- tively low initialization cost, powerful MLP relevant for tabular data. Among the tree-based initializers, RF is on par with or outperforms GBDT initialization on all data sets but Hous- ing. DF initialization, for its part, cannot compete with RF and GBDT initialization, despite showing some improvement over the random one (except for Covertype and Volkert). This underlines that injecting prior information via tree-based methods into the first layers of ML. is, among all the aforementioned methods, the best way to improve its performance.
7

---

Published as a conference paper at ICLR 2023
Data set
Housing
Airbnb
Diamonds
Adult
Bank
Blastchar
Heloc
Higgs
Covertype
Volkert
Model
MSB
MSE
(st0")
(stl")
MSE
1
(in
ADC1
N)
ADC
%)
ADC
(mn%
AUC t (mn))
Acc 1 (mn))
Acc. f
Random Forest
0.26310.009
5,39
9,80
91.6103
92.8.003
84.511.
91.3106
80,4101
83.610.
64.2.10.3
GBDT
0.208
4.71=0.15
0.38
92.7=033
93.3
84.7=10
92.1104
82,8
97.02000
71.3
Deep Forest
0,225x0.008
4.68-0.16
8.23-029
91.8a3
92.9+02
83.7=12
90.3+03
81 2=our
924+01
66.3+04
MLP rand. init.
0.258:0001
5.07=0.16
15.5=125
90.5::0
91 91.0003
81,4212
80.1
83.2:003
96.7200
72.2:10.
MLP Xavier init.
0.26310.012
5.05..0.17
1214_619
90,5100
90.8.005
81.7
79,911.1
82.810.1
96.8100
72.1
MLP LUSV init.
0,295=x.018
4.99
14.1-500
90.5=a5
90.2+005
84.3=12
79,9
81.7+01
96.5.00
20.8+:0.5
MLP WT prun.
0.248=00001
5.26=2.11
9.83=5.07
90.6:204
90.9=a5
64.4=112
89 89.6
82.9+0.1
97,0++0
71.5+004
MLPP RFinit.
0.222:00009
4.66:0016
7.93=022
92.1+003
92.4+04
1.
91.7:00
83.6:001
96.7200
74.1+004
MLP GBDT init.
0.206
4.70
8.15
92.2.103
92.5100
84.6
91.510%6
83.0100
96,2.000
73.5
MLP DF init.
0.234::00016
8.28-0021
91.9=44
92 2:003
8422
914106
83.3
94.5--0.3"
71.++0.
SAINT
0,258=0.011
4.81-0.15
17.7=385
91.6=03
92.2+004
84.0=0.8
90.2
83.7+0.1
96.6=t011
70.1+004
Table 1: Best scores and their standard deviations for Protocol 2. For each data set, predictors per- forming at least well the best over all (resp. best DL) score up to its standard deviation highlighted in bold (resp. underlined). All scores are based on a 5 times repeated (stratified) 5-fold cross validation. For each model, HP have been chosen via the "optuna" library with 100 iterations. See Appendix(E.4. E.4.2 for a comparison with literature results. "score based on a simple 5-fold cross val.
The interested reader may find a comparison of the optimization procedures of all MLP methods and SAINT (Figure 13p and tables summarizing all HP (Tables and 11p in Ap- pendix [E.522 We remark that tree-based initializers generally bring into play wider net- works with similar depths (fixed width of 2048 and adaptive depth between 4 and 10) com- pared to MLP with default initialization. Yet, for most data sets, the overall procedure is computationally more efficient than state-of-the-art deep learning architectures like SAINT, both in terms of number of parameters and training time (see Tables in Appendix E.4
3-5 ANALYZING KEY ELEMENTS OF THE NEW INITIALIZATION METHODS
Influence of the MLP width We mainly use standard spaces from (Borisov et al., 2021) to determine the optimal hyper-parameters for each model. However, the MLP width is an exception to this. The standard search spaces used in the literature usually involve MLP with few hundred neurons per layer (e.g. up to 100 neurons in Borisov et al. 2021 ); yet, in this work, we consider MLP with width up to 2048 neu- rons. Large MLP are actually very beneficial for tree-based initialization methods as they allow the use of more expres- sive tree-based models in the initialization step.
6244
G22
M4P wddts
VILP runcem
ML COST IN.
Figure 3: Influence of width on the generalization perfor- mance for random and GBDT initializations. Mean values over 5 times repeated 5-fold cross-validation on Housing.
Figure 3 compares the performance of an MLP with ran- JDBT initializations and various widths. There is no gain in prediction by using wider (thus more complex) NN, when randomly initialized. This is corroborated by the results of Table for all regression and binary classification data sets, the performance of our (po- tentially much wider) MLP with random initialisation is consistently close to the literature values, and only increases for multi-class classification tasks. However, an MLP initialized with GBDT significantly benefits from enlarging the NN width (justifying a fixed width of 2048 for tree-based initialized MLP). This confirms the idea that tree-based initialization helps revealing relevant features to the MLP, all the more as the width increases, and by doing so, boosts the MLP performance after training.
Performance of the initializer Another interesting step in unraveling the essence of the new initialization method is to understand which characteristics of a tree-based model are relevant to its success as initializer. Undoubtedly, its predictive accuracy plays an impor- tant role, but does this aspect alone suffice to characterize the success of the new initializa- tion method? Figure 14 compares the predictive performance of different RF/GBDT initial- izers and the performance of the respective MLP after training. As the figure illustrates, a better performance of the tree-based predictor used for initialization does not always lead
8

---

Published as a conference paper at ICLR 2023
to a better performance of the MLP after training (see Airbnb and Volkert). This obser- vation suggests that other aspects, such as the expressiveness the feature interactions captured by the initializer, the structure it induces the MLP or the weight distributions of the initializer, must also play significant role in the initialization method's success.
MLP sparsity Finally, we investigate the structure that tree-based initialization induces on the MLP after training. Figure shows the weight distributions of the three first and the last layers before and after MLP training, for random, RF and GBDT initializations on Housing (see Appendix F..7 for more data sets). It indicates that the weight distribution on the first two layers change significantly during training when the MLP is randomly ini- tialized: the weights are uniformly distributed at epoch 0 but appear to be Gaussian after training. When RF or GBDT initializers are used instead, the weights are sparsely distributed at epoch 0 by construction, and their distribution is preserved during training (notice the logarithmic y-axis for these plots in Figure 4). Note that the (uniform) distribution of the weights in other layers also preserved through training (third and last lines of Figure 4). This means that our initialization technique, in com- bination with SGD optimization strategies, introduces
the first two lay-
implicit regularization of the NN optimization: the sparse structure of the initialisation (on first layers) is maintained. This very similar to the CNN architecture (constrained by design), very successful class of NN designed for image processing. Besides, the weight distributions are not squeezed towards zero during learning when sparse initialization is used, preventing poor gener- alization performances according to previous works (Neal 2012: Blundell et al., 2015) Looking at Figure 4 one may get the impres- sion that the weights in the first layers remain unchanged during GD training, and that ul- timately no learning takes place in these lay- ers. However, numerical experiments (see Ap- pendix E.4.3) show that the weights of all layers are modified during learning; the first two lay- actually undergo the greatest changes. As layers
such, training the RF/ RF/GBDT-initialzee via GD strategies essential for the success of the new initialization method.
4 CONCLUSION AND FUTURE WORK
This work builds upon the permeability that ex- ists between tree methods and neural networks, in particular how the former can help the latter during training, with tabular inputs. We pro- posed new methods for smartly initializing the first layers of standard MLP using pre-trained tree-based methods. The sparsity of this initialization is preserved during training, which shows that it encodes relevant correlations between the data features. Among deep learn- ing methods, such initializations of MLP always improve the performance compared to the widely used random initialization, and provide an easy-to-use and more efficient alterna- tive to SAINT (attention-based method) for tabular data. The performance of this wisely- initialized MLP is remarkably approaching that of XGBoost, which so far reigns supreme for learning tasks on tabular data.
Randem init.
RF init.
GBDT init.
Figure 4: Histograms of the first three and last layers' weights before and after the MLP training Housing. Comparison between random, and GBDT initializations.
Limitations & future work While our procedure is quite generic, some restrictions are noticeable. First, our analysis only allows to initialize neural networks with tanh activation functions; removing this limitation by considering ReLU is good avenue for future work. Besides, while quite reasonable, our initialization is more time-consuming than the random (default) one. Moreover, need to further investigate the benefits of our initialization method on very large data sets. Finally, another interesting direction could be using the efficient hyperparameter search in tree-based methods automatically determine good default NN architecture.

---

Published as a conference paper at ICLR 2023
REFERENCES
Takuya Akiba, Shotaro Sano, Toshihiko Yanase, Takeru Ohta, and Masanori Koyama. Op- tuna: A next-generation hyperparameter optimization framework. In Proceedings the 25nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, 2019.
Sercan Ô Arik and Tomas Pfister. Tabnet: Attentive interpretable tabular learning. In Pro- ceedings of the AAAI Conference on Artificial Intelligence, volume 35, pp. 6679-6687, 2021.
Pierre Baldi, Peter Sadowski, and Daniel Whiteson. Searching for exotic particles in high- energy physics with deep learning. Nature communications,, :11)119, 2014.
James Bergstra, Rémi Bardenet, Yoshua Bengio, and Balázs Kégl. Algorithms for hyper- parameter optimization. In J. Shawe-Taylor, R. Zemel, P. Bartlett, F. Pereira, and K.Q. Weinberger (eds.), Advances in Neural Information Processing Systems, volume 24. Cur- ran Associates, Inc., 2011. URL /tbp///8eefff3322fffee wttece112777222222 7bc261 9bc635690 2at0666-0---per....
Gérard Biau, Erwan Scornet, and Johannes Welbl. Neural random forests. Sankhya A, 81 (2):347-386, 2019.
Christopher M. Bishop and foreword from Geoffrey Hinton. Neural Networks for Pat- tern Recognition. Advanced Texts Econometrics. Clarendon Press, Oxford University Press, 1995. ISBN 9780198538646. URL htttss://book.. goroookks.oooloooooooo fr/books ?
TOSOBGAAQRAJ]
Charles Blundell, Julien Cornebise, Koray Kavukcuoglu, and Daan Wierstra. Weight un- certainty in neural network. In International conference machine learning, pp. 1613-1622.
PMLR, 2015.
Vadim Borisov, Tobias Leemann, Kathrin Seßler, Johannes Haug, Martin Pawelczyk, and Gjergji Kasneci. Deep neural networks and tabular data: A survey, 2021. URL https:
01 (1889
Leo Breiman. Random 10.1023 A:100993441199944444 /tttt///////...oo//////022212100
doi:
Leo Breiman. Random forests. Machine learning, 45(1):5-32, 2001b.
Leo Jerome H Friedman, Richard Olshen, and Charles Stone. Classification and regression trees. 1984.
Richard Brent. Fast training algorithms for multilayer neural nets. IEEE Transactions on Neural Networks, (33):346,354, 1991.
Tianqi Chen and Carlos Guestrin. Xgboost: A scalable tree boosting system. In Proceedings of the 22nd acm sigkdd international conference on knowledge discovery and data mining, pp- 785-794, 2016.
Dheeru Dua and Casey Graff. //archive. ics. wuu.ucceee edu/ml
UCI machine learning repository, 2017. URL http:
Wei Fan, Haixun Wang, Philip $ Yu, and Sheng Ma. Is random model better? on its accuracy and efficiency. In Thind IEEE International Conference on Data Mining, pp. 51-58. IEEE, 2003.
Jonathan Frankle and Michael Carbin. The lottery ticket hypothesis: Finding sparse, train- able neural networks. Xiv preprint arXio18803.03333 2018
Jerome Friedman. Greedy function approximation: gradient boosting machine. Annals of statistics, pp. 1189-1232, 2001.
Xavier Glorot and Yoshua Bengio. Understanding the difficulty of training deep feedfor- ward neural networks. In Proceedings of the thirteenth international conference on artificial intelligence and statistics, pp. 2op.249-255 JMLR Workshop and Conference Proceedings,
10

---

Published as a conference paper at ICLR 2023
Yury Gorishniy, Ivan Rubachev, Valentin Khrulkov, and Artem Babenko. Revisiting deep learning learning models for tabular data, 2021. URL
models
Léo Grinsztajn, Edouard Oyallon, and Gaël Varoquaux. Why do tree-based models still outperform deep learning on tabular data? arv preprint wrXiu:2207.08815, 2022.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Delving deep into rectifiers: Surpassing human-level performance on imagenet classification. In Proceedings of the IEEE international conference on computer vision, pp. 1026-1034, 2015.
Guolin Ke, Qi Meng, Thomas Finley, Taifeng Wang, Wei Chen, Weidong Ma, Qiwei Ye, and Tie-Yan Liu. Lightgbm: A highly efficient gradient boosting decision tree. Advances in neural information processing systems, 30, 2017.
Guolin Ke, Jia Zhang, Zhenhui Xu, Jiang Bian, and Tie-Yan Liu. Tabnn: A universal neural network solution for tabular data. 2018.
Diederík P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arviv preprint ar Xio: 412.6980, 2014.
Alex Krizhevsky, Ilya Sutskever, and Geoffrey E Hinton. Imagenet classification with deep convolutional neural networks. Advances in neural information processing systems, 25, 2012.
Yann LeCun, Yoshua Bengio, et al. Convolutional networks for images, speech, and time series. The handbook of brain theory and neural networks, 3361 (10):1995, 1995.
Pengfei Liu, Xipeng Qiu, and Xuanjing Huang. Recurrent neural network for text classifi- 2016
cation with multi-task learning. ar Xrv preprint ar
Dmytro Mishkin and Jiri Matas. All you need is good init. arXiv preprint arXit:1511.06422, 2015.
Radford M Neal. Bayesian learning for neural networks, volume 118. Springer Science & Business Media, 2012.
Kelley R Pace and Ronald Barry. Sparse spatial autoregressions. Statistics & Prob- ability Letters 33(3) 291-297, 1997. ISSN 0167-7152. doi: https///doi.orr/100 1016//00677-7152266600000111 00140--X URL https:://ww.. sci www.aaewwwwweeeeeeeeeeooeeeeeeee aaticieep11/50017715555658
Adam Paszke, Sam Gross, Francisco Massa, Adam James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, Alban Desmaison, An- dreas Kopf, Edward Yang, Zachary DeVito, Martin Raison, Alykhan Tejani, Sasank Chil- amkurthy, Benoit Steiner, Lu Fang, Junjie Bai, and Soumith Chintala. Pytorch: An im- perative style, nigh-performance deep learning library. In Advances in Neural Information Processing Systems 32, pp. 8024-8036
Curran Associates, Inc., 2019.
David L Richmond, Dagmar Kainmueller, Michael Y Yang, Eugene W Myers, and Carsten Rother. Relating cascaded random forests to deep convolutional neural networks for mantic segmeetation. Xie preprint ar Xiv:1507 .070000777733.... 2015.
David E Rumelhart, Geoffrey E Hinton, and Ronald J Williams. Learning internal represen- tations by error propagation. Technical report, California Univ San Diego La Jolla Inst for Cognitive Science, 1985.
David E Rumelhart, Geoffrey E Hinton, James L McClelland, et al. A general framework for parallel distributed processing. Parallel distributed processing: Explorations the mi- crostructure of cognition, (45-76)::26, 1986.
Andrew M Saxe, James L McClelland, and Surya Ganguli. Exact solutions to the nonlinear dynamics of learning in deep linear neural networks. ar arx preprint ar arXiv:1312.6120, 2013.
Ishwar Krishnan Sethi. Entropy nets: from decision trees to neural networks. Proceedings of the IEFF, 18010):1605-1111 1990.
11

---

Published as a conference paper at ICLR 2023
Ravid Shwartz-Ziv and Amitai Armon. Tabular data: Deep learning is not all you need. In- formation Fusion, 81:84-90, 2022. ISSN 1566-2535. doi: https///doi.ogg//1000//////
2021.11.011. URL https:/ http://www...aaaaaaaooooooo/oo/a $156625352 002360
Gowthami Somepalli, Micah Goldblum, Avi Schwarzschild, C. Bayan Bruss, and Tom Gold- stein. Saint: Improved neural networks for tabular data via row attention and contrastive pre-training, 2021. URL /ttpp:://arxllooor////////////// 01342
Ruo-Yu Sun. Optimization for deep learning: An overview. Journal of the Operations Research Society of China, 8(2):249-229) 2020.
Ryan Turner, David Eriksson, Michael McCourt, Juha Killi, Eero Laaksonen, Zhen Xu, and Isabelle Guyon. Bayesian optimization is superior to random search for machine learn- ing hyperparameter tuning: Analysis of the black-box optimization challenge 2020, 2021. 10201
Johannes Welbl. Casting random forests as artificial neural networks (and profiting from it). In German Conference on Pattern Recognition, pp. 765-771 Springer, 2014.
Zhi-Hua Zhou and Ji Feng. Deep forest: Towards alternative to deep neural networks. In Proceedings of the Twenty-Sixth Infernational Joint Conference on Artificial Intelligence, IJCAI- 17, pp. 3553 3559, 2017. doi: 10.24963 / URL htttps://do..ogg///
Zhi-Hua Zhou and Ji Feng. Deep forest. National Science Review, 6(1):74-86, 2019.
12

---

Published as a conference paper at ICLR 2023
A DETAILS ON DEEP FOREST (DF) AND ITS TRANSLATION
The layers of DF are composed of assortment of Breiman's Random Forests and Completely-Randon Forests (CRE, Pan et al. (2003)) and are trained one after another cascade manner. At given layer, the outputs of all forests are concatenated, together with the raw input data. This new vector serves as input for the next DF layer. This process is repeated for each layer and the final output is obtained by averaging the forest outputs of the best layer (without raw data).
Forest
Forest
Forest
Input Feature Vector
Forest
Forest
Forest
Forest
Forest
Forest
Ave.
Forest
Forest
Forest
concatenote
Level 1
Level
Level N
Figure 5: Illustration of the Deep Forest cascade structure for a classification problem with 3 classes. Each level of the cascade consists of two Breiman RFs (black) and two completely random forests (blue). The original input feature vector is concatenated to the output of each intermediate layer. Figure taken from (Zhou & Feng 2017).
Idantily mapping
Foreet
Input
Deep Fornat byye
More Deop Forest layers
Output
Figure 6: Illustration of the MLP translation of a Deep Forest. Yellow nodes use the x H> 2. (a>0) activation function and green nodes use the identity activation function.
B DETAILS OF THE TRANSLATION OF A DECISION TREE INTO AN MLP
Recall that a decision tree codes for a partition of the input space as many parts as there are leaf nodes in the tree. To know in which partition cell an input feature vector æ Rd falls into, we move in the tree from the root to the corresponding leaf using simple rules: at each m-th inner node, than equal to some threshold tmm and to the right child node otherwise. The decision rule at each inner node of the tree introduces a split of the feature space into two subsets H,,, = {ir E Rd (ff..) & tm} and H = {i E Rd (f..) > tm). Consistent with how the intentionally define H and H such that at each inner node m, H., Htt Rd Let N be the number of inner nodes of the decision tree; note that the decision tree has exactly + leaf nodes, since it is by definition a complete binary tree, see Figure [ for illustration. For leaf node & (1....,N N + 1} of the tree, let
passed onto the left child node if i, coordinate is less
MLPP translation works,
PT C (1....,M) NJ (respectively PT) be the set of all inner nodes whose left (respectively
13

---

Published as a conference paper at ICLR 2023
right) subtree contains that is, Pł U UP is the set of all parent nodes of t. Then, the decision tree sorts observation R into its leaf R₃ if and only if
(n( H
n mEP₂
Rr
Him
(1)
iEPP
In fact, (Re Frec the feature space partition coded by the tree, Figure for example. Finally, the tree returns the average response of all training samples that fall into the same leaf as the input data; let us call dg the average response of all training samples in Re. The final prediction of the decision tree can therefore be expressed as
N+1
g(x)
Σ ag (xERs)- E=1
Let us now explore how an MLP can be designed to reproduce the prediction of a decision tree. Consider an MLP of depth 3 with N neurons the first layer. For each inner node m {1...., ]], the m-th neuron of the first layer indicates on which side of the split introduced by this inner node given feature vector lies: equals -1 if the feature vector lies in H., and +1 if it lies in H. This can be achieved applying the following affine transformation
and sign activation function to the feature vector,
if 0 --1
A1 x E Rd + (6m)) tm and P1 : x
if x 0.
The second layer of the 3-layer MLP has N ++ neurons. For each leaf node E , 1, the f-th neuron of the second layer indicates whether a given feature vector x E R2 lies in Rr not: it equals +1 if Rr and -1 if a & Rr. Using equation (1), this be achieved by applying the following affine transformation and sign activation function to the output of the first layer,
A2 A
Σ (m)
Σ
(m))
|tt UP + and :
{i
J-1
x S 0
if x
0.
mEP:
mEP,
The last layer of the MLP contains a single output neuron that returns the tree prediction. Using the output of the second layer, this can be achieved by applying the following affine transformation and an identity activation function,
N+1
N+1 Σ 8=1
Ag A
1
(6) at
ag
and
PP : 2
(2)
2
8=1
where a, the average response of all training samples in R., Note that {ac) is of real numbers in regression problems and a set of probability vectors representing class distributions classification problems. An illustration of the MLP translation of a decision tree is shown in Figure This translation procedure is explained, for example, in Biau et al. (2019) with more details.
C ILLUSTRATION OF OUR INITIALISATION METHOD
showing how the whole MLP initialised using We provide below an illustration (Figure 7
both the tree-based method for the first layers and the random initialisation for the deeper layers.
D DETAIL ON THE MLP TRANSLATION ACCURACY
Recasting a Deep Forest into a deep MLP using our method may suffer from numerical instabilities altering the predictive behaviour. This is due to phenomenon of catastrophic cancellation, more likely to occur with deep MLP translations. This is explained in the following section.
14

---

Published as a conference paper at ICLR 2023
Dre
Tee
(a)
(b)
Figure 7: Illustration of the initialization technique pre-trained tree-based method composed of trees represented in NN fashion involving indicator functions as activation functions. In (b), an MLP of arbitrary depth and involving tanh activation functions is represented at initialization: the weights of the first two layers are initialized using the information captured in (a) (note that all connections marked in transparent blue initialized to 0). The weights of the subsequent layers randomly initialized (orange).
MLP with inputs and 1 output. In (a),
D.1 ON THE CHOICE EYPER-PARAMETERS
four hyper-parameters were introduced to approximate the sign and identity functions through the layers of an elementary MLP. We address here the choice of the HP's and propose an optimal range for these parameters in the sense that they are as small as possible while guaranteeing a faithful MLP translation.
In Section 2.3
We focus on the analysis of deep forest translation, the structure of all other tree-based methods can be seen as a truncated variant of a deep forest. The deep forest is trained and translated into an MLP on each data set (see Section 2 for different values of the HPs. To identify the influence of each HP, we make them vary in some range while the other three HPs are fixed to 10'0 resulting in almost perfect approximation of the respective sign and identity functions. Figure shows the predictive performance of a deep forest and its MLP translation playing with different HPs.
Figure shows in particular that
increasing the HPs beyond some limit value is no longer beneficial as the activation functions are already perfectly approximated;
iii) across multiple data sets, these limit values are similar.
One could note that the coefficients in the first layer of a decision tree translation should be of larger order of magnitude than those corresponding achieve an accurate translation. To give some insight into why this is the case, recall that the m-th neuron of the first layer determines whether the input vector belongs to Hi,,, or H and note that its outputs can be of arbitrarily small size because the vector can be arbitrarily close to the decision boundaries. Note also that an MLP translation would better compromise on translation accuracy to ensure sufficient gradient flow. Based on these observations, we mark that choosing the HP of the following orders allows for maximum gradient flow while still providing an accurate translation: strength01 [1. 10*), st stregth12 E [10-², 10"), st stegthh2 E [10-3 102] and strength id E [10-2 102]. This will actually help us later on to calibrate the search spaces when empirically tuning these HPs for each data set.
the other activation functions to
15

---

Published as a conference paper at ICLR 2023
Nousing
fren!
Aduit
Bank
Covettype
Volkert
strengtho1
strengtha2
strength23
strength Jd
Figure 8: Comparison of the performance of a trained deep forest and its neural network encoding. Deep forest architecture: maximal depth of 8 per tree, trees per forest, forests per layer, 3 layers.
D.2 A FUNDAMENTAL NUMERICAL INSTABILITY OF THE NEURAL NETWORK ENCODING
The encoding of a decision tree by neural network proposed in Section 2.3 is numerically unstable, i.e., does not necessarily yield the same result the tree itself, even when using the original, non-approximated activation functions. This is the result of a catastrophic cancellation that occurs within the MLP translation. The term catastrophic cancellation describes the remarkable loss of precision that occurs when two nearly equal numbers are numerically subtracted. For example, take the numbers a and b 10-10, and perform the computation (a +bb) on machine with limited precision, say to 8 significant digits. The machine will return (a + 6) a = 1 = O, although this result is clearly not correct. This phenomenon occurs in the third layer of the MLP encoding, see equation (2). The two sums calculated in this layer are almost equal in magnitude but have opposite signs, resulting in catastrophic cancellation that has greater impact the more partitions of the input space the decision tree uses, i.e. the deeper it is.
illustratss the effect of this phenomenon, comparing the mean approximation error between a simple decision tree and its neural network encoding on the airbnb data set. In Figure 9a| the result at the output layer of the tree was replaced by the exact training mean of the corresponding decision tree partition, compensating for the catastrophic cancellation. No such compensation was done for Figure 96] This shows the grave implications of this instability: the mean error grows exponentially with the depth of an individual tree.
Although the errors introduced by this phenomenon may not be large for given decision tree, they might accumulate when several such trees are composed, for example in Random or Deep Forests. Figure 10 compares the mean approximation error between Random/ /Deep Forests of different complexities and their corresponding neural net encoding on the Airbnb data set. It shows that the composition of several trees in a cascade manner, as performed by the Deep Forest, leads to stronger amplification of their individual inaccuracies than the parallel composition expected because decision trees composed in parallel do not influence each other's predic- tions, whereas in a cascade architecture the results of the first layer of decision trees affect the input of the subsequent layers and inaccuracies can thus develop stronger effects.
trees, as performed by the Random Forest. This result is to be
16

---

Published as a conference paper at ICLR 2023
of crror
atsolure
0.82
20--
-0.02
1014
20
10
Depth the tree
Dapth of the tree
(a) replacing the output result with the ex- act training mean of the corresp. tree partition
(b) using the output layer's result with catas- trophic cancellation
Figure 9: Illustration of the fundamental numerical instability of the decision tree encoding.
2.0
10 12
54 16
5
10
of layer
(a) Random Forest.
(b) Deep Forest
Figure 10: Effects of numerical instabilities on more complex tree-based predictors. Airbnb data set. Random Forests are composed of trees of depth 7. Deep forest architecture: tree depth of 7, trees per forest, forest per layer and a variable number of layers.
We note that this catastrophic cancellation can be easily circumvented by introducing an additional layer. If this maps the output of the second layer from {-1, to {0, 1], the last layer could then simply multiply each of these outputs by the average response of partition set. However, Figure 10 also shows that the error introduced by the catastrophic cancella- tion remains relatively small, except for deep forests with many layers. Therefore, did not immediately address this issue and planned to fall back on this analysis if the MLP coding did not produce the expected results later in our analysis. However, this somewhat imprecise MLP coding worked well for all our purposes.
E
SUPPLEMENTS TO NUMERICAL EVALUATIONS
E.1 DATA SETS
Data sets description In the sequel, we run numerical experiments on 10 real-world, het- erogeneous, tabular data sets, all but two of which have already been used to benchmark deep learning methods, see Borisov al. (2021): Somepalli al. (2021). The chosen data sets represent a variety of different learning tasks and sample sizes. Tables 2 & 3 respec- tively give links the platforms storing the data sets (four of them are available on the UCI Machine Learning Repository, Dua & Graff 2017) and an overview of their main properties.
The Housing data set contains U.S. Census household attributes and the associated learning task is to predict the median house value for California districts (Pace & Barry 1997). The Airbnb data set is provided by the company itself and holds attributes on different Airbnb listings in Berlin, such as the location of the apartment, the number of reviews, etc. The goal to predict the price of each listing. Similarly, the diamond data set contains characteristics of different diamonds (e.g., carat weight cut quality), and the goal is to predict the price of a diamond. The Adult data set contains Census information on adults (over 16-year olds) and its prediction task is to determine whether a person earns over $50k year. The Bank
17

---

Published as a conference paper at ICLR 2023
Data set
Link
Housing
Scikit-learn
Airbnb
Inside Airbnb
Diamond
OpenML
Adult
UCI Machine Learning Repository
Bank
UCI Machine Learning Kepository
Blastchar
Kaggle
Heloc
FICO
Higgs
UCT Machine Learning Repository
Covertype
UCI Machune Learning Repository
Volkert
AutoML
Table Links to data sets.
Housing Airbnb Diamonds
Adult
Bank
Blastchar
Heloc
Higgs
Covertype
Volkert
Dataset size
640
169268
53940
32561
45-211
7043
9871
550
581 012
58310
Num. features
8
7
3
27
44
147
# Cat. features
0
3
3
8
9
17
2
10
Task
Regr.
Regr.
Regr.
Classif.
Classif.
Classif.
Classif.
Classif.
Classif. 7
Classif. 10
# Classes
2
2
2
2
Table 3: Main properties of the data sets.
data set related with direct marketing campaigns (phone calls) of a Portuguese banking institution, the classification goal is to predict whether the client will subscribe term de- posit. The Blastchar data set features information on customers of a fictional company that provides phone and internet services. The classification goal is to predict whether tomer cancels their contract in the upcoming month. The Heloc data set contains personal and credit record information on people that recently took on line of credit, the classifica- tion task being to predict whether they will repay this credit within 2 years. On the Higgs data set (Baldi et al. 20144), the classification problem is distinguish between signal pro- cesses that produce Higgs bosons and background processes that do not. For this purpose, contains kinematic properties measured by the particle detectors in the accelerator that have been produced using Monte Carlo simulations. The Covertype data set contains car- tographic variables on forest cells and it's task is to predict the forest cover type. Finally, for the Volkert data set, different patches of the same size have been cut from images that be- long to 10 different landscape scenes (coast, forest, mountain, plain, etc.). Each observation contains visual descriptors of one patch, the goal of this classification problem is to find the landscape type of the original picture.
E.2 IMPLEMENTATION DETAILS
and SandomForestClase fier classes with default configuration for all parameters that are not mentioned explicitly. DFs are implemented using the ForestLayer library (Zhou & Feng 2017) and GBDTss are implemented using the XGBoost library (Chen & Guestrin 2016)- MLPs are implemented and trained with pytorch, using the mean- squared error and the cross entropy objective function for regression and classification problems respectively. The SAINT model is implemented using the library provided by
RFs are implemented using sklearn's RandomeooresttRggeee
Somepalli et al. (2021).
All methods are trained on a 32 GB RAM machine using 12 Intel Core i7-8700K CPUs, and one NVIDIA GeForce RTX 2080 GPU when possible (only the GDBT and MLP implemen- tations including SAINT use the GPU). Hyper-parameter searches are parallelized on up to 4 of these machines.
Haper-parameter optimization We tune all hyper-parameters using the ontuna library (Akiba et al. 2019) with a fixed number of iterations for all models. this context, an fleration corresponds to set of hyper-parameters whose performance evaluated with
18

---

Published as a conference paper at ICLR 2023
Covertype*
Volkert
13
6.
3.2
Thaning
MLP rand. Init
MLP GBDT init.
Traiin Set
Test
Figure 11: Optimization behaviour of randomly, RF and GBDT initialized MLP and SAINT evaluated over 5 times repeated (stratified) 5-fold of each data set, according to Protocol PI, but where the MLP width fixed 2048 for all methods. The lines and shaded areas report the mean and standard deviation. "evaluation on single 5-fold cross validation.
respect to given method. The optuna library uses Bayesian optimization and, in par- ticular, the tree-structured Parzen estimator model (Bergstra et al. 2011) to determine the parameters be explored at each iteration of hyper-parameter optimization. This approach has been reported to outperform random search for hyper-parameter optimization (Turner et 2021
Data pre-processing Machine learning pipelines often include pre-processing transfor- mations of the input data before the training phase, necessary step, especially when using neural networks (Bishop & from Geoffrey Hinton 1995). We follow the pre-processing that used in Borisov et al. al. (202)] and SSmepaali et al. (202)). Hence, normalize all contin- uous input features to zero mean and unit variance. This corresponds to linearly transform the input features as follows
where x, is the j-th continuous feature of either train, validation or test observations, and are the mean and standard deviation calculated over the train set only. This way we assure that no information from the validation or test sets is used in the normalization step. Moreover, all categorical features are label encoded, i.e. each level of a categorical variable is replaced with an integer in {l,..., levels].
E.3 WORKING WITH AN ARBITRARY WIDTH IN P1 (OPTIMIZATION BEHAVIOUR)
Figure shows the optimization behaviour of the randomly, RF and GBDT initialized MLP on the multi-class classification problems. Note that in contrast to Figure 2 in this setting, which is less restrictive for RF initialization, this method does indeed lead to faster conver- gence and a better minimum (in terms of generalization).
However, for these multi-class classification problems, the GBDT initialization tends de- teriorate the optimization compared to RF or random initialization methods. Indeed, RF are genuinely multiclassification predictors whose splits are built using all output classes simultaneously whereas splits in GBDT are only built following one-vs-all strategy. This implies that, with a fixed budget of splits (and therefore of neurons), RF are likely be more versatile than GBDT.
E.4 ADDITIONAL MATERIAL FOR PROTOCOL. P2 (GENRRAIIZATION BEHAVIOUR)
E.4.1 DETAILS ABOUT ADDITIONAL NN TRAINING TECHNIQUES
In Protucol 2, we assess the performances generalization of the proposed methods, of the predictors described in Section 3.2 but also three additional NN techniques:
19

---

Published as a conference paper at ICLR 2023
1. the Xavier initialization (Glorot & Bengio 2010) corresponds to rescaled uniform initialization U ~~ +
v6 Vn
where ny are the number of neurons in layer j-
This random initialization is very close to the one used by default in this paper and simply denoted "random init";
2. the Sayer-sequential unit-variance orthogonal initialization (LSUV) MMishkin & Matas 2015) consists in a simple initialization that combines elements of (Glorof & Bengio 2010) and (Saxe et (013)). In a first step, the weights of each layer are initialized as random orthogonal matrices. Then, the variance in the outputs on each layer on the training data is scaled close to 1 by repeatedly dividing the layer's s weights by the empirically determined standard deviation. Although targeted to Computer Vision applications, this approach seems easily adaptable for our case;
3. the winning ticket network pruning (Frankle & Carbin 2018) is more a simplifi- cation approach of the NN architecture during training than an initialization tech- nique. That being said, it remains interesting to compare this strategy to the one developed in the paper, as the winning ticket network pruning enforces NN spar- sity during training. This can be indeed put in parallel the sparsity of the first layers introduced by the proposed initialization and preserved during training, The principle is to train a randomly initialized network, pruning it to obtain a sparse NN with similar performance and then re-train the sparse network second time using the same instance of random initialization as before. These steps are repeated a certain number of times. The winning ticket network pruning is therefore compu- tationally very intense and has to the best of our knowledge only been studied on medium-sized data sets. We thus use a slightly different procedure than (Franklee & Carbin, 2018) to determine winning tickets. First of all, we allocate at most N training epochs to determining a winning ticket where N is the number of epochs during the final model training itself. This fixed number of training epochs is then distributed among n pruning rounds, each of which consists in training the model (for N/n epochs), pruning it, and resetting all non non-pruned weights to their initial (random) coefficients. This approach takes the same time as one-shot pruning but proves to be more efficient.
E.4.2 EXTENSION OF TABLE BEST PERFORMANCES)
a comparison of the performances obtained by ourselves and the literature (where available) for each model. Notice that our results are broadly consistent with those in the literature, with two exceptions. First, our random initialized MLP tends to perform better than in the literature, which can be explained by the fact that we use much larger search space than usual for the MLP width (see Section our performance on Higgs is significantly lower than in the literature. This can be explained by the fact that we only include 5% of the original data set's observations in our analysis due to hardware limitations that do not allow us to train large MLP on 11M samples.
discussion on this). Second,
E.4.3 BENEFITS OF TRAINING THE FEATURE EXTRACTOR VIA GRADIENT DESCENT
In Section 3 we demonstrated ways in which our initialization method can be beneficial for MLP training, resulting in faster convergence towards better minima (in the sens of gen- eralization). natural question that might arise in this context is whether translating the tree-based method into a MLP framework is actually beneficial. After all, one could be tempted to directly use the tree-based method as a feature pre-processing (without trans- lating it into an MLP) and feed the resulting features into an MLP. In this case, the MLP would be trained via gradient descent without the feature extraction. However, turns out that (i) the weights on the sparse feature extraction layer are indeed modified during gradi- ent descent optimization and (ii) training the feature extractor via gradient descent largely contributes to the competitive generalization performance of our initialization method.
Figure 12 shows the histograms of the differences between all MLP parameters at initial- ization (RF strategy) and after training. As the histograms indicate, the weights in all layer
20

---

Published as a conference paper at ICLR 2023
Housing (1)
Airbnb
Diamonds
Covertype (1)
Volkert (8)
Data set
Model
MSE.
MSE x10"
MSE 4 x10
Accuracy
Accuracy
perf.
pert. in
perf. in
our
our results
our results
our
our
literature
results
literature
results
literature
results
Random Forest
0.272400.006 0.000
0.263=0.009
5.39:10.13
9.80=
78.1+0.1
83.6:10.1
66.3+11.3
64.2.40.3
GBDT
0.205=0.010
4.71:10.15
7.38
97.3+0.00
97.0+-0.0
69.0++0.5
71.3++0.4
Deep Forest
0.225=0.008
4.68:10:16
8.23:10.29
92.41:01*
66.3:10.4
MLP rand. init.
0.263:40.088
0.258=0.001
5.07++0.11
15.5++12.5
91.0#0.4
96.7++0.00
63.0+ 1.56
72.2++0.4
MLP RF init.
0.222=0.009
4.661-0.16
7,93:1022
96.7::0.00
74.1:10.4
MLP GBDT init.
0.206=0.007
4.70-10.09
8.15:0335
96,2-400.0
73,5+10.5
MI DF init.
1.234-0.016
0.113
8.28+4024
94.3++00.3*
71.3++0.5
SAINT
0.226:10.044
0.258=0.011
4.811-0.15
17.7+3.83
96.3#0.1
96.61:011
70.14:066
70.110.4
Adult (t)
Bank (9)
Blastchar (8)
Heloc (t)
Higgs (t)
Data
Model
AUC in
AUC 1 in %
AUC f in %
AUC in
AUC in
perf. in
our results
perf.
our results
perf.
our results
perf. in
our
perf.
our
literature
literature
literature
literature
results
literature
results
Random Forest
91.740.2
91.6+t0.3
89.1+-0.3
92.8-0.3
80.6-00.7
84.5
90.04-0.2
91.3--0.6
79.7+-0.0
80.4+0011
GBOT
9284+0.1
92.7=0.3
93.0+02
93.3=20.3
8188=0.3
84.7++10
92.2+000
92.1=0.4
85.9+0.0
82.8+0.1
Deep Forest
91.8=0.3
92.9=0.2
83.7::12
90.3=0.5
81.240.0*
MLP rand. init.
90.3.10.2
90.510.4
91.510.2
91.0.10.3
59.6.10.3
81.4.1122
80.3.0011
80.110.1
85.610.0
83.2.00.3
ML.P RF init
92.1+-03
92.4+0044
84.4+-1.2
91.7-0.4
83.6++0.1
MLP GBDT init.
92.2+00.
92.5=0.3
84.6+11.2
91.5=006
83.0+0.0
MLP DE init. SAINT
91.9=0.4
92.2=20.3
84.2+110
91.4=006
83.3=t..1
91.6.0.4
91.610.3
92.2.0.4
84.710.3
90.7100.2
90.2.0.7
88.310.0
83.7.1..1*
Table 4: Best scores for Protocol P2. For each data set, our best overall score highlighted in bold and our best Deep Learning score underlined. Our scores are based on times repeated (stratified) 5-fold cross validation. For each of our models, HP were selected via the optuna library (100 itera-
tions). Sources for literature values: Borisovv et al. 2021 (f) and Somepalli et al. (021) ($). "score based single 5-fold cross validation.
Diamonds
16'
10:
Bank
3e-2
Covertype
184
22"
10
10-3
16-3
16-1
Je-3
16-1
16-3
16-1
Layer
Layer 2
Layer 3
Layer
Layer 5
Layer 6
Layer?
Figure 12: Histograms of the difference between all MLP parameters at initialization (RF strategy) and after training. Three data sets have been chosen for illustrative proposes. The behaviour in the light of our analysis (see E.4.3) similar on the 7 other data sets.
throughout the MLP are modified during training. In particular, the weights of the first two (RF initialized) layers are not stationary but change to a large extent.
Table shows the generalization performance of MLP initialized with the RF strategy, and compares the two scenarios in which the parameters of the first two layers (that is, the feature extraction layers built using the RF) modified or frozen during MLP training. These results show that training the feature extraction layers is essential for the success of our initialization method.
21

---

Published as a conference paper at ICLR 2023
Data set
Housing
Airbnb
Diamonds
Adult
Bank
Blastchar
Heloc
Higgs
Covertype
Volkert
Model
MSE
A
AUC
ADC1
ADC
AUC
Acc f
Acc. f
MSC
(<to')
(s10")
(in%)
(in%)
%)
MLP rand. init.
0,258=00m
5.07+0.16
15,5+125
90.5=04
91,0+03
81.4+12
80.1=0a
96.7+00 83.2+03
22.2+04
ML.P frozen
0.262=000
14.5+271
13.7+148
91.1203
90,905
84.4409
91.0=06
75.9402
92.2:0a
69.4
MLP RF init.
0.222.0009
4.6611..6
7,9810.22
92.1103
92,4104
84,4112
91.7004
83.610.11
96.1.00
74.1
Table 5: Best scores for Protocol 2. The scores are based on 5 times repeated (stratified) 5-fold cross validation. MLP RF init. frozen refers to the MLP RF init. model where the parameters of the first two layers (that are initialized using the Random Forest) are frozen during training, that is, they are kept at their initial values.
Data set
Housing Airbnb Diamonds Adult
Bank
Blastchar
Heloc
Higgs
Covertype
Volkert
Model
MLP rand. init.
2.47M
1.86M
363k
1.09M
52.4K
13.1M
11.3M
11.6M
1.14M
9.03M
MLP RF init.
33.6M
12.6M
8.42M
29.4M
8.43M
25.2M
16.8M
4.26M
21.1M
17.1M
MLP CBDT init.
8.41M
12.6M
12.6M
33.6M
8.43M
16.8M
25.2M
8.46M
4.32M
21.3M
MLP DF init.
88.1M
34.0M
59.3M
42.0M
46.2M
34.36M
25.8M
43.2M
57.6M
34.1M
SAINT
5688M
27.0M
53.1M
7.20M
6.12M
322M
98.2M
43.2M
6.44M
169M
Table 6: Comparison the number of parameters for each model.
E.4.4 NUMBER OF PARAMETERS OF BEST NEURAL NETWORKS
In Table we compare the number of parameters of each NN method. Although the tree- based inifialised MLP contain more parameters than the randomly initialized ones, the for- mer are mostly sparse and the execution times are close (see Table Z Finally note that the number of parameters the RF/GBDT init. MLP is globally on par with that SAINT (sometimes more, sometimes less) but for smaller execution times (Table and mostly better performances (Table{4) 4
E.4-5 COMPARISON OF EXECUTION TIMES
NEURAL NETWORKS
Table 7 presents a comparison of the execution times of the training of different NN meth- ods using the hyper-parameters determined by the protocol P2. For each model, the total training time (initialization + gradient descent optimization) given, measured up the point where the best validation loss is reached ("early stopping"). ). It shows that RF/GBDT initialized MLP train faster than SAINT and a bit slower than randomly initialized MLP. For completeness, Table 8 gives the execution time for the initialization and training step separately.
E.4.6 OPTIMIZATION BEHAVIOUR
For completeness, Figure 13 shows the optimization behaviour of the randomly, RF and GBDT initialized MLP as well as SAINT under the Protocol P2.
Data set
Model
Housing
Airbnb Diamonds
Adult
Bank
Blastchar
Heloc
Higgs
Covertype
Volkert
MLP rand. init.
5.96 (32)
91.9 (98)
13.3 (31)
11.8 (37)
21.6 (62)
6.78 (24)
14.3 (60) 467 (32)
312 (69)
12.3 (31)
MLP RF init.
37.8 (29)
131
26.8 (25)
17.2
21.5 23)
8.58 (3)
6.61 (15)
253
2040 (9)
28.3 (25)
MIP GBDT init.
22.9 (w)
280 (85)
53.6 (8))
34.5 (31)
7.47 (8)
7.76 (?)
8.54 (9)
63.0 (5)
437 (66)
52.8 (97
MLP DF init.
233 oz)
(48)
182 pm)
99.4 (59)
105 (26)
29.0 (s)
14.7 (19
3280 (76)
5580 (9)
(m)
SAINT
81.9
640 (63)
394 (84)
15.6
52.7 (2)
7.23
51.0 (31) 2310 (19)
6580 (97)
394 (84)
Table 7: Comparison of the execution time in seconds for model initialization and training until the best validation lost is reached. The number of training epochs is indicated in parentheses.
22

---

Published as a conference paper at ICLR 2023
Model
Heasing
Airhmb
Diamords
Blastcher
Higgs
Covertype
Vollert
MLP mand. ini..
0,00/5.6
0.00/91.33 (m) 0,00:133
0.03/1188 (2)
0,002166
0.00/6 000//678.00 em
0.00/14.3
0.00/467
0,00/312
0.00/12.5
MLP KF mil.
2.21/356
1.87/199 122
2.30/24-6
1.16/16.0 (1)
1,89/19
281/5777
1.78/4.83
6.31/247
3.6/20040 (0440
3.70/24.6
MLP GBDT ind
5.36/17.
4.19/226
4.65,48.5 (0)
2.32/52.2 (i)
4,27/3.30
6.7/1
4.63/3.99
4.38/58.8 8
2.30/443
3.88/48.9
MLP LF init
15.1/218
5,31/355
7.19/175
8,36,91 (9)
9.25/96
6.64/223
5.64/9334
18.9 289/3320000 69 11.2/357U
3,57/175
Table 8: Comparison the execution
seconds for MLP Initialization/trainiiin until the best validation lost is reached. The number training epochs is indicated in parentheses. A value of 0.00 indicates running times smaller than 5 10 seconds.
Housing
Airbnb
Diamonds
Adult
125
2001
1200
Bank
Blastchar
Heloc
Higes*
Covertype*
Volkert
GBDT
MLP init.
SAINT
Test
Figure 13: Optimization behaviour of randomly, RF and GBDT initialized MLP and SAINT evaluated 5 times repeated (statisfied) 5-fold of each data set, according to Protocol P2. The lines and shaded areas report the and standard deviation. "evaluation single 5-fold cross validation.
23

---

Published as a conference paper at ICLR 2023
0.10
0.30
00:00
0.75
5000
0.33
Housing
Airbnb
6.330
6.900
3.055
0.335
0.015
2.000
0.300
Adult 1
Bank
0.00
0.75
0.15
0.60
5.70
0.85
0.00
6.75
Covertype
Volkert 1
Ki
GBDT Init.
MLP after training
Figure 14: Comparison of the performance of the RF and GBDT models used for initialization and the final performance of the corresponding MLPs.
E.5
HYPER-PARAMMTER SETTING
E.5.1 SEARCH SPACES
Table 9 shows the HP search spaces that were used determine an optimal HP setting. The same search spaces were used for the experimental protocols P1 and P2. Note that, in Table n_classes corresponds to the number of classes for classification problems and is 1 for regression problems. Furthermore, the different search spaces given for SAINT were used for smaller smaller/larger data sets, where a data set qualifies as smaller if it has less that 50 explanatory variables.
E.5.2 EXPERIMENTAL PROTOCOL P2
Tables 10 and 11 show the HP setting used for the experimental protocol P2. For the search spaces and descriptions of the function of each HP see Table 9
E.6 PERFORMANCES OF TREE-BASED METHODS USED FOR INITIALISATION OF MLP
Figure 14 compares the performance of RF and GBDT models and the performance of opti- mized MLP, initialized with RF and GBDT respectively. We can notice that the difference in performance between GBDT and RF does not systematically turn into the same difference in performance for the corresponding trained networks. This suggests that beyond their respective performances, the very structures of RF and GBDT predictors play an important role in the final MI.P performances.
24

---

Published as a conference paper at ICLR 2023
Method
Parameter
Search space
Function
dexth
(I.... 12)
Random Forests
estimators
[1000)
see here
max features
[0, 1
max depth
(1....12)
esstimabors
[1000]
GBDT
reg alpha
[00
see
reg. rag.landda
[0-
learning rate
0.01,0.3)
forest_depth
(1,2,3)
Number Deep Forest layers Number of forests per Deep Forest layer
n_forests
(1)
Deep Forest
tlestimators
(1000)
depth
(I....12)
RF parametens, see here
max. features
[0.
learningrrate
[100", 10-0"
learning rate SGD training
depth
(I.... 10)
number of layer
MLP random Init.
width
(1....,00 2048)
number of neurons per layer
epochs
(100)
number of SGD training epochs batch size of SGID training
batch_size
(256)
max, dexph
{I....
n_estimators
2048 2008//2xx.epth
Parameters of the RF initializer, see here
max max features
(0. 1]
leamiig.rate
[10 10-1]
learning rate number of layer
training
MI P RF init.
depth
(3.... 10)
width
(2048)
number of neurons per layer
epochs
(100)
number of SGD training epochs batch size of SGID training
batch_size
(256]
strength01
10
strength12
[0.01, 100)
MLP translation parameters, see Section
max, max_depptl depth
(113
n_estimators
(044/((_ceasseee dage
reg-alpha reg. lambaa
[10 8.1
Parameters of the GBDT initializer, see see here
[10-a.1
learning.rate
0.011.0.5 0.5)
ML GBDT init.
learming_rate
10 1,100
learning rate of SGD training number layer
depth
(3 (3.....0
width
(2048)
number of neurons per layer number SGD training epochs batch size of SGD training
epochs
(100)
batch size
[256)
strength01
[1. 10
MLP translation parameters, see Section 2.3
strength12
00011
forest depth n_forests n_estimators
(1,2.3)
Number of Deep Forest layers Number forests per Deep Forest layer
[1}
2048 /mmex depeth
max, depth
(I.... 12)
RF parameters, see here
max features
10. 1
learmingrrate
[10-8 10
learning rate SGD training number of layer
depth
(3. 10)
MLP DF init.
width
20048)
number of neurons per layer
epochs
(100)
number of SCD training epochs batch of SGD training
batch
[256]
strength01
10
strength12
0.00. 100)
strength22
0.01, 100
MLP tramslation parameters, see Section 2.3
strength.id
[0.01, 100)
epochs
(100)
number SGD training epochs batch size of SGD training
batch_size
(256)/(64)
dim
[32, 64, 18, 16]
number neurons per layer in attention block
SAINT
depth
(1.2.3)
number of layers in each attention block number of head in each attention layer dropout used during SGD training
heads
(2. 4.8)
dropout
(0.0.1.0.2, 0.,,
Table 9: Hyper search spaces used for numerical evaluations.
25

---

Published as a conference paper at ICLR 2023
Method
Parameter
Housing
Airbnb
Adult
Bank
Covertype
Volkert
max depth
12
12
11
12
12
12
Random Forests
nesstimators
1000
1000
1000
1000
1000
max features
0.623
0.596
0.943
max max depth
12
9
7
11
10
n estimators
1000
1000
1000
1000
1000
1000
GBIT
reg alpha
0.305
4.60x10-1
2.39x10-8
1.52x10-4
0.728
4.47x00-
reg lambda
1.13x10-2
1.75x10-
1035x10-1
1.07x10-3
6.51x10-+
1.71x00-0
learning.res rate
3.82x10-2
0.238
1.08x 10
1-34x10-2
0.181
0.107
forest depth
9
2
3
n_forests
1
1
1
Deep Forest
n.estimators
1000
1000
1000
1000
1000
1000
max depth
12
11
9
12
12
max features
0.361
0.410
0.166
0.206
0.218
0.134
learning, rate
x.01x10-1
4.21 x0-1x10-1
2.07 x x10
1.1x10
1.15x 10x
2.29 10-5
depth
4
4
4
4
MLP random init.
width
1100
959
1175
856
738
1482
epochs
100
100
100
100
100
100
batch.size
256
256
256
256
256
256
max depth
10
8
8
10
8
nestimators
2
8
8
2
8
max features
0.442
0.321
0.613
0.650
0.897
0.825
learning.ras rate
x.04x10 10
1072x10-1
1.55
10
1.01 10
1.04x 10
1.45x 10-
depth
10
5
4
6
MLP RF init.
width
2048
2048
2048
2048
2048
2048
epochs
100
100
100
100
100
100
batch.size
256
strength01
1090
71.4
strength12
0.0749
34.5
1.05x10-2
5.53x10-*
max.depth
4
4
neestimators
256
128
128
reg-alpha
1.30x10
1.10x10
1,26x
10-x
0.413
x
6.76x10*
reg ragba
1.57x10
9.52x10-4
x.85x10
7.8xx10-
0.643
1.99 x10-7
learning. rate GBDT
0.211
0,297
0.202
0.285
0.112
0.272
MLP GBDT init.
learning.rate
1.11x10
1.97x10-5
x.77x 10
6.22x10
6.19x10-5
106
depth
5
width
2048
2048
2048
epochs
100
100
100
batch size
256
256
256
256
256
256
strength01
575
7830
132
20.5
7280
4.08
strength12
5.60
0.461
66.0
5.52
93.4
7.11x10
forest.depth
n_forests
estimators
2
32
max_depth
10
5
10
mextuuatures
0.350
0.598
0.992
0.322
0.633
0.342
learning rate
1.04x10-
6.67x11--
1.54x 10
3.08
10
1.58x10-5
2.31x10
depth
23
10
13
15
9
MLP DF
width
2048
2048
2048
2048
2048
2048
epochs
100
100
100
100
100
100
batch_size
256
256
256
256
256
256
strength01
515
36.6
41.0
15.5
51.6
1.41
strength12
0.162
0.242
10.6
0.213
0.124
0.154
strength23
1.94
10.4
47.8
1.94
-
0.149
strength.id
3.63x
6
7.44
2.75x
5.09 10
3.69
epochs
100
100
100
100
100
100
batch_size
256
256
256
256
64
256
dim
128
64
32
32
16
SAINT
depth
2
2
2
heads
8
dropout
Table 10: Hyper-parameters used for the experimental protocol P2.
26

---

Published as a conference paper at ICLR 2023
Method
Parameter
Diamonds
Blastchar
Heloc
Higgs
max max.depth
12
12
Random Forests
n estimators
1000
1000
1000
1000
max_features
0.967
0.547
0.607
0.577
depth
11
n.estimators
1000
1000
1000
1000
GBDT
reg.alp reg.alpha
0.341
7.15x10-1
0.123
x
reg.lambda
5.15x10-1
1.59x10-1
1.44x10
0.391
learning-rate
9.17x10-²
10-
0.282
2046x10
forpst_depth
3
n_forests
1
Deep Forest
n. estimators
1000
1000
1000
1000
max_depth
12
2
4
12
max featuress
0.454
0.641
0.196
0.163
learning rate
2.35x10
1.05x 10
1.14x 10
2.26x 10
depth
9
MLP random init.
width
1011
1475
1369
1284
epochs
100
100
100
100
batch size
256
256
256
256
max depth
9
n_estimators
4
max features
0.904
0.425
0,728
0.670
learning_rate
6.07
5.07 100~1000
7.33 1
2.17x10-5
depth
4
6
3
MLP RF init.
width
2048
2048
2048
2048
epochs
100
100
100
100
batch_size
256
256
256
256
strength01
4500
331
1.43
strength12
0.420
42,9
1.06
0.329
max depth
3
5
n. e.eetmmatooss
256
1024
256
64
reg. reg.alpha
4.56x10 2
1.63x
6.21x10-1
x0
reg.lamm lambda
6.17x10
2.19x10-4
3.03x
3.20x10-0
learning rate.aateraa. GBDT
0.214
-
8.42x10-2
0.290
MLP GBDT init.
learning rate
8.94x10-5
4.54x 50-0x10-6
1.36x
depth
6
8
4
width
2048
2048
2048
2048
epochs
100
100
100
100
batch size
256
256
256
strength01
3870
4690
6550
4780
strength12
56.6
21.0
31.8
0.423
forest depth
2
3
n_forests estimators
1
1
128
64
8
max depth
8
max_features
0.695
0.516
0.280
0.572
learning rate
2.04x10-5
2.00x
1,91
9. -6
depth
16
10
8
12
MLP DF init.
width
2048
2048
2048
2048
epochs
100
100
100
100
batch size
256
256
256
256
strength01
21.0
93.0
97.8
1.12
strength12
0.119
20.0
0.987
9.22x100 2
strength23
5.34x10-2
27.1
0.207
strength.id
0.358
0.475
9,70
0.152
epochs
100
100
100
100
batch size
256
256
256
64
dim
128
64
16
SAINT
depth
2
heads
8
dropout
0.2
0.5
0.8
0.8
Table 11: Hyper-parameters used for the experimental protocol P2.
27

---

Published as a conference paper at ICLR 2023
Airbnb
Diamonds
Adult
Bank
shormeshg
Blastchar
Figure 15: Histograms of the first three first and the last layers' weights before and after the MLP training on the Airbnb, Diamonds, Adult, Bank and Blastchar data sets. Comparison between random, RF and GBDT initializations.
E.7 ADDITIONAL FIGURES TO SECTION INIITAIIATION METHODS)
3-5 (ANALYZING KBY ELEMENTS OF THE NEW
Figures and 16 show the same histograms Figure evaluated on the other data set con- sidered in protocol P2. Note the logarithmic y-axis for the first two RF and GBDT initialized layers.
28

---

Published as a conference paper at ICLR 2023
Heloc
Higgs
Covertype
Volkert
Figure 16: Histograms of the first three first and the last layers' weights before and after the MLP training on the Heloc, Higgs, Covertype and Volkert data sets. Comparison between random, RF and GBDT initializations.
29

---

Published as a conference paper at ICLR 2023
30


Reviews:
[{"review_id": "xvXV9ln5ih", "reviewer": "Reviewer_c3Rb", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper proposed a new method to initialize neural networks using pertained tree based models for tabular related data. The new proposed framework includes two components: \n- [1] As proposed by the prior work 'Neural random forest', 2019, tree based models can be converted into a 3-layer MLP.\n- [2] Design the non-linear function with proper selection of parameter to fit into gradient descent training framework.\n\nNumerical result shows that MLP with tree based initialization method converges better than random initialization. Parameters distribution analysis also shows model trained with tree based initialization converges to a solution as compared to random initialization.", "summary_of_the_review": "The paper considers an interesting and important topics on building a neural network schema to handle tabular data. Initialization with tree based model with designed fine tuning process is reasonable sound. However, the novelty of the paper seems limited since majority of the work seems to be inherited from Biau et al. (2019). Experimental results seem weak as compared to the baseline models. Although there are some interesting findings on the converged models, while there is not theoretical analysis on those behavior. ", "strength_and_weaknesses": "Strength\n[+]  Paper is well organized and well motivated. Literature is comprehensive and overall paper is well organized.\n\nWeakness\n[-] Novelty seems limited. The algorithm and contribution of the paper is mainly on developing a schema with neural network to handle tabular data and the major source of improvement is from tree based initialization and fine tuning process, which mostly seems to be obtained from Biau et al. (2019).\n\n[-] Some details might need more clarification. \nCould the author provides an example or demo graph to illustrate more clearly on what the initialized the neural network look like. Figure 1 is obtained from the original paper to show how tree based model can be converted to neural nets in a intuitive way. Could some demo example listed here to more precisely show the initialization step? Same applies to section 2.3 on the details to convert the model to allow gradient descent training.\n\n[-] Experimental results seem weak.\nFrom the results in table 1, it seems MLP initialized with GBDT still lower perform than GBDT along. Is there any analysis on the source of changes, considering the neural networks is fine tuned on top of GBDT models? \n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "1: The contributions are neither significant nor novel.", "empirical_novelty_and_significance": "1: The contributions are neither significant nor novel."}, "extra_text": {}}, {"review_id": "YzrjKafDdcG", "reviewer": "Reviewer_mdb9", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "Authors propose a method to 'translate' a Decision Tree (DT) into a Multi layer perceptron (MLP). They use this method as the basis for a Neural Net (NN) weights' initialization technique that allows them to considerably boost NN's performance on tabular (unstructured) data. Authors provide a thorough set of experimental tests where they compare the performances of their proposed approach to that of standard (random) NN init, as well as against the performances of different DT-based methods and another NN architecture aimed at  tabular data. The results obtained back the validity of their method.", "summary_of_the_review": "In general, I think this paper is very good. My only doubt is how similar is this DT-MLP transformation technique to approaches previously proposed in the field. Other than that, I think the paper is well written and presented, experimental evidence and results check, and novelty and significance are good enough for ICLR.", "strength_and_weaknesses": "- The paper is very well written in structure, language and style, with only minor issues here and there.\n\n- The proposed method is very interesting, although remarkably simple. Authors do a very good job explaining it, nonetheless.\n\n- Experimental evidence provided is comprehensive, and provides good empirical support for the proposed approach.\n\n- Results are highly encouraging to continue this line of research, where different ML algorithms combine and blend, in simple and very effective ways.\n\n- Authors also do not fall short in providing discussion regarding all aspects of their works (inner workings and behavior of their methods, results, consequences and possibilities resulting from their  work).", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "X3SsQujIwUv", "reviewer": "Reviewer_znja", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper proposes a method to initialize MLP neural networks when working with tabular data. The method uses tree-based techniques to initialize the weights of the NN. The authors argue that using this initialization in the first layer of an MLP is sufficient to improve its performance (the initialization of the other layers is random).", "summary_of_the_review": "The proposed paper presents a promising approach with interesting results and analysis. The authors focus on comparing the proposed method with random initialization methods but should consider other initialization approaches to compare. The author also should discuss the proposed approach's limitations.\n", "strength_and_weaknesses": "+ The proposed approach is interesting and presents promising results. \n+ The idea of using decision trees to capture relevant features and their interactions and define a mapping to encode extracted relationships into a neural network is not new. Still, the authors presented a focused approach and interesting experiments to justify its use.\n\n- The paper suggests the proposed approach operates in an implicit regularization during the NN training. However, I’m not convinced that it was entirely demonstrated in the experiments, results, and analysis.\n- The authors compared the proposed NN initialization approach with random weights initialization. However, this is an active research area, and classical approaches should be considered for comparison:\n   - Understanding the difficulty of training deep feedforward neural networks, Glorot and Bengio, 2010\n   - Exact solutions to the nonlinear dynamics of learning in deep linear neural networks, Saxe et al., 2013\n   - Random walk initialization for training very deep feedforward networks, Sussillo and Abbott, 2014\n   - Data-dependent Initializations of Convolutional Neural Networks, Krähenbühl et al., 2015\n   - All you need is a good init, Mishkin and Matas, 2015\n   - Fixup Initialization: Residual Learning Without Normalization, Zhang et al., 2019\n   - The Lottery Ticket Hypothesis: Finding Sparse, Trainable Neural Networks, Frankle and Carbin, 2019\n- The authors do not consider the computational cost of training the tree-based methods to initialize the NN model. \n- What is the performance of the proposed approach on deeper models or using different activation functions? Please provide more comments on that.\n- The authors did not discuss the limitation of the proposed method. Therefore, it will be meaningful to discuss the gap between the experiments in the current version of the paper and the real-world applications.\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Decision: Accept: poster