DOFEN: Deep Oblivious Forest ENsemble
Authors: Kuan-Yu Chen
Ping-Han Chiang
Hsin-Rung Chou
Chih-Sheng Chen
Tien-Hao Chang
Venue: NeurIPS 2024 poster (2024)
Keywords: Tabular Data
Structured Data
Deep Neural Network
Architecture Design
Primary area: deep_learning_architectures
TL;DR: DOFEN (Deep Oblivious Forest ENsemble): a novel deep neural network architecture for tabular data, achieving sota performance compared to deep nerual network baselines and comparable performance with tree-based models.

Abstract: Deep Neural Networks (DNNs) have revolutionized artificial intelligence, achieving impressive results on diverse data types, including images, videos, and texts. However, DNNs still lag behind Gradient Boosting Decision Trees (GBDT) on tabular data, a format extensively utilized across various domains. This paper introduces DOFEN, which stands for Deep Oblivious Forest ENsemble. DOFEN is a novel DNN architecture inspired by oblivious decision trees and achieves on-off sparse selection of columns. DOFEN surpasses other DNNs on tabular data, achieving state-of-the-art performance on the well-recognized benchmark: Tabular Benchmark, which includes 73 total datasets spanning a wide array of domains. The code of DOFEN is available at: https://github.com/Sinopac-Digital-Technology-Division/DOFEN

DOFEN: Deep Oblivious Forest ENsemble
Kuan-Yu Chen Sinopac Holdings 1 Lavamore@sinopac.comm
Ping-Han Chiang Sinopac Holdings u10000129@gmail. com
Hsin-Rung Chou
Sinopac Holdings sherry. www.:cchosssspcccooomm com
Chih-Sheng Chen Sinopac Holdings
Darby Tien-Hao Chang
Sinopac Holdings
sheng770sinopac.com com
National Cheng Kung University
darby@sinopac. com
Abstract
Deep Neural Networks (DNNs) have revolutionized artificial intelligence, achiev- ing impressive results on diverse data types, including images, videos, and texts. However, DNNs still lag behind Gradient Boosting Decision Trees (GBDT) on tabular data, a format extensively utilized across various domains. This pa- per introduces DOFEN, which stands for Deep Oblivious Forest ENsemble. DOFEN is a novel DNN architecture inspired by oblivious decision trees and achieves on-off sparse selection of columns. DOFEN surpasses other DNNs on tabular data, achieving state-of-the-ar performance on the well-recognized benchmark: Tabular Benchmark M. which includes 73 total datasets span- ning a wide array of domains. The code of DOFEN is available at: lttps:
//github. com Sinopac-Diiittal -eettalll-eehhoogyy.ooo D Division/DDDDEFN
Introduction
Tree-based models, including RandomForest [2]. Extra Trees 3 and Gradient Boosting Decision Tree (GBDT) frameworks such as XGBoost LightGBM (5). and CatBoost (). are widely recognized for their simplicity, efficiency, and remarkable performance with tabular data. This has inspired numerous studies investigating the integration tree-based algorithms with deep neural networks (DNNs), leading tree-inspired DNNs such Deep Forest 77. NODE (8) and TabNet [9. In another line of tabular DNN research, novel DNN architectures such as SAINT [10. FT-Transformer [TI], and Trompt [12 have been proposed. These novel architectures, which are essentially ceanntion-basee demonstrate better performance compared with tree- tree-inspired DNNs but require significantly more time and space. While these tabular DNNs have shown promising performance in specific contexts, recent surveys and benchmarks generally indicate that they do not surpass the performance of GBDT' tabular data
15
Hence, we begin by questioning what's missing in existing tabular DNNs and identify one key difference: in tree-based models, only limited number of features are used in the construction of each tree. This concept of "sparse selection of columns" not only increases feature diversity but also helps mitigate overfitting in tree-based models 2 However, existing tabular DNNs are unable to achieve sparse selection. For example, attention-basee models [10-12] use the softmax operation to aggregate column information, resulting in "dense selection" across columns. Some iiceiinsprred DNNs [8,9) have utilized methods like entmax and spansemax
[16] [7 to enhance sparsity, but they still only achieve near-sparse effects. Therefore, opt to develop new approach to achieve this characteristic.
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

Nom Nomskeatioon www.kkodicclooteeeooo bock
CI 40 81 Aeraiio
Nomra Aoreaaaoo ho aouee look
mo
120
13
180
Rumber ofrasdere search Denattrn
aranno soarch Ieratons
Nanber rencer seacc lintons
Nurber of altascors seurch Benstrr
(a) Medium, Classification. (b) Medium, Regression.
(c) Large, Classification.
(d) Large, Regression.
Figure 1: Evaluation results on the Tabular Benchmark. The model names are sorted by their performances the end of the random search of hyperparameters. The result are averaged over various datasets included in each benchmark respectively, detailed number of datasets of each benchmark is provided in Appendix B.1
For deep learning models, the biggest challenge is that generating a sparse matrix for on-off column selection is non-differentiable. In this study, we propose a novel two-step process to work around this issue: (1) enumerating as many sparse selections of columns as possible, and (2) weighting the importance of these sparse selections, making the weights differentiable and trainable by DNN model. We name this new tree-inspired DNN DOFEN, abbreviation for Deep Oblivious Forest ENsemble, and further demonstrate how DOFEN implements these two steps below:
1. Condition Generation and TODT Construction. In DOFEN, the step of enumerating sparse selections is further divided into two parts. The first part generates conditions, each involving exactly one column and corresponding a decision rule of tree node, as described in Section 3.2.1] The second part combines conditions using shuffle-then-resaap procedure, detailed in Section 3.2.2 Each resultant combination of conditions can be seen as differentiable counterpart to the Oblivious Decision Tree (ODT) [8], referred to as relaxed ODT (rODT) in the context. Consequently, all the combinations collectively form a pool of rODTs.
2. Two-level rODT Ensemble. To ensure that the weighting of a limited number of rODTs can achieve good predictive performance, the previous step requires a sufficiently large pool. However, assembling all the rODT's in the pool into a single giant forest tends to cause overfitting, as shown in Appendix H. and Figure 9 Thus, DOFEN implements the step of importance weighting using two-level ensemble procedure. The first level involves ensembling only randomly selected subset of the rODT pool to form individual rODT forests, which is similar to applying dropout [99 to the rODT pool. The second level treats each rODT forest as a weak learner and aggregates them into a forest ensemble. This level is designed to enhance performance and stability, similar to standard ensemble learning. Both level of ensemble detailed in Algorithm [ of Section 3.2.3
To evaluate DOFEN comprehensively and objectively, we have chosen recent and well-recognized benchmark: the Tabular Benchmark [T] This benchmark addresses the issue of inconsistent dataset selection, which is prevalent in deep learning research tabular data. It includes variety of regres- sion and classification datasets with standardized feature processing for consistency. Additionally, we have conducted detailed analyses focusing the distinct features of DOFEN, thereby offering insights into its functionalities. In summary, our research makes two key contributions:
1. Innovative Neural Network Architecture. The DOFEN model is fundamentally inspired by ODTs and incorporates an innovative two-step process to achieve on-off sparse selection of columns. This unique approach enhances performance beyond that of current tree-inspired DNNs and offers differentiability compared to conventional tree-based models.
2. State-of-the-Art Performance. The DOFEN model exhibits outstanding performance, surpassing that of other neural network models and competing closely with GBDT's on the Tabular Benchmark. This achievement underscores its robustness and versatility across various tasks, illustrated in Figure
2

---

2 Related Work
In this section, we start by exploring ODT and detail our rationale for selecting ODT as the founda- tional element in our study. We then systematically categorize deep tabular neural networks into two distinct streams: tree-inspired DNN architectures and novel DNN architectures. Through comparing DOFEN with these established models, our goal is to highlight its unique contributions and position it within the broad landscape of deep tabular network research.
Oblivious Decision Tree. The ODT is a variant of the traditional decision tree algorithm 220 which makes series of feature-based decisions along its root-to-leaf path to deliver a prediction. In the context, a ceature--bsed decision rule, e.g. age > 18, is called a condition. The traditional decision tree algorithm [18] chooses different conditions on different nodes, while in ODT, all nodes the same level apply the same condition, resulting in a more uniform decision-makin
process. This sreision-making thus enhancing computational efficiency, while it also comes at the cost of capacity [8. However, studies have shown that ensembles of ODTs can achieve remarkable performance with sufficient capacity 6 8 In this research, we integrate ODTs as the foundational element in the DOFEN model and capitalize on the strengths of ODTs while mitigating their limitations through ensemble strategies and deep learning techniques.
uniformity allows for streamlined and vectorized
Tree-inspired DNN Architectures. Integrating decision tree (DT) algorithms with DNNs has become prominent for handling tabular data. Pioneering works like Deep Forest (7), NODE g TabNet ())
GradTree 21 and GRANDE [22 have each introduced unique methodologies.
Deep Forest adapts the random forest algorithm and incorporates multi-grained feature scanning to leverage the representation learning capabilities of DNNs. TabNet models the sequential decision- making process of traditional decision trees using DNN, featuring a distinct encoder-decoder architecture that enables self-supervised learning. Grad recognizes the importance of hard, axis- aligned splits for tabular data and uses a straight-through operator handle the non-differentiable nature of decision trees, allowing for the end-to-end training of decision trees. NODE and GRANDE share a similar observation and high-level structure DOFEN, in that they ensemble multiple tree- like deep learning base models. NODE uses ODT as a base predictor and employs a DenseNet-like multi-layer ensemble to boost performance. GRANDE, a successor to GradTree, uses DT as a base predictor and introduces advanced instance-wist wise weighting for ensembling each base model's prediction.
However, DOFEN distinguishes itself from NODE and GRANDE through its unique architectural design. First, DOFEN employs different approach to transform tree-based models into neural networks. Unlike NODE and GRANDE, which explicitly learn the decision paths (i.e., selecting features and thresholds for each node) and the leaf node values of a tree, DOFEN randomly selects features to form rODT's and uses a neural network to measure how well a sample aligns with the decision rule. Additionally, the leaf node value of an rODT is replaced with an embedding vector for further ensembling. Second, DOFEN introduces novel two-level ensemble process to enhance model performance and stability. Unlike NODE and GRANDE, which simply perform a weighted sum on base model predictions, DOFEN first constructs multiple rODT forests by randomly aggregating selected rODT embeddings and then applies bagging on the predictions of these rODT forests.
Novel DNN Architectures. Beyond merging decision tree algorithms with DNNs, significant progress has been made in developing novel architectures for tabular data. Notable among these are TabTransformer (23) FT-Transformer [T]] SAINT [0], TabPFN [24), and Trompt [12] These
models primarily leverage the transformer architecture 25 utilizing self-attention mechanisms to
capture complex feature relationships.
TabTransformer applies transformer blocks specifically to numerical features, while FT- Transformer extends this approach to both numerical and categorical features. SAINT enhances the model further by applying self-attention both column- wise and sample-wise, increasing its capacity. TabPFN, variant of the Prior Fitted Network (PFN) (26). is particularly effective with smaller datasets. Trompt introduces innovative approach by incorporating prompt learning techniques from natural language processing (27), aiming to extract deeper insights from the tabular data's 's columnar structure.
These models have demonstrated impressive performance across various studies and benchmarks. As result, we choose them as our baselines to offer a comprehensive evaluation for deep learning models on tabular data.
3

---

DOFEN: Deep Oblivious Forest Ensemble
In this section, we begin with discussion about how DOFEN relax an ODT to be differentiable in Section 3.1 and elaborate the details of the overall architecture design in Section 3.2 In the following figures and equations, three sub-networks composites of fundamental neural network layers such as linear layers, layer normalization, and dropout- are simplified into symbols A1. 22 and Aa for readability. The detailed configurations of these sub-networks can be found in Appendix A.2
3.1 ODT Relaxation
An ODT operates input vector E, where H R and Nool is the number of columns in tabular dataset, as described in Equation (T). Although these columns can be either numerical or categorical, we focus on real numbers in Equations 2 and (3) to simplify the notations.
== (x; i 1,2, .... Nool), 1.2..... E R
(1)
Fundamentally, an ODT of depth d is a decision table consisting of entries [28] as depicted in Equation (2). Here, 1j indicates the index of selected column, and ø denotes its column value at depth j. The corresponding threshold is denoted by bj. and H denotes the Heaviside function. In practice, the choice of æ1, is decided by predefined criterion, e.g., entropy or Gini impurity. is possible for raw column to be selected multiple times at different depths, each with a varying threshold.
ODT(F) = (H(1,, by)], F Noolh by E R, j = (1,2....,d)
decided entropy. Gini impurity, etc.
((11, by)}.
1
(2)
Equation 2 involves non -on-differentiable caleulations, including the Heaviside function and the predefined criterion. Consequently, the key integrating an ODT within neural network model lies in making the following operations differentiable: selecting columns, deciding thresholds, and modeling H.
To address these challenges, DOFEN proposes a method to relax an ODT, as shown in Equation (3) In DOFEN, the columns of an ODT at different depths are selected randomly. The thresholds and the Heaviside function for column I are replaced with a sub-network Au,, which employs the sigmoid activation function to create soft conditions. To avoid confusion, we introduce new term, relaxed ODT (rODT), in this context. This term distinguishes between the original ODT and the relaxed version proposed in this study, which can be integrated to neural networks.
roDT(F) = (-A,, (11, I randomly sclect
{o1, },
1 E {1, Nowt = =(1,2,..... 21,
3.2
DOFEN Model
3.2.1
Condition Generation
This module transforms imput vector 7 into multiple soft conditions for subsequent modules. The raw input in tabular data comprises combination of numerical and categorical columns. In this study, soft condition is defined as scalar indicating how well a column adheres to a decision rule.
This transformation process creates matrix M, as shown in Equation (4), where Neond is a hyper- parameter denoting the number of conditions we aim generate for each column. Notably, each column I; is processed by individual sub-network Azs in this context, where i {l..... design is derived from the original ODT, where each condition involves only a single column. The sub-network A is embedding layer for categorical column or linear layer for numerical column. Further details of A can be found Appendix A.2 As depicted in Figure 2h three instances of A generate four conditions for each column, resulting in x 4 matrix.
.........e Nool This

---

67
E
inprtt
A.
90
-palled
6g
M
m0g
10
Ag
65
X1
1301
10,11
12:11
Somuuaa 1o0. with
10
Ag
Na
Ava
An
"paind
Aia
-paihed --
Es
Nend
Ag
paleed
6g
(4) Condition Generation
(b) Relased OOT Construction
[c] Forest Construction
generated through individual sub-network Ar. The aggregate of the conditions of all columns is denoted by the matrix M. (b) Relaxed ODT Construction: The condition matrix M is shuffled (i.e. permutation with and reshape into O, representing Norr rODTs each with depth d. (c) Forest Construction: To compute the weights wi, individual sub-networks A2: is applied to each rODT. In addition, each w; is paired with a learnable embedding vector e; The aggregate of all weights and their corresponding embedding vectors denoted ú and E, respectively.
Figure 2: (a) Condition Generation: For each column Ti, Neond conditions
Mi1
mi Noat
M
IR Nound (ma..... M3Noms) Anr(z,), (1,2, .... Nea)
MNNon
Nol
(4)
3.2.2 Relaxed ODT Construction
This module constructs multiple rODTs. Unlike traditional ODT, which selects columms and their corresponding thresholds based predefined criteria. DOFEN randomly selects d elements from the NNond X Nool conditions in matrix M without replacement to build an rODT with depth In our implementation. M is shuffled and reshaped into matrix O with dimensions NorT d, as shown in Equation (5) Here, we use æ to represent a bijective function that maps the index of each element in M to unique position in O (i.e. permutation). The whole process is also illustrated in Figure
Specifically, Noor = N Noomd-Noot/e To guarantee that Noor is an integer, we introduce an interme- diate parameter, m, which ensures that Neond is always multiple of by formulating Neond md. In practice, we use m to adjust Noond instead of directly changing N Noond-
On the other hand, note that each row in O represents an rODT, which is crucial for subsequent operations. To ensure this consistency and the stability during training. the permutation is done only once during model construction and the configuration is then maintained throughout.
0n
O1d
E R
Movr xd
ONomm1
O
( 0yh lj= (n)
kk = π (n) mod d, u x Noo +0] = (m C M,
where 1 S u S Noond. 1 v $ Nool
(5)
5

---

Nettmmator
softmax operation weighted operation
(ws. witho replemmmnnt and repeat Mores times
semple Vosttaation
Stroe
601
90g
903
N Nirest
Noeest
Noees
W
s shered As
N Nentmatee
loss
Miseet
V
6
prodiction
E
(a) Forest Construction
(b) Forest Ensemble
Figure 3: (a) Forest Construction: First, Nestimator pairs of (w, E.) are randomly sampled to form W and E'. Secondly, w is transformed through softmax function, and is used for computing the weighted of E' to form forest embedding f. (b) Forest Ensemble: shared- weight sub-network A3 is employed to make a prediction ý for each embedding. The final prediction the average of all y values, and the total loss is the sum of their individual losses.
3.2.3 Two-level Relaxed ODT Ensemble
This module integrates rODTs to construct forests and then assembles multiple forests to conduct final prediction.
Forest Construction. To construct an rODT forest using the generated rODTs, DOFEN introduces sub-network and standalone embedding vector for each rODT, denoted as A and respectively, where E {1..... NoNT }. The role of 2 is to evaluate how well sample aligns with the conditions of rODT, producing weight scalar W;, as shown Equation (6) and Figure 3
A21 ((on)
07
(wi, .... WNoooT R Nioo
Ag N.arr ((0 Niar ..... Neumma)
The embedding vector é; represents the tree information and is independent of the samples. The embedding vectors are combined into a matrix E, as depicted in Equation (7). where Nudden represents the size of the hidden dimension. Importantly, each tree embedding vector is directly linked to the specific conditions of its corresponding rODT. It crucial to keep this association consistent throughout each training session effectively train the embedding vectors.
El
E
F N where EG Nunten (1, ... NoDD))
EN
To further construct rODT forest, Nestimator of paired weights and embeddings are sampled from Vi and E. This process is graphically represented in Figure E| and described in line to 7 of the pseudo-code for the two-level ensemble (Algorithm []. The weights are processed through softmax function and the weighted The magnitude these softmaxed weights indicate the importance of the selected rODTs for making predictions. Noted that this process is repeated Nives times to form Niorss instances of rODT forests.
of embeddings forms the embedding vector for rODT forest.
Forest Ensemble. To make a prediction, DOFEN applies shared sub-network Ag to the embedding of each rODT forest to make individual predictions. The predictions then averaged for bagging

---

Algorithm 1: Two-level Relaxed ODT Ensemble
Input: Wi, E, Niorest y, L
Output: y. loss
Initialize g, 1.loss 0,0,
2 for r + to Norest do
W. E' sample without replaceement
8 E
1a Nesimatur paired elements are sampled. */
W E R Nantmaree
E' E R
Naimure x Ninkles
Natimon
fth Σ
softmax (aamaax(x')) E
Element-wise multiplication with broadcast.
I R Nusan
represents an rODT forest embedding.
ÿ' Aa(f):
1e Give prediction with shared Aa- * /* Calculate loss with loss function L and aggregate. *
loss loss L() 10 1/'
/ Aggregate each forest's 's prediction. *
end
12 D f- 0/ Novest
13 return (ū, loss);
ensemble. The process is detailed in line I, 8, 10, and 12 in Algorithm T and is illustrated in Figure Notice that the output is scalar for regression tasks and a vector for classification tasks.
During training, DOFEN updates the model parameters by aggregating the loss from each prediction, as shown in line 9 in Algorithm [ The loss function L is cross-entropy for classification tasks and mean squared error for regression tasks.
Notably, the sampling of weight-embedding pairs allows resampling in each forward pass without disrupting the training. In fact, the two-level rODT ensemble essentially implements form of bootstrap aggregating (i.e. bagging) of trees. Conventional tree-based models like random forest bootstrap samples to generate a variety of trees, which are then combined form forest. DOFEN, the ái and E represent tree pool. From this pool, trees are sampled with replacement to create diverse tree sets, or forests, represented by vi' and E'. These forests are then integrated make the final prediction. The design of this tree bagging method enables the construction of varied base models (in this case, forests rather than individual trees) within single training session, which particularly suited to deep learning contexts. Although the randomization may seem chaotic, experiments shows that this approach contributes the model's stability and generalizability, which discussed detail in Section 4.3. and Appendix F.I
4 Experiments
This section presents comprehensive analysis of our experimental results, organized as follows: The Tabular Benchmark and the baseline models are first introduced in Section In Section [.2] we evaluate DOFEN on the medium-sized Tabular Benchmark, while leaving the results for large-sized benchmark Appendix G.T. Section 4.3 delves into DOFEN to elucidate the underlying mechanics that drive its performance. Additionally, we discuss DOFEN's computational efficiency in Appen- dices C.T to C.3 analyze DOFEN's scalability in Appendix and show DOFEN's interpretability in Appendix
4.1 Tabular Benchmark Setup
Datasets, We strictly follow the protocols the Tabular Benchmark as detailed in its official imple- mentation This includes dataset splits, preprocessing methods, hyperparameter search guidelines, and evaluation metrics. For full details, please refer to the original paper [ The Tabular Bench- mark categorized datasets into classification and regression, with features being either exclusively numerical or a combination of numerical and categorical (heterogeneous). These datasets are further
https//githbb.com//aoGmmaaaatatt henp:///abbbrrbbbebmmemc......

---

Womakaaa ancuaan modh ofed to taratin
Norma out scor Comet to
20153
108
of strncorr seurch Bentorn
Runber of rascon search
Mebber rurdem search Barakona
Number ofrrdder Beradons
Exclusively Numerical, Classification
(b) Hcterogencous, Classification
(c) Exclusively Numerical, Regression
(d) Hoterogencous,
Regression
Figure 4: Results on medium-sized classification and regression datasets.
classified according to their sample size: medium-sized or large-sized. The dataset counts from Tabu- lar Benchmark provided in Appendix B. and the detailed datasets used in Tabular Benchmark are provided in Appendix B33
Model Selection. For model comparison, Tabular Benchmark includes four tree-based models: RandomForest, GrasiinttoostinggTre [29]. HGBT [30]. and XGBoost; two generic DNN models: MLP and ResNet [II; and two tabular DNN models: SAINT and FT-Transformer. To ensure comprehensive comparison, we also included two additional tree- based models: LightGBM and CatBoost, and three tabular DNN models: NODE, Trompt, and GRANDE. LightGBM and CatBoost are selected due to their widespread use across various domains. NODE and GRANDE both share similar motivation and high-level structure with DOFEN, while Trompt represents the current state- of-the-art tabular DNNs when following the origin protocols of the Tabular Benchmark. The default hyperparameter configuration of DOFEN and hyperparameter search space of different models presented in Appendices A.I and H.2 and the list of some missing model baselines from Tabular Benchmark is provided in Appendix B.2
4.2 Performance Evaluation
We analyze the results of medium-sized benchmark on classification and regression tasks separately. The evaluation metrics adhere the Tabular Benchmark protocols, which use accuracy for classifica- tion datasets and the R-squared score for regression datasets. We discuss the overall performance in this section and provide comprehensive results for each dataset Appendix G.2
Classification. In Figure 4a the models can be roughly categorized into three groups: (1) tree-based models and three tabular DNN models: DOFEN, Trompt and GRANDE, (2) three other tabular DNN models, and (3) the two generic DNN models. Prior to DOFEN, Trompt was the sole DNN model comparable to tree-based models. DOFEN not only matches but also surpasses the performance of most brsed models, establishing new benchmark for DNN models in tabular data. In Figure DOFEN and Trompt are again the only two DNN models grouped with tree-based models, yet they are positioned at the bottom of this group.
Regression. In Figure 4c. XGBoost stands out as distinct category. Meanwhile, CatBoost and DOFEN represent a second level of performance. Notably, XGBoost and DOFEN demonstrate a significant improvement during the hyperparameter optimization, whereas CatBoost maintains strong performance consistently. In Figure 4d XGBoost and CatBoost continue to hold the top two positions. DOFEN, ending up in sixth place, is overtaken by Gradient comparable with FT-Transformer towards the end the hyperparameter search process.
as well as HGBT, and is
The analysis of Figure 4 allows us to draw several conclusions. When compared to DNN models, DOFEN consistently either ranks first or shares the top positions. Additionally, DOFEN exhibits strong competitiveness against tree- based models. In datasets with numerical features, it consistently places within the top three. However, in the context of heterogeneous features, DOFEN's performance is moderate, typically falling in the middle or lower tiers in comparison with tree-based models. This challenge in managing heterogeneous features is a prevalent issue among all DNN models, highlighting area for potential improvement in future tabular DNN models.
8

---

Table Mean (1) and standard deviation (σ) of DOFEN's performance with 15 random seeds on datasets from different tasks.
Nives
20
50
100 (default)
400
jannis
A (t)
0.7382
0.7747
0.7782
0,7800
0.7808
0.7814
(numerical classification)
a (U)
0.0060
0.0019
0.0015
0.0006
0.0007
0.0004
road-sat safe-y
A (t)
0.7517
0.7712
0.7720
0.7728
0.7732
0.7732
(heterogeneous classification)
a (U)
0.0118
0.0010
0.0007
0.0004
0.0005
0.0003
delays-zurich
J
0.0054
0.0248
0.0258
0.0265
0.0268
0.0270
(numerical regression)
σ (U)
0.0033
0.0009
0.0005
0.0003
0.0003
0.0002
abalone
JA (T)
0.5469
0.5810
0.5846
0,5862
0.5868
0.5870
(heterogeneous regression)
σ (U)
0.0181
0.0038
0.0026
0.0017
0.0010
0.0004
4.3 Additional Analysis
This section is dedicated to deeper exploration of the DOFEN model. Randomness plays an important role in DOFEN, as both the condition selection of an rODT and rODT selection of forest involve random processes. straightforward concern is the stability of DOFEN, which is examined in Section 4.3.1 Moreover, given that the conditions are randomly selected, we investigate whether this randomness leads to redundant trees in Section 4.3.2 addition to randomness, another distinct feature of DOFEN is the introduction of a higher-level ensemble that combines multiple forests, instead merely assembling trees into a forest. Appendix F.I discusses the impact of removing this higher-level ensemble on DOFEN.
All experiments in this section are conducted using the default hyperparameters and medium-sizze sized datasets from the Tabular Benchmark. For evaluation metrics. accuracy is used for classification datasets, while the R-squared score for regression datasets. Except for the cases evaluated on individual datasets, the results represent the averaged metrics across the corresponding datasets.
4.3.1 Model Stability
DOFEN incorporates randomness at two steps: firstly, in the selection of conditions as shown in Equation (5) for rODT construction, and secondly, in the sampling of rODT's as shown line of Algorithm I for two-level rODT ensemble. This section explores how randomness affects the stability of DOFEN.
We start by analyzing the variation in performance of four datasets, where DOFEN ranks first, as shown Table I The standard deviations are even negligible when Niorest (about 0.1% to 1% to mean), except for the delays-zurich dataset. Moreover, with increased Nirress the standard deviations become even smaller (about 0.01% 0.1% to mean). These results suggest that the stability of DOFEN is not issue in most cases (N (Nforest 10), and using the default setting of DOFEN (N (Nforest = 100) ensures both adequate performance and stability for most datasets. Furthermore, the performance improves as the Nivest increases, indicating that the tree bagging of DOFEN not only mitigates instability but also enhances the model's s generalizability.
In addition to analyzing the intrinsic instability. we also replace the steps involving randomness with deterministic alternatives to assess the impact of the randomness on DOFEN from different perspec- tive. For the selection of conditions, we utilize CatBoost to choose columns based on a predefined criterion. The detailed results, presented in Appendix [F3] reveal that the predefined criterion perform only slightly better than the potential for end-to-end training, random selection of conditions remains a viable and promising option. For sampling rODTs, we implement sliding window technique to lock in the selected trees for each forest. The results are detailed Appendix [F4] which suggests that our straightforward approach is comparable to more sophisticated approach.
process. Considering the differentiability and the

---

Relaxed ODTs with large weight variation.
Relaxed ODTs with small weight variation.
Figure 5: In the covertype dataset, Figure shows that the average weights of true positives differ significantly from those of true negatives. Conversely, Figure 5b reveals a contrasting result for rODTs with small weight variation.
4.3.2 Weights of Individual Relaxed ODT
In DOFEN, an rODT is assigned a weight to predict a sample, as shown in Equation (6). In this section, we analyze binary classification dataset (covertype) to observe the variation in the weights assigned to individual rODTs, as shown in Figure
Figure Sigure5aashone that, for most rODTs ranked in the top 25 according to their standard deviations of weights, there a significant difference between the average weights of true positives and those of true negatives. Conversely, Figure 5b shows opposite trend for rODTs with the smallest standard deviations of weights. These trends are also observed another dataset, as shown Appendix FS These observations imply that rODTs with larger standard deviations of weights is more erucial role in classifying samples.
In addition, we come up with an idea to examine the performance change after pruning weights with small standard deviations and their corresponding embeddings, since they are not sensitive to samples with different label. The results are provided in Appendix [n6]and suggest that the variation serves as a reliable indicator of the importance rODTs. Moreover, pruning the less important rODT's not only enhances the model's efficiency but also its performance.
5 Limitation and Conclusion
Limitation. Although DOFEN shows promising results, still contains two weaknesses. First, the inference time of DOFEN is relatively long compared to other DNN models, shown in Appendix C.T However, Appendix [.I also shows that DOFEN possesses the fewest floating point operations (FLOPs). This inconsistency between inference time and FLOPs is mainly caused by the group convolution operation for calculating weights for each rODT (Appendix C.2 which can be improved in the future implementation of DOFEN. Second, the randomization steps involved in DOFEN result in a slower convergence speed. meaning that DOFEN requires more training steps to reach optimal performance. This reflected in the relatively larger number of training epochs needed for DOFEN. Therefore, the workaround strategy of differentiable sparse selection proposed in this study is merely starting point, demonstrating its potential. Finding more efficient strategies will be the future work.
Conclusion. In this work, we proposed DOFEN, novel tree-inspired DNN for tabular data that achieves on-off sparse selections of columns. DOFEN first constructs sufficiently large number of rODTs and randomly ensembles these rODTs into multiple rODT forests make prediction. DOFEN was evaluated on the Tabular Benchmark, achieving state-of-the-art results compared to DNN- -ased models and proving competitive with tree-based ones. Furthermore, we showed that the randomization steps involved in DOPEN do not compromise stability but do yield redundant rODTs. Nevertheless, redundant rODTs can be efficiently removed through our pruning method. In summary, based on DOFEN's s outstanding performance, it has the potential to serve as the backbone model for tabular data across various scenarios, including self- and cami-uuereiieed learning, as well as multi-modal training.
10

---

References
[1] Grinsztajn, L.:; Oyallon, E.; Varoquaux, G. Why do tree-based models still outperform deep learning typical tabular data? Advances in Neural Information Processing Systems 2022, 35, 507-520.
[2] Breiman, I.. Random forests. Machine learning 2001, 45, 5-32.
[3] Geurts, P.; Ernst, D.; Wehenkel, L. Extremely randomized trees. Machine learning 2006, 63, 3-42.
[4] Chen, T.; Guestrin, C. Xgboost: A scalable tree boosting system. Proceedings of the 22nd aem sigkdd international conference on knowledge discovery and data mining. 2016; pp 785-794.
[5] Ke, G.; Meng, Q.; Finley, T.; Wang, T.; Chen, W.; Ma, W.; Ye, Q.; Liu, T.-Y. Lightgbm: A highly efficient gradient boosting decision tree. Advances in neural information processing systems 2017, 30.
[6] Prokhorenkova, L.; Gusev, G.; Vorobev, A.; Dorogush, A. V.; Gulin, A. CatBoost: unbiased boosting with categorical features. Advances in neural information processing systems 2018, 31.
[7] Zhou, Z.-H.; Feng, Deep forest. National science review 2019, 6, 74-86.
[8] Popov, S.; Morozov, S.; Babenko, A. Neural oblivious decision ensembles for deep learning on tabular data, arXiv preprint arXiv: 19909006312 2019,
[9] Arik, S. Ó.; Pfister, T. Tabnet: Attentive interpretable tabular learning. Proceedings of the AAAI conference on artificial intelligence. 2021; pp 6679-6684 6687.
[10] Somepalli, G.; Goldblum, M.; Schwarzschild, A.; Bruss, C. B.; Goldstein, T. Saint: Improved neural networks for tabular data via row attention and contrastive pre-training, arXiv preprint (0t::2/006003442 2021,
[11] Gorishniy, Y.; Rubachev, I.; Khrulkov, V.; Babenko, A. Revisiting deep learning models for
tabular data. Advances Neural Information Processing Systems 2021, 34, 18932-18943.
[12] Chen, K.-Y;; Chiang, P.-H.; Chou, H.-R.; Chen, T.-W.; Chang, T.-H. Trompt: Towards a Better Deep Neural Network for Tabular Data. arXiv preprint arXiv:2305.18446 2023,
[13] Shwartz-Ziv, R.; Armon, A. Tabular data: Deep learning is not all you need, Information Fusion 2022, 81, 84-90.
[14] Borisov, V.; Leemann, T.; Seßler, K.; Haug, J.; Pawelczyk, M.; Kasneci. G. Deep neural
networks and tabular data: A survey. IEEE Transactions on Neural Networks and Learning Systems 2022,
[15] McElfresh, D.; Khandagale, S.; Valverde, J.; Ramakrishnan, G.; Goldblum, M.; White, C.: others When Do Neural Nets Outperform Boosted Trees on Tabular Data? arXiv preprint arXiv:2305.022 02997 2023,
[16] Peters, B.; Niculae, V.; Martins, A. F. Sparse Sequence-to-Seqqeence Models. Proc. ACL. 2019.
[17] Martins, A.; Astudillo, R. From softmax to sparsemax: A sparse model of attention and multi-label classification. International conference on machine learning. 2016; pp 1614-1623.
[18] Kohavi, R. Bottom-up induction of oblivious read-once decision graphs. European Conference on Machine Learning. 1994; pp 164-169.
[19] Srivastava, N.; Hinton, G.; Krizhevsky, A.; Sutskever, I,; Salakhutdinov, R. Dropout: a simple way to prevent neural networks from overfitting. The journal of machine learning research 2014, 15, 1929-1958.
[20] Quinlan, J. R. Induction of decision trees. Machine learning 1986. 1, 81-106.

---

[21] Marton, S.; Lüdtke. S.; Bartelt, C.; Stuckenschmidt, H. GradTree:: Learning axis-aligned deci- sion trees with gradient descent. Proceedings of the AAAI Conference on Artificial Intelligence. 2024; pp 14323-14331.
[22] Marton, S.; Lüdtke. S.; Bartelt, C.; Stuckenschmidt, H. GRANDE: Gradient- Decision Tree Ensembles for Tabular Data. The Twelfth International Conference on Learning Represen- tations. 2024.
[23] Huang, X.; Khetan, A.; Cvitkovic, M.; Karnin, Z. Tabtransformer: Tabular data modeling using contextual embeddings. arXiv preprint arXiv:2012.0065 266777777777888 2020,
[24] Hollmann, N.; Müller, S.; Eggensperger, K.; Hutter, F. Tabpfn: A transformer that solves small tabular classification problems in second. arXiv preprint arXiv:2207. 018488 2022.
[25] Vaswani, A.; Shazeer, N.; Parmar, N.; Uszkoreit, J.; Jones, L..; Gomez, N.; Kaiser, E..;
Polosukhin, I. Attention is all you need. Advances in neural information processing systems 2017, 30.
[26] Müller, S.; Hollmann, N.; Arango, S. P.: Grabocka, J.; Hutter, F. Transformers can do bayesian inference. arXiv preprint arXiv:2112.10510 2021.
[27] Radford, A.; Narasimhan, K.; Salimans, T.: Sutskever, L; others Improving language under- standing by generative pre-training. 2018,
[28] Lou, Y:; Obukhov, M. Bdt: Gradient boosted decision tables for high accuracy and scoring efficiency. Proceedings of the 23rd ACM SIGKDD international conference on knowledge discovery and data mining. 2017; pp 1901.
[29] Friedman, J. H. Stochastic gradient boosting. Computational statistics & data analysis 2002, 38, 367-378.
[30] Pedregosa, F. et al. Scikit-learn: Machine Learning in Python. Journal of Machine Learning Research 2011, 12,
[31] Paszke, A.; Gross, S.; Massa, F:; Lerer, A.; Bradbury, J.; Chanan, G.; Killeen, T.; Lin, Z.; Gimelshein, N.; Antiga, L.; others Pytorch: An imperative style, high-performance deep learning library. Advances in neural information processing systems 2019. 32.
[32] Loshchilov. L; Hutter, F. Decoupled weight decay regularization. arXiv preprint 17X10:1711.05101 2017,
[33] fvcore http::///ggatttocceccccettteeeee
[34]
Averagemn LGBM with hyperop tuning. 2019; https://www/ www.:...gg....m com cttp///oee//eeeee /oooey/////yy//////o 1gbm- Shhhahiitthhheepprprr-tttttintto (Onliee; accessed 5-January-2023),
[35] Bahmani, M. Understanding LightGBM Parameters (and How to Tune Them). 2022; https
//aeppaiee//////g/ggggggggggeeer guide [Online; accessed --Janaay-2223]]
[36] Vanschoren, J.; van Rijn, J. N.; Bischl, B.; Torgo, OpenML: Networked Science in Machine
Leamiing SIGKDD Explorations 2013, 15, 49-60.
12

---

Appendix
Table of Contents
A More DOFEN Settings
14
A.1 Default Hyperparameters Settings for DOFEN
14
A.2 Detailed Model Configurations.
14
A.3
Actual N Notimatr each Dataset
B
More Tabular Benchmark Settings
16
Dataset Counts)
16
B.2
Missing Model Baselines
16
B.3 Mappings of OpenML Task ID and Dataset Name
16
Computational Efficiency Analysis
18
Computational Efficiency Analysis
Long Inference Time DOFEN
................................
Training Time of DOFEN
20
D Scalability of DOFEN
21
E
Interpretability of DOFEN
22
More Analysis
23
F.1
Sampling in Relaxed ODT Forest Ensemble
23
F2
Seed Ensemble
24
F3
An Alternative Strategy for Condition Selection
25
F4
An Alternative Strategy for Weight Selection
25
F55
More Experiments for Section 4.3.2 (.3.2 (Activated TODT for Different Classes)
26
F6
Pruning of Relaxed ODT
26
More Evaluation Results on Tabular Benchmark
27
Performance Evaluation on Large-sized Benchmark
27
2
Detailed Evaluation Results
28
H
More Experiment Settings
44
Hardware Used
44
H2
Hyperparameter Search Space
44
13

---

A More DOFEN Settings
A.1 Default Hyperparameters Settings for DOFEN
In this section, we describe the hyperparameters used in our DOFEN model, along with their default values, shown in Table 2 All notations used here have been previously introduced in Section 3 except for dropout aroe. rate. The dropout rate is applied in dropout layers, and its usage is detailed in Appendix A.2
The calculated Nestimator for each dataset can be found in Appendix A.3 Additionally, the hyperparam- eter search spaces for both the DOFEN model and all baseline models are detailed in Appendix H..2
DOFEN implemented in Pytorch M For hyperparameters used in model optimization (e.g. optimizer, learning rate, weight decay, etc.), all experiments share the same settings. Specifically, DOFEN uses AdamW optimizer [32] with 1e-3 learning rate and no weight decay. The batch size set to 256, and DOFEN is trained for 500 epochs without using learning rate scheduling or early stopping.
Table 2: The default hyperparameters of DOFEN.
Hyperparameter
Default Value
Naa
depends dataset
d'
16
Nood
md
NODT
Noa Naon /d = NasTa
N Nestimatr
max [/Noal] Nondd/ /
N Forss
100
Nindden
128
Nalass
depends dataset
dropout_rate
0.0
depth of rODT an intermediate parameter to ensure that Noor an integer
A.2 Detailed Model Configurations.
In this appendix, we elucidate the specific configurations of the neural network layer composites, denoted as A1, A2, and As in the main paper.
1. A Generate conditions for each column:
A designed to generate conditions for both numerical and categorical data columns, as detailed in Figure For categorical columns in particular, employ embedding layers. These layers utilized to transform categorical features into format that the neural network can effectively process.
2. 22 and A3 Derive weights and make predictions: The layers represented by A₂ and Ag are responsible for generating weights based on the combination of conditions and making predictions, respectively. The relevant structures and processes are illustrated in Figure and Figure
3. Key Parameters:
categories:: This parameter represents the number of distinct categories in a given categorical column.
_rte_rate: This hyperparameter defines the extent of dropout operations applied within the network.
A.3 Actual Nestimator for each Dataset
The Nestimator is calculated through pre-defined formula shown in Table In this section, provide the calculated Nestimater for each dataset Table 3 when using default hyperparameters. Datasets are represented by their OpenML ID described in Appendix B.3
14

---

Dolomi
Calire
Mand
Figure Detailed network layer composite for A1.
(Nuati)
Leyeformni
Depa.tors_ja
UnvetMMutte (iNaa)
Figure 7: Detailed network layer composite for A2.
Figure Detailed network layer composite for Az.
Table Nestimator for each dataset, as long as their Neal and NoDD.
OpenML ID
361086
361294
361094
361289
361293
361085
361082
361 103
361080
Neal
3
4
4
5
6
6
6
NODT
48
48
64
64
80
96
96
96
96
Nestimator
32
32
32
32
OpenML ID
361273
361066
361060
361280
361093
361110
361288
361281
361277
Neal
7
7
7
8
8
8
8
NiOT
112
112
112
112
112
128
128
128
128
Nesimator
32
32
32
OpenML ID
361081
361078
361104
361083
361096
361055
361061
361065
361095
Neal
8
9
9
10
10
10
10
NOOT
128
128
144
144
144
160
160
160
160
Nestimtior
32
32
48
48
48
48
48
48
48
OpenML ID
361076
361098
361291
361099
361286
361087
361084
361063
361074
Neal
11
11
11
13
15
16
NORT
176
176
176
208
240
256
Nesimatar
48
48
48
48
48
48
48
64
64
OpenML ID
361079
361101
361 102
361070
361275
361072
361283
361278
361111
Neal
16
17
20
20
21
22
23
NODT
256
256
272
320
320
336
336
352
368
Nesimator
64
64
64
64
64
64
64
64
64
OpenML ID
361069
361062
361073
361282
361285
361077
361279
361068
361113
Neal
24
26
50
NODT
384
416
416
496
512
528
672
800
864
Nestmatar
64
80
80
80
80
80
96
112
112
OpenML ID
361274
361088
361091
361292
361287
361097
361276
New
54
91
255
359
419
NODT
864
1264
1456
1984
4080
5744
6704
Nestimttrr
112
128
144
176
240
288
320
15

---

B
More Tabular Benchmark Settings
B.1 Dataset Counts
In this section, provide the dataset counts for each task for your reference, presented in Table
Table 4: Dataset counts for each task.
Task
Feature
Count
numerical
16
medium-sizer classification
heterogenous
7
numerical
19
medium sized regression
heterogenous
17
numerical
4
large-sized classification
heterogenous
2
numerical
3
large-sized regression
heterogenous
5
B.2 Missing Model Baselines
We found that two baselines, MLP and HGBT, absent from the evaluation results in the large-sized classification task because they are missing from the official repository. Furthermore, MLP, HGBT, and RandomForest included in the large-sized regression task for the same reason.
B.3 Mappings of OpenML. Task ID and Dataset Name
In this section, we introduce the mappings between OpenML Task IDs and elaborate how to download the corresponding datasets using these IDs.
The mappings are provided in Tables 5 to To access the datasets, please follow the links below, which direct you the OpenML website for each type of dataset. You can then search using the OpenML ID.
Classification datasets with numerical features only: www.:e..pmmm....mmmm http:///wwwwwptooooocoocootsstts
Classification datasets with heterogeneous features: features:
Regressioon datasets with numerical features only:
https::/www. www.:l.pp.mm...m................
//www. httpp///ww..eeeeeecec/c////atete (only for task ID) 361091)
Regression datasets with heterogeneous features:
htttps://wwz http:///ww.....aaoo.toooooaaaaa.
https:: www. /ttt:::///wo..opmmml..m.. http://wwwppoll..oraaaaaaraaaaa/
htt.://ww.....aaaooooooaaaaaaaea (only for task ID 361095)
16

---

Table OpenML Task ID mappings for classification datasets with numerical features only.
OpenML ID
Datasct
361055
credit
361060
electricity
361061
covertype
361062
pol
361063
house_
361065
Megic'oeesccope
361066
bank-marketing
361068
MiniBooNE
361069
Higgs
361070
eye movements Diabetes
361273
361274
jannis
361275
default-of-crede couuut-of-credit-aan candit-erd--lliets
361276
Bioresponse
361277
california
361278
heloc
Table 6: OpenML Task ID mappings for classification datasets with heterogeneous features.
OpenML
Dataset
361110
electricity
361111
yve_movements
361113
covertype
361282
albert
361283
deffull-o--rreddi--aadddeisssss
361285
road-safety
361286
compas-two-years
Table 7: OpenML Task ID mappings for regression datasets with numerical features only.
OpenML ID
Dataset
361072
cpu_act
361073
pol
361074
elevators
361076
wine cine__ullity
361077
Ailerons
361078
houses house 16H
361079
361080
diamonds
361081
Brazilian houses
361082
Like_Sharing__ee
361083
ce-taxi-ggree-dce-200
361084
house_ssales
361085
sulfur medical charges
361086
361087
MiamiHousing0016
361088
superconduct
361091
year
361279
yprop_4_1
361280
abalone
361281
delays_zurich_trameg transport
17

---

Table 8: OpenML Task ID mappings for regression datasets with heterogeneous features.
OpenML ID
Datasct
361093
analcatdata analcatdata_suprreme
361094
visualizing_ soil
361095
black friday
361096
diamonds
361097
Mencedes Benz Maniufatrnn
361098
Brazilian_houses
361099
Saae_Shhrrng__Dmmaaee
361 101
gyc-taxi-green-dce20000
361102
house sales
361 103
particulae-mmatte--uki----000
361 104
SGEMM_ GPU kernel Berneel_peroommnce
361287
topo_2_1
361288
abalone
361289
seattlecrime6
361291
361292
Allstate_ LlaaaCmaaameeeeeee
361293
Airlines_ Daaines_DDDeeela_Il IM
361294
medical medical_charges
C
Computational Efficiency Analysis
C.1 Computational Efficiency Analysis
To discuss the computational efficiency, analyzed the average floating point operations (FLOPs) 33 parameter sizes, and inference time of DOFEN and other baseline models. Our analyses covered both the default and optimal hyperparameter settings, where the optimal hyperparameter delivers the best performance for each model on each dataset. The experiments involving DNN-based models were performed using an NVIDIA GeForce RTX 2080 Ti, while those for the GBDT-basee models utilized an AMD EPYC 7742 64-core Processor with 16 threads.
We begin with the comparison between DNN-based and GBDTT-based models. This comparison primarily focuses on inference time, as FLOPs and parameter sizes are applicable for evaluating the efficiency of DNN-based models but cannot be applied to GBDTs. Additionally, inference times under the optimal parameters are provided only when those parameters are available. As shown in Tables 9 Table 12] the inference times for all DNN-based models are slower than those for GBDT-based models. This expected due to the inherent differences between the two types of models.
When compared other DNN baselines, DOFEN achieves the highest performance, the lowest FLOPs, and the smallest parameter sizes but exhibits the relatively long inference time among all the DNN-based models. This inconsistency between FLOPs and inference time suggests that there is still room for implementation improvements in DOFEN. Hence, we conduct additional experiments to analyze which part the DOFEN model is the computational bottleneck, as discussed in Appendix C.2 showing that the bottleneck of DOFEN arises from using group operations when constructing rODTs. Although this does not affect DOFEN's article, improvements can be made during future open-source releases.
18

---

Table Computational efficiency analysis of default hyperparameterss on medium- sized classification
datasets.
Model
Performance (Accuracy)
FLOPs (M)
Parameters (M)
Inference time (sec.)
DOFEN
0.7725
0.1845
0.0140
0.0125
Trompt
0.7704
53.2127
3.8608
0.0225
FI-Transformer
0.7662
3.3147
0.0908
0.0058
NODE
0.7658
0.8299
0.7525
0.0041
XGBoost
0.7717
0.0015
LightGBM
0.7757
0.0016
CatBoost
0.7777
0.0029
Table 10: Computational efficiency analysis of optimal hyperparameters on medium-sized classifica-
tion datasets.
Model
Performance (Accuracy)
FLOPss (M)
Parameters (M)
Inference time (sec.)
DOFEN
0.7805
0.2093
0.0437
0.0213
Trompt
0.7797
38.7712
2.0398
0.0202
FT-Transformer
0.7686
6.0696
0.2514
0.0061
NODE
0,7677
3,2860
2.6778
0.0033
XGBoost
0.7848
0.0014
LightGBM
0.7838
N/A
CatBoost
0.7858
N/A
The evaluation results are obtained from the Trompl paper without the corresponding optimal hyperparame- ters. Thus, the inference time under the optimal hyperparameters is unavailable.
Table 11: Computational efficiency analysis of default hyperparameters on medium-sized regression
datasets.
Model
Performance (R2 Score)
FLOPs (M)
Parameters (M)
Inference time (sec.)
DOFEN
0.6611
0.1875
0.0173
0.0105
Trompt
0.6541
45,8507
3.8591
0.0224
FT Transformmr
0.6359
2,7795
0.0909
0,0039
NODE
0.1080
0.5839
0.5065
0,0039
XGBoost
0.6719
0,0012
LightGBM
0.6832
0.0014
CatBoost
0.6896
0.0030
Table 12: Computational efficiency analysis of optimal hyperparameters on medium-sized regression
datasets.
Model
Performance (R2 Score)
FLOPs (M)
Parameters (M)
Inference time (sec.)
DOFEN
0.6882
0,2030
0.0364
0.0182
Trompt
0.6830
17.9560
1.2857
0,0200
FT-Transformer
0.6834
9,0576
0.2965
0.0065
NODE
0.6631
2.1379
1.6930
0.0035
XGBoost
0.6985
0.0014
LightGBM
0.6896
N/A
CatBoost
0.6940
N/A
The evaluation results are obtained from the Trompt paper without the corresponding optimal hyperparame- ters. Thus, the inference time under the optimal hyperparameters is unavailable.
C.2
Long Inference Time of DOFEN
To find out the computation bottleneck DOFEN, we analyzed the inference time of each DOFEN module in proportion, shown in Table 13 and Table [4 which is averaged across 59 medium-sized
19

---

datasets with default hyperparameterr. Table Al shows that the Forest Construction module consumes the most inference time. Table A2, more detailed operations reveal that the sub-module A₂ in the Forest Construction module, which generates weights for each rODT, has the longest inference time.
The sub-module 22 is designed with multiple MLP and normalization layers, implemented using group convolution and group normalization to parallelize scoring for each rODT. However, the efficiency of group convolution in Py PoToch has been problematic and remains unresolved. Specifically, the operation efficiency decreases as the number of groups increases, sometimes making it slower than separate convolutions in CUDA streams (see PyTorch issues 18631, 70954, 73764). The A1 also uses group convolution to parallelize condition generation across different numerical columns, resulting slower inference times compared other operations, though less significant than Ag due to fewer groups being used.
However, we mainly foeus the concept and model structure in this paper, acknowledging that model implementation can be further optimized. For example, attention operations are originally slow due quadratic complexity, and many recent works have successfully accelerated the speed of attention operations and reduced their memory usage. Hence, we believe there will be better implementations of these group operations with much greater efficiency in the future.
Table 13: Average inference time proportion of each DOFEN module across 59 medium-sized datasets.
Inference time proportion Inference time proportion (mean)
Module Name
Source
(std)
Condition Generation Relaxed ODT Construction
Figure
7.03
5.29 %
Figure
1.64
0,90
Forest Construction
Figure
and Figure
87.39 %
7.68 %
Forest Ensemble
Figure
3,94 %
2.36 %
Table 14: Average inference time proportion of each DOFEN operation across 59 medium-sizer datasets.
Inference time proportion (mean)
Module Name
Operation
Source
Condition Generation Relaxed ODT Construction Forest Construction
A
Equation
7.03 %
permutation and reshape
Equation
1.64
A2
Equation
85.11
get 1ODT embedding sample rODTs form forests softmax weighted sum As average forest predictions
Equation
0.22 %
Algorithm Ine
1.01 %
Algorithm
line
1.05 %
Forest Ensemble
Algorithm
line
3.55
A Algorithm
line 10 and 12
0.39 %
C.3
Training Time of DOFEN
To know more about how the slow inference time will affect the training time of DOFEN, we also conducted experiment compare the training time of DOFEN with other deep learning methods included our paper (i.e. Trompt, FT Transformer, and NODE). We measured the training time on medium-sized datasets using both default and optimal hyperparameter settings, where the optimal hyperparameters refers to the settings that deliver the best performance for each model on each dataset.
This experiment was conducted using a single NVIDIA Tesla V100 GPU. During model training, carefully ensured that other computational processes were rumning concurrently enable fair comparison. Additionally, we excluded datasets that would cause OOM (Out of Memory) issues during training, resulting the selection of 50 out of 59 medium-sized datasets.
The average training time across datasets for each model provided in Table A7. The results show that the training time for DOFEN is relatively long, approximately twice as long Trompt when
20

---

using optimal hyperparameters. This extended training time may be due to the inefficient group operations involved in DOFEN, which consume about 80% of the computation time during the forward pass. For more details, please refer to Appendix C.2 Therefore, improving the efficiency of group operations could reduce both the training and inference time DOFEN.
Table 15: Average training time of different methods using default and optimal hyperparameter settings on 50 medium-sized datasets. Numbers are in Seconds, with lower values indicating faster training speed.
Model Name
Training Time (Default)
Training Time (Optimal)
DOFEN
332.6998 +/- 125,1965 1965
1143.7674 +/- 804.3809
Trompt
552.3495 +/- 213.3278
535.1781 +/- 291.9933
FT Tranrssffrrrrr
80.3425 +/- 57.2647
99.1068 +/- 79.2272
NODE
95.0274 +/- 54.7463
427.8625 +/- 394.1191
D Scalability of DOFEN
To discuss the scalability of DOFEN, we have conducted experiments to investigate its performance given changes in hyperparameters m, d, and the number of MLP layers stum_layers). In detail, changes in m and d affect the number of conditions (Neond), while alterations in m impact both the total number of rODTs (Novr and the number of rODTs within an rODT forest (Nestmmtor) For further details on these parameters, please refer to Table The num layers hyperparameter, newly introduced, refers to the number of MLP layers in neural networks A1, Ag, and Ag. A detailed introduction to A,, 2 and A3 can be found in Appendix A.2
Due to limited computational resources, only conducted this experiment on datasets that would not cause out-of-memory (OOM) issues on our machine across all hyperparameter settings. This selection resulted in 51 out of 59 medium-sized datasets and 10 out of 14 large-sized datasets.
Based on Table 16 to Table 21 we observed that larger values of m and enhance DOFEN's performance. Notably, improvements are more significant with large-sized datasets than with medium- sized datasets, likely because larger datasets benefit more from increased model capacity. In contrast, Table reveals that increase in hum_loyers generally results in poorer performance. This could be attributed to the substantial growth in parameter size and FLOPs, compared to adjustments the and d, potentially leading to overfitting.
Table 16: Analysis of performance and efficiency across varied settings of m on medium sized datasets.
4
8
16 (default)
32
64
Performance ( (Accuracy)
0.7491
0.7552
0.7602
0.7601
0,7603
Classification
Parameters (M)
0.0029
0.0042
0.0070
0.0134
0.0296
FLOPs (M)
0.1797
0.1802
0.1815
0.1849
0.1951
Performance (R2 score)
0.6496
0.6488
0.6796
0.6940
0.6603
Regression
Parameters (M)
0.0026
0.0035
0.0056
0.0105
0.0235
FL.PPs (M)
0.1783
0.1787
0.1797
0.1825
0.1912
Table 17: Analysis of performance and efficiency across varied settings of m large-sized datasets.
m
4
8
16 (default)
32
64
Performance (Accuracy)
0.7498
0.7635
0.7800
0.7922
0.8010
Classification
Parameters (M)
0.0033
0.0050
0.0084
0.0159
0.0333
FLOPPs (M)
0.1798
0.1804
0.1819
0.1854
0.1949
Performance (R2 score)
0.7227
0.7521
0.7583
0.7698
0.7697
Regression
Parameters (M)
0.0025
0.0034
0.0058
0.0127
0.0350
FLOPSs (M)
0.1783
0.1788
0.1803
0.1856
0.2045
21

---

Table 18: Analysis of performance and efficiency across varied settings of on medium-siee sized datasets.
d
2
(default)
Performance (Aceuracy)
0.7402
0.7588
0.7602
0.7583
0.7545
Classification
Parameters (M)
0.0058
0.0064
0.0070
0.0087
0.0108
FLOPs (M)
0.1801
0.1807
0.1815
0.1834
0.1857
Performance (R2 score)
0.5961
0.6699
0.6796
0.6111
0.6914
Regression
Parameters (M)
0.0047
0.0051
0.0056
0.0069
0.0087
FL.PSs (M)
0.1786
0.1791
0.1797
0.1812
0.1831
Table 19: Analysis of performance and efficiency across varied settings of d on large-sized datasets.
d
2
3
4 (default)
6
8
Performance (Accuracy)
0.7433
0.7726
0.7800
0.7853
0.7916
Classification
Parameters (M)
0.0071
0.0077
0.0084
0,0102
0.0125
FLOPPs (M)
0.1803
0.1810
0.1819
0.1840
0.1865
Performance (R2 score)
0.6572
0.7443
0.7583
0.7694
0.7704
Regression
Parameters (M)
0.0043
0.0050
0.0058
0.0082
0.0113
FLOPs (M)
0.1787
0.1794
0.1803
0.1828
0.1860
Table 20: Analysis of performance and efficiency across varied settings of nummlayeess on medium-
sized datasets.
Run_layers
Default (1, 2,
Twice (2, 4, 4)
Triple (3,
Performance (Accuracy)
0.7602
0.7592
0.7481
Classification
Parameters (M)
0,0070
0.0189
0.0308
FLOPs (M)
0.1815
0,5311
0.8808
Performance (R2 score)
0.6796
0.6595
0.7731
Regression
Parameters (M)
0.0056
0.0150
0.0245
FLOPs (M)
0.1797
0.5267
0,8737
Table 21: Analysis of performance and efficiency across varied settings of num_layers large-
sized datasets.
namerayers
Default (1, 2, 2)
Twice (2, 4, 4)
Triple (3,
Performance (Accuracy)
0.7800
0.7959
0.7638
Classification
Parameters (M)
0.0084
0.0231
0.0379
FLOPPs (M)
0.1819
0.5346
0.8873
Performance (R2 score)
0.7583
0.7575
0.6715
Regression
Parameters (M)
0.0058
0.0139
0.0220
FLOPs (M)
0803
0.5259
0.8715
E
Interpretability of DOFEN
This section aims to demonstrate the interpretability of DOFEN. Specifically, adopt feature importance metric akin to the "split" or "weight" importance used in LightGBM and XGBoost, which counts how often a feature is used in the model.
To calculate DOFEN's feature importance of a specific sample, let NoNo a matrix of feature occurrences across different rODTs. We then use the output of sub-module A2, vector E Nan (Fquation (6)), to represent the importance across all rODTs for each sample, as this weight w is used for constructing rODT forest perform prediction in DOFEN model. A softmax
22

---

operation is further applied to the vector प to ensure the importance sums to (also done in line of Algorithm|] D. Finally, we perform a weighted sum between the feature occurrences and the importance of each rODT, resulting in single vector INNo representing DOFEN's feature importance for a specific sample. To calculate DOFEN's overall feature importance of dataset, we simply average the feature importance of all samples in training dataset.
We tested the reliability of DOFEN's feature importance on three real-world datasets: the mushroom dataset, the red wine quality dataset, and the white wine quality dataset, following the experimental design used by Trompt. The results of these three datasets are shown in Tables [22 to 24 respectively, The results indicate that the top-3 important features identified by DOFEN align closely with those selected by other tree-based models, with only minor ranking differences. This demonstrates DOFEN's ability reliably identify key features while maintaining interpretability despite its deep learning architecture. This further indicates that DOFEN may contain similar decision-making process as tree-based model does, as it is tree-inspired deep neural network.
Table 22: Top 3 Feature importance of DOFEN mushroom dataset. 2nd
1st
Random Forest
odor (15.11%)
gill-size (12.37%)
gill-color (10,42 %)
XGBoost
spore-print-color (29.43%)
odor (22.71%)
cap-color (14.07%)
LightGBM
spore-print-color (22.08%)
gill-color (14,95%)
odor (12.96%)
CatBoost
odor (72.43%)
spore-print-color (10.57%)
gill-size (2.71%)
Gradient
gill-color (31.08%)
spore-print-color (19.89%)
odor (17. (17.4%%)
Trompt
odor (24.93%)
gill-size (8.13%)
gill-color (5.73%)
DOFEN (ours)
odor (13.15%)
spore-print-color (6.84%)
gill-size (5.58%)
Table 23: Feature importance of DOFEN on red wine dataset.
1st
2nd
3rd
Random Forest
alcohol (27.17%)
sulphates (15.44%)
volatile acidity (10.92%)
XGBoost
alcohol (35.42%)
sulphates (15.44%)
volatile acidity (7.56%)
LightGBM
alcohol (26.08%)
sulphates (15.75%)
volatile acidity (10.63%)
CatBoost
sulphates (16.29%)
alcohol (15.67%)
volatile acidity (10,40%)
GradientBoostinggTre
alcohol (26.27%)
sulphates (16.24%)
volatile acidity (11,12%)
Trompt
alcohol (11.83%)
sulphates (10.94%)
total sulfur dioxide (9.78%)
DOFEN (ours)
alcohol (11.16%)
volatile acidity (10.77%)
sulphates (10.17%)
Table 24: Feature importance of DOFEN on white wine dataset.
1st
2nd
3rd
Random Forest
alcohol (24.22%)
volatile acidity (12.44%)
free sulfur dioxide (11.78%)
XGBoost
alcohol (31.87%)
free sulfur dioxide (11.38%)
volatile acidity (10.05%)
LightGBM
alcohol (24.02%)
volatile acidity (12.47%)
free sulfur dioxide (1 (11.45%)
CatBoost
alcohol (17.34%)
volatile acidity (12.07%)
free sulfur dioxide (1 (11.47%)
GradientBoostingTTie Tree
alcohol (27.84%)
volatile acidity (13.59%)
free sulfur dioxide (12.87%)
Trompt
fixed acidity (10.91%)
volatile acidity (10.47%)
pH (10.37%)
DOFEN (ours)
alcohol (10.90%)
free sulfur dioxide (10.21%)
volatile acidity (10.01%)
F
More Analysis
F1
Sampling in Relaxed ODT Forest Ensemble
The forest ensemble in DOFEN is a level higher than common tree-based models. This section attempts to explore its impact on model performance by ablating this higher-level ensemble. The experiment involves using all the constructed rODTs from Equation (5) to form single forest, without the sampling of weights and embeddings described in line Algorithm In practice,
23

---

we directly apply softmax function to the weight vector wi and calculate the weighted sum of corresponding embeddings E. As result, there is only one prediction for each sample, unlike Noxest predictions as in Algorithm
From the results in Table 25 it can be that with the ensemble of forests, the performance of DOFEN is improved across all datasets, independent of the types of tasks and features. The average improvement in classification datasets has reached 0.0363 in accuracy, while in regression datasets has reached 0.3367 in the R-squared score.
To further investigate the drastic drop in performance without using sampling in forest ensembles, analyze performance at various training checkpoints. As shown in Figure | when sampling is not used in forest ensembles, training performance is significantly better compared to testing performance, and testing performance decreases with increasing training epochs, indicating an overfitting issue. Conversely, with ensemble of multiple forests, both training and testing performance improve concurrently, thus mitigating the overfitting issue.
Based on the performance improvements from introducing forest ensemble, we have applied an extra level of ensemble that combines multiple DOFEN models. However, the enhancement in performance is negligible. The detailed results can be found in Appendix F2
Table 25: Comparing DOFEN with and without sampling in forest ensemble
w/ (default)
Classification
0.7725
0.7362
Numerical Only
0.7920
0.7526
Heterogeneous
0.7281
0.6988
Regression
0.6605
0.3238
Numerical Only
0.6814
0,1867
Heterogencous
0.6371
0.4770
0.95
Accurecy
0.90
R2 Score
train,
test, w/
0.85
0.7
train, test, w/o
0.80
0.6
0.75
100
200
300
400
500
100
200
300
400
500
Number of Training Epochs
Number of Training Epochs
(a) Classification
(b) Regression
Figure 9: Overfitting arises when not using sampling in the forest ensemble, affecting both (a) classification and (b) regression tasks. "Train" refers to training performance, and "Test" refers to testing performance. "w/" indicates the use of sampling construct multiple forests, while "w/o" indicates the use of all constructed rODTs to form single forest.
F.2 Seed Ensemble
In this section, we explore the results of applying an additional layer of bagging ensemble to the DOFEN model, technique we denote as the seed ensemble. Specifically, we infer the trained DOFEN model using 15 different random seeds. This results in 15 distinct predictions due to the random sampling procedure conducted in the rODT Forest Construction module. Building upon these predictions, we average the 15 different predictions to create another layer of bagging ensemble, which we then present as the final prediction. This experiment is conducted on a medium-siee sized tabular benchmark. As shown in Table 26 this seed ensemble approach further enhances the performance of the DOFEN model, even with a small Niorss The results further suggest that the DOFEN model can casily benefit from ensemble strategies, thanks the random sampling procedure.
24

---

Table 26: Comparing evaluation performance with and without seed ensemble at varying Noorest.
Nivest
10
20
50
100 (default)
300
w/o seed ensemble Classification
0.7698
0.7713
0.7725
0.7725
0.7726
Regression
0.6568
0.6586
0.6589
0.6605
0.6607
Classification
0.7727
0.7732
0.7731
0,7731
0.7731
seed ensemble
Regression
0.6619
0.6619
0.6616
0.6616
0.6616
F.3 An Alternative Strategy for Condition Selection
In the rODT Construction module, we implement a shuffle-then-reshape procedure to construct rODT's as outlined in Equation ( The shuffle of matrix M serves as a straightforward approach to delivering diverse set of condition combinations for subsequent segmentation. In this section, we aim to experiment with an alternative strategy for selecting columns to construct rODTs, which we adopt the column selection strategy used by CatBoost to form the rODT's in our DOFEN model, we denote this approach as 'Catboost-Init'. Specifically, we begin by selecting machine learning algorithm that also employs ODT as the base element, namely, CatBoost. Subsequently, we train the CatBoost model and use the columns it identifies post-training to construct our rODTss In the context of experimental configurations, to ensure a fair comparision, it crucial to equate the capacity of CatBoost model with the default settings of DOFEN. To achieve this, we set the depth and number of boosting iterations of Catboost as the depth an rODT (d) and the total number rODT (Norr)) in DOFEN, respectively. The CatBoost trained based on these configurations is denoted as "CatBoost** in the context.
The results are presented in Table 27 As can be seen, the 'Catboost-Inir approach achieves performance comparable to the "CatBoost*' approach and generally outperforms the default shuffle approach. This indicates that designing a more sophisticated approach for rODT construction indeed contributes to better performance. However, adopting a selection strategy from tree-based model results in two-stage modeling process, which contradicts our goal of designing end-to-end differentiable DNN model. This intriguing insight leads us to consider a more innovative end-to-end condition selection approach, which we will explore in future work.
Table 27: Comparing the column selection strategy of DOFEN.
Shuffle (default)
Catboust-Init
Catboost*
Classification Regression
0.7725 0.6604
0.7769 0.6792
0.7722
0.6811
E4 An Alternative Strategy for Weight Selection
In the Two-level rODT Ensemble module, our default method involves random sampling of weights. We contrast this approach with an alternative weight selection strategy, denoted as sliding window selection. The method creates Nioess windows, each containing Nestimater weights. These windows are then evenly distributed across NODD weights. Within each window, the weights are collectively treated as rODT forest. As shown in Table 28 our results indicate that random sampling yields better performance compared to the sliding window selection. This finding substantiates that the random sampling approach already works well compared to a sophisticated alternative. Investigating a more advanced and effective approach is worth considering in future work.
Table 28: The comparison of random sampling and sliding window selection weights.
random sampling (default)
sliding window selection
Classification
0.7 0.7725
0.7716 0.6593
Regression
0.6605
25

---

F,5 More Experiments for Section 4.3.2 (Activated rODT for Different Classes)
In this section, we aim to test whether the findings presented in Section 4.3.2 can be replicated on another dataset. Consequently, we have chosen another binary classification dataset (compass-tww) years) and repeated the experiment using the same settings. The experimental results are depicted in Figure TO As shown in Figure [0a] weights with higher standard deviation exhibit distinct distributions between true positive and true negative samples. Conversely, Figure TOb illustrates an opposite trend. Overall, conducting the experiment on another dataset further validates our findings.
(a) Relaxed ODT's with large weight variation.
(b) Relaxed ODTs with small weight variation.
Figure 10: In the compass dataset, the weights wi of rODT are sorted based on the standard deviation calculated across true positive (TP) and true negative (TN) samples in the testing data. Figure T0a shows that the weights of TP samples differ significantly from those TN samples when the standard deviation of the weights higher. Conversely, Figure reveals contrasting results for weights with a lower standard deviation.
F.6 Pruning of Relaxed ODT
Following Section 4.3.2 in this section, we aim to examine the performance change after pruning weights with small standard deviations and their corresponding embeddings.
Table 29 shows the performance under different pruning ratios. The column labeled "by dataset' indicates that we tailored the pruning ratio for each dataset based on its validation data. As shown in Table 29 pruning these rODTs does not negatively affect performance. In fact, a minor degree of pruning can actually enhance performance, with the optimal pruning ratio being 0.02 for classification datasets and 0.1 for regression datasets. Notice that the 'by dataset' approach is better suited to real-world scenarios, even though it does not always yield the best performance.
Table 29: Pruning of rODT with varying ratio. Weights w; with lower standard deviation are pruned.
Ratio
0.0 (default)
0.02
0.1
0.2
by dataset
Classification
0.7725
0.7733
0.7726
0.7709
0.7732
Regression
0.6605
0.6629
0.6630
0.6621
0.6657
We then investigate the outcomes when weights with higher standard deviations are pruned. Conse- quently, sort the weights and prune them from the higher end. The results, presented in Table 30 show that the performance in both classification and regression tasks monotonically drops as the prune ratio increases. This finding suggests that the standard deviation of weights is good indicator of their importance making predictions. It further validates why pruning weights with lower standard deviation does not harm performance and, in some cases, even helps.
Table 30: rODT pruning with varying ratio. Weights w; with higher standard deviation are pruned.
Ratio
0.0(default)
0.02
0.05
0.10
0.2
Classification
0.7725
0.7725
0.7715
0.7667
0.763
Regression
0.6605
0.6571
0.6484
0.6383
0.601
26

---

In addition, we discuss another, potentially more straightforward, pruning approach. Specifically, we prune the weights w, based on their average value across samples. Similar to the experiments that use standard deviation the metric for pruning, this time we sort the weights by their average. We then attempt to prune the weights from both the top and bottom ends. The results are provided in Table 33 and Table 32 suggesting that the value of weights is not an effective indicator for pruning. Although there is some improvement in performance at a low ratio, this approach generally diminishes performance with larger ratios, regardless of whether the weights are pruned from the higher or lower end.
Table 31: rODT pruning with varying ratio. Weights wy with lower average value are pruned.
Ratio
0.0(default)
0.02
0.05
0.10
0.2
Classification
0.7725
0.773
0.7715
0.7722
0.7703
Regression 0.6605
0.6611
0.6592
0.6575
0.6425
Table 32: rODT pruning with varying ratio. Weights w, with higher average value pruned.
Ratio
0.0(default)
0.02
0.05
0.10
0.2
Classification
0.7725
0.7731
0.7725
0.7704
0.7643
Regression
0.6605
0.6619
0.6573
0.6352
0.4881
G More Evaluation Results on Tabular Benchmark
G.1 Performance Evaluation on Large-sized Benchmark
This section discusses the evaluation results large-sized classification and regression tasks. Overall, the results demonstrate a similar trend as the medium sized tabular benchmark. Notably, DOEFN achieves the top ranks in both tasks with numerical features.
morkl (on valid set up this teeaton
DOFEN
seli up to this heerton
CatBosst Lighf
Normalzed test accuracy of best
Normal ized test accuracy of best model (on valid set
0.9
0.9
Grag
Salle
0.7
RandanEerdd
GRANDE 20
0.7
RODE
SAIT
Reshet
GRANDE
0.6
0.6
Number of search iterations
Number of search iterations
(a) Exclusively Numerical
(b) Heterogeneous
Figure 11: Results on large-sized classification datasets.
Classification. In Figure 11a DOFEN even surpasses CatBoost to become the top performer.
Conversely, in Figure TTb CatBoost clearly outperforms other models. FT-Transformer, Trompt and DOFEN are the best- best-performing tabular DNN models, though they rank in the middle among all models. As a result, with the current development of tabular DNN models, their performance in processing numerical features is already on par with or even surpass that of tree-based models, and they are more advantageous for large-sized datasets. However, DOFEN and other DNN models are still less efficient in handling heterogeneous features.
27

---

Normalized test score of best model valid se9 up to this itarztion
Normallzed test P2 score of best model (on valie so0 up to this ntarztnnn
DOFEN
Gridkint Grddkin5@aaaaggg 6a
Catlooss
Lights BM
BM
Flalry Flalnfesmme festmner
0.7
Tamet
0.7
F1-Transtorme SAINT
Becket
NO3E
0.5
GREMDE
0.
_RRANO
100
100
Number of random search lierations
Number of random search Rerations
(a) Large; Numerical
(b) Large; Heterogeneous
Figure 12: Results on large-sized regression datasets.
Regression, In Figure 12a the leading models remain DOFEN, XGBoost, and CatBoost. DOFEN's proficiency in handling numerical features, further enhanced by the increased data volume, enables it to secure the top position once again. In Figure [2b] DOFEN and Trompt barely maintain their positions within the leading group, yet they still stand out from the other DNN models.
G.2 Detailed Evaluation Results
the main paper, we have discussed the overall performance of DOFEN. To simplify tables, we map dataset names with their OpenML ID, described in Appendix B.3 The evaluation results of each task are organized in Table 33 Please refer to the detailed figures and tables for each task of your interest. The evaluation metrics accuracy for classification tasks and R score for regression tasks, consistent with our main paper. Furthermore, we calculate the mean and standard deviation ranks across datasets provide the rank for each model in the tables.
Table 33: Tables and figures for each task.
Task
Feature
Figure
Table
numerical heterogencous
Figure
13
Tables 34
and
35
medium-sizze sized classification
Figure 14
Tabl 36
medium
medium-sized sized regression
numerical
Figure 15
Tables
37
and
38
heterogeneous
Figure T6
Tables 39
numerical
Figure 17
Table
41
large-sized classification
heterogeneous
Figure
T8
Table
numerical
Figure
19
Table
43
large-sized regression
heterogeneous
Figure
Table
28

---

bank maakenark
Bioresponse
california
covertype
0.80
0.84
0.90
0.805
0.78
0.89
0.82
0.800
0.795
0.76
0.88
0.80
0.790
0.74
0.87
0.78
10
100
10
100
10
100
10
100
credit
bull-of-creil-card--
Diabeles13OUS
electricity
0.78
0.72
0.60
0.86
(on valid set up to this iteration
0.77
0.71
Test accuracy of best model
0.84
0.58
0.76
0.70
0.57
0.82
0.75
0.69
0.56
10
100
10
100
10
100
10
100
eye_ movements
heloc
Higgs
house 16H
0.890
0.650
0.71
0.72
0.885
0.625
0.70
0.71
0.880
0.600
0.70
0.68
0.875
0.575
10
100
100
10
100
100
jannis
MagicTelescope
MiniBooNE
0.87
0.940
0.77
0.935
0.97
0.86
0.76
0.930
0.95
0.75
0.85
0.925
0.93
0.920
100
100
Number of random search iterations
CatBoost DOFEN FT-Transformer
GradientB costiggn
LightGBM MLP NODE
RandomForess - Trompt
GRANDE HistGradientooostinggee
ResNet SAINT
XG Boost
Figure 13: Results on each medium-sized classification datasets with only numerical features.
29

---

albert
compas twC -yars
covertype
cult-of-credit-ee carditt-arr--tte clie
0.66
0.88
0.720
0.86
0.715
(on valid set) up this iteration
Test Test accuracy of bestmodel
0.64
0.84
0.710
0.63
0.64
0.82
0.705
0.80
0.700
0.62
0.62
10
100
100
electricity
eye_ movements
road-safety
0.650
0.77
0.85
0.625
0.76
0.83
0.600
0.75
1
10
100
10
100
100
Number of random search iterations
CatBoost
GradientB Tree
LightG BM MLP
RandomF orest
Trompt XGBoost
DOFEN
GRANDE
ReeNeet SAINT
FT-Transfommer
HistGradientBBosttigTeee
NODE
Figure 14: Results each medium-sized classification datasets with heterogeneous features.
30

---

abalone
Allerons
Bike_ Sharing_ Demand
Brazilian jousess
0.6
1.00
0.75
0.4
0.75
0.50
0.4
0.2
0.25
0.2
0.25
0.0
0.00
0.0
0.00
1
10
100
10
100
10
100
10
100
cpu_act
delays_zurich zuriccurccchhhhh transport
diamands
elevators
1.00
0.03
0.9
0.75
0.75
0.50
0.02
0.50
0.8
0.25
0.01
0.25
0.7
0.00
0.00
0.00
(on valid set) up to this iteration
T2st score of best model
1
10
100
10
100
1
10
100
10
100
house, 16H
house sales
houses
medical charges
0.6
1.00
0.75
0.50
0.50
0.50
0.2
0.25
0.25
0.25
0.0
0.00
10
100
10
10
100
MiamiHousing2016
(yc gyc-tax-ggreen meexxx-grree-dee-220
pol
sulfur
1.00
0.90
0.75
0.5
0.75
0.85
0.50
0.4
0.50
0.80
0.3
0.2
0.00
0.00
10
100
10
100
10
100
10
100
superconduct
wine quality
yprop
0.5
0.75
0.4
0.075
0.50
0.3
0.050
0.25
0.1
0.025
0.00
0.0
0.000
1
10
100
10
100
1
10
100
Number of random search iterations
CatBoost
GradientB oostingree GRANDE oosttraee
LightG MLP NODE
RandomF orest Trompt ResNet SAINT
-DOFEN FT-Transformer
XGBoost
Figure 15: Results on each medium-sized regression datasets with numerical features.
31

---

abalone
A Airlnes__ DepDelay 1M
Allstate, Claims_ Severity
analcoldato supreme
0.6
0.05
0.98
0.04
0.4
0.03
0.4
0.96
0.02
0.2
0.94
0.0
0.00
0.0
10
100
100
100
10
100
Bike Sharing__Deeee Demand
crazsiianhhouses
Pelays_zurich, transport
diamonds
1.00
0.08
1.00
0.75
0.06
0.75
0.50
0.50
0.04
0.50
0.25
0.25
0.02
0.25
0.00
0.00
0.00
0.00
(on valid set) up to this iteration
T2 score of best model
1
10
100
10
100
10
100
10
100
house sales
medical, charges
15. Benz Greener Manut
tyc we-ttai--geee---ae -ew--reeee-dd--000
1.00
0.6
0.6
0.75
0.50
0.50
0.4
0.4
0.25
0.25
0.2
0.2
0.00
0.00
100
100
icculte-matter--kkaii
seattlecrime6
4M_ [er_GPU_kemmen ker_kkeeee__eeee_ pprroom_ppeoom
topo_2. 2_1
1.00
0.6
0.15
0.75
0.06
0.4
0.10
0.50
0.04
0.2
0.05
0.25
0.02
0.0
0.00
0.00
0.00
10
100
100
100
10
100
visualizing soil
1.00
0.50
0.25
0.00
1
10
100
Number of random search iterations
CatBoost
GradientB Dostingrre
LightG MLP ResNet SAINT
RandomF orest Trompt
DDFENN -TT-Trnsformer
GRANDE
XGBoost
oostinggeee
NODE
Figure 16: Results on each medium-sisee -ized regression datasets with heterogeneous features.
32

---

(on valid set up this iteration
Test accuracy of best best model
covertype
Higgs
jannis
MiniBcoNE
0.95
0.73
0.79
0.72
0.94
0.84
0.77
0.71
0.93
0.80
0.75
0.70
10
100
100
100
10
100
Number random search iterations
CatBoost
GradientBoostingTTes GRANDE LightGBM
NODE RandomForess ResNet
SAINT Trompt XG XGBoost
DOFEN FT-Transformer
Figure 17: Results on each large-sized classification datasets with only numerical features.
(on valid set up to this iteration
Test accuracy best model
covertype
road-safety
0.80
0.90
0.79
0.78
0.85
0.77
0.76
0.80
0.75
100
Number random search iterations
CatBoos:
GrndeniB.oosingieee
NODE
SAINT Tromet
DOFEX
GRANDE LightGSM
Randomf crest Restet
FT Transfomer
XGBoost
Figure 18: Results on each large-sized classification datasets with heterogeneous features.
score of best model (on valid set) up to this iteration
diamonds
nyc-taxi-green-dec-
year
0.950
0.6
0.30
0.925
0.900
0.4
0.25
0.875
0.20
0.2
R2 R2
0.850
0.15
Test
0.825
1
10
100
10
100
Number of random search iterations
CatB oost
GradientB GRANDE
Trompt NODE ResNet XGBoos
DOFEN
FT-Transformer
LightG BM
SAINT
Figure 19: Results on each large-sized regression datasets with numerical features.
33

---

black_ friday
diamonds
gee-eii--gee---eee -ec-
ticulate mattee--tte------e -ukttrrr-kkki-2222
0.975
0.70
(on valid set) up to this iteration
0.5
0.950
0.4
0.68
Test R2 score of best
0.925
0.2
0.66
0.64
0.0
100
100
100
4M 4M_GPU _ernell perform
1.00
0.96
0.94
0.92
1
10
Number of random search iterations
CatBoost DOFEN
GradientB oosting
NODE ResNet XGBoost SAINT
Trompt
G GRANDE LightG BM
FT-Transformer
Figure 20: Results each large-sized regression datasets with heterogeneous features.
Table 34: The performance of medium-sized classification task (numerical features only) (1).
361276
361273
361069
361065
361068
361066
361277
361061
361055
361275
Default
DOFEN (ours)
0.7839
0.6016
0.7113
0.8662
0.9369
0.8030
0.8827
0.7901
0.7732
0.7151
Trompt
0.7831
0.5823
0.6926
0.8630
0.9382
0.7936
0.8909
0.8268
0.7584
0.6994
GRANDE
0.7776
0.6023
0.7099
0.8586
0,9334
0.8039
0.8845
0.7880
0.7796
0.7206
FT Fr-Traasformer
0.7463
0.6025
0.7031
0.8553
0,9320
0.7958
0,8846
0.7944
0.7745
0.7137
ResNet
0.7424
0.6029
0.6755
0.8548
0.9345
0.7864
0.8641
0.7820
0.7706
0.7093
MLP
0.7277
0.6033
0.6752
0.8520
0.9307
0.7886
0.8661
0.7727
0.7710
0.7077
SAINT
0.7537
0.6044
0.6967
0.8534
0.9348
0.7891
0.8791
0.7775
0.7741
0.7133
NODE
0.7360
0.6039
0.7060
0.8581
0.9363
0.7957
0.8763
0.8108
0.7750
0.7169
CatBoost
0.7881
0.6001
0.7130
0.8614
0,9364
0.8045
0.9021
0.8016
0.7695
0.7129
LightGBM
0.7878
0.5934
0.7079
0.8547
0,9316
0.8033
0.9006
0.7950
0.7717
0.7109
XGBoost
0.7831
0.5850
0.6925
0.8531
0,9329
0.7981
0.9030
0.7987
0.7591
0.6974
HisGGradienBoostingg
0.7909
0.5619
0.7018
0.8647
0.9364
0.7880
0.9007
0.8193
0.7490
0.6884
GradientBoosting Treee
0.7657
0.6018
0.7048
0.8444
0,9216
0.8027
0.8800
0.7685
0.7752
0.7184
RandomForee
0.7859
0.5579
0.6998
0.8514
0.9208
0.7958
0.8876
0.8124
0.7635
0.7029
Searched
DOFEN (ours)
0.7992
0.6043
0.7160
0.8715
0.9404
0.8017
0.8958
0.8162
0.7747
0.7220
Trompt
0.7831
0.6032
0.7090
0.8635
0,9374
0.7930
0.8913
0.8373
0.7760
0.7217
GRANDE
0.7795
0.6042
0.7125
0.8620
0,9368
0.8005
0.8885
0.8021
0.7759
0.7150
Fr-Traasformer
0.7566
0.6044
0.7042
0.8582
0.9350
0.7929
0.8865
0.8048
0.7754
0.7145
ResNet
0.7652
0.6046
0.6931
0.8565
0.9357
0.7866
0.8783
0.7942
0.7692
0.7054
MLP
0.7658
0.6033
0.6855
0.8535
0.9346
0.7888
0.8676
0.7834
0.7686
0.7067
SAINT
0.7629
0.6044
0.7063
0.8478
0.9355
0.7912
0,8870
0.8041
0.7613
0.7155
NODE
0.7596
0.6043
0.7084
0.8556
0,9375
0.7952
0.8808
0.8194
0.7753
0.7169
CatBoost
0.7898
0.6052
0.7144
0.8599
0.9382
0.8047
0.9003
0.8299
0.7768
0.7201
LightGBM
0.7942
0.6050
0.7105
0.8572
0.9378
0.8011
0.9017
0.8188
0.7715
0.7196
XGBoost
0.7917
0.6057
0.7138
0.8606
0,9369
0.8031
0,9016
0.8176
0.7732
0.7156
BistGrradeenBoootingggi
0.7859
0.6050
0.7092
0,8592
0,9372
0.8108
0.8932
0.8231
0.7772
0.7194
BondientBoostingTree
0.7694
0.6044
0.7100
0.8557
0,9331
0.8015
0.8974
0.8182
0.7728
0.7172
MandomForess
0.7936
0.6047
0.7053
0.8541
0.9269
0.7985
0.8924
0.8275
0.7727
0.7182
34

---

Table 35: The performance of medium-sized classification task (numerical featurre only) (2).
361060
361070
361278
361063
361274
361062
Ranking
Default
DOFEN (ours)
0,8169
0.6196
0.7189
0.8895
0.7806
0.9822
4.81 + 3.42
Trompt
0.8289
0.6160
0.6987
0.8805
0.7689
0.9849
7.41 + 4.22
GRANDE
0.8131
0.6010
0.7210
0.8890
0.7788
0.9783
5.38 +.6
FT-Transformer
0.8082
0.5864
0.7175
0.8816
0.7562
0.9780
8.19 2.62
ResNet
0,8062
0.5852
0.7186
0.8755
0.7449
0.9366
10.75 3.87
MLP
0,8105
0.5808
0.7151
0.8765
0.7418
0.9153
11.56 1: 3.82
SAINT
0.8098
0.5799
0.7146
0.8842
0.7668
0.9718
8.88 1 3.56
NODE
0.8151
0.5931
0.7271
0.8823
0.7651
0.9701
6.75 3.62
CatBoost
0.8448
0.6387
0.7222
0.8859
0.7785
0,9846
3.91 3.55
LightGBM
0.8434
0.6439
0.7148
0.8843
0.7727
0,9838
6.06
XGBoost
0.8611
0.6475
0.6948
0.8816
0.7600
0.9835
8.12 4.05
BoosGrag
0.8623
0.6633
0.7024
0.8848
0.7721
0.9846
6.22 5.10
GradientBoosting
0.8216
0.6233
0.7157
0.8767
0.7618
0.9671
8.50 3.88
RandomFores
0.8458
0.6308
0.7173
0.8782
0.7611
0.9803
8.47 3.69
Searched
DOFEN (ours)
0,8257
0.6323
0.7281
0.8898
0.7791
0.9802
4.41
Trompt
0.8307
0.6271
0.7263
0.8846
0.7782
0.9838
5.69
GR BRANDE
0.8217
0.6050
0.7217
0.8881
0.7805
0.9792
7.31
PT-Transformer
0.8191
0.5786
0.7194
0.8791
0.7675
0.9836
10.06
3.229
ResNet
0.8097
0.5801
0.7189
0.8734
0.7509
0.9511
11.88 1 3.266
MLP
0.8048
0.5823
0.7187
0.8773
0.7444
0.9474
12.94 3.11
SAINT
0,8188
0.5859
0.7194
0.8835
0.7709
0.9803
10.44 1: 2.62
NODE
0,8175
0,5895
0.7257
0.8813
0.7694
0.9693
9.09 3.14
CatBoost
0,8627
0.6532
0.7230
0.8855
0,7802
0.9846
2.88 2.96
LightGBM
0,8594
0.6526
0.7217
0.8869
0.7777
0.9819
5.00
XGBoost
0.8687
0.6615
0.7171
0.8882
0.7790
0.9815
4
0.8625
0.6578
0.7203
0.8849
0.7739
0.9835
4.69
GradientBoostingTThe
0.8653
0.6343
0.7179
0.8817
0.7735
0.9813
Random Forest
0.8608
0.6506
0.7164
0.8798
0.7724
0.9812
3.51
35

---

Table 36: The performance of medium-sized classification task (heterogeneous features).
361282
361286
361113
361283
361110
361111
361285
Ranking
Default
DOFEN (ours)
0.6495
0.6823
0,8240
0.7162
0.8275
0.6241
0.7730
6.00 + 4.17
Trompt
0.6191
0.6743
0.8729
0.7017
0.8450
0.6425
0.7580
8.43 4.43
GRANDE
0.6549
0.6759
0.8278
0.7208
0.8266
0.6208
0.7659
+.43
FT-Transformer
0.6543
0.6820
0.8565
0.7156
0.8252
0.5952
0.7635
5.71 3.93
ResNet
0.6459
0.6756
0.8214
0.7055
0.8200
0.5883
0.7517
4.399
MLP
0.6506
0.6826
0.8259
0.7078
0.8161
0.5939
0.7486
9.43 1 5.21
SAINT
0.6501
0,6750
0,8261
0.7059
0.8234
0.5958
0.7618
8.86 1 2.70
NODE
0.6497
0.6753
0.8397
0.7146
0.8172
0.5895
0.7597
CatBoost
0.6570
0.6715
0.8369
0.7120
0.8501
0.6462
0.7680
4.60
LightGBM
0.6489
0.6747
0.8323
0.7123
0.8637
0.6448
0.7643
6.29 2.92
XGBoost
0.6315
0.6632
0.8413
0.6969
0.8786
0.6477
0.7594
7.71 + 5.24
HistGra Tree
0.6500
0.6625
0.8334
0.7090
0.8685
0.6446
0.7647
6.43 + 3.45
GradientBoostinggTre
0.6559
0.6798
0.7892
0.7187
0.8293
0.6181
0.7501
7.29 + 4.60
RandomForest
0.6482
0.6219
0.8471
0.7052
0.8629
0.6400
0.7531
8.71 + 4.32
Searched
DOFEN (ours)
0.6581
0.6763
0.8618
0.7139
0.8515
0,6377
0.7760
6.14 3.96
Trompt
0.6431
0.6801
0.8829
0.7159
0.8513
0.6429
0.7709
6.29
GRANDE
0.6596
0.6534
0.8425
0.7186
0.8393
0.6060
0.7670
7.79
FT- TT-Transformer
0.6529
0.6819
0.8594
0.7118
0.8315
0.5865
0.7715
7.71
ResNet
0,6510
0.6843
0,8380
0.7028
0.8258
0.5937
0.7621
1.663-14.63
MLP
0.6524
0.6788
0.8339
0.7108
0.8242
0.5888
0.7556
12.00 1..00:4.55
SAINT
0.6525
0.6724
0.8483
0.7142
0.8294
0.5862
0.7647
10.71
NODE
0.6498
0.6766
0.8397
0.7146
0.8205
0.5927
0.7580
1..7
CatBoost
0.6596
0.6775
0,8745
0.7226
0.8773
0.6655
0.7714
3.29 + 4.63
LightGBM
0.6574
0.6747
0.8647
0.7209
0.8864
0.6596
0.7643
4.86
XGBoost
0.6561
0.6798
0.8596
0.7183
0.8861
0.6673
0.7679
4.14 3.58
HistGraatennBoossinggTeeee
0.6563
0.6786
0.8467
0.7169
0.8785
0.6327
0.7661
6.71 +
GradientBoostingTTeee
0.6538
0.6805
0.8545
0.7186
0.8780
0.6355
0.7625
6.21 + 2.84
RandomForest
0.6542
0.6795
0,8587
0.7167
0.8773
0.6575
0.7597
7.00 2.27
36

---

Table 37: The performance of medium-sized regression task (numerical features only) (1).
361077 361082
361081
361087
361280
361072
361281
361080
361074 361079
Default
DOFEN (ours)
0.2011
0.6874
0.9931
0.9194
0,5651
0.9837
0.0269
0.9352
0.8921
0.5398
Trompt
0.8480
0.6829
0,9970
0.9275
0.5443
0.9723
0.0162
0.9415
0.8969
0.5453
GRANDE
0.7399
0.6376
0.9126
0.8340
0.5281
0.8812
0.0053
0.8239
0.6423
0.6173
FT-Transformer
0.8436
0.6691
0.9958
0.9205
0.5308
0.9594
0.0000
0.9419
0.9115
0.5571
ResNet
0,8331
0.6423
0.9923
0.9145
0.4696
0.9747
0.0000
0.9404
0.8979
0.4995
MLP
0.8299
0.6634
0.9939
0.9091
0.5435
0.9570
0.0000
0.9411
0.8958
0.5057
SAINT
0,0000
0.6816
0.9938
0.9158
0.5658
0.9835
0.0178
0.9422
0,8500
0,4679
NODE
0,4500
0.0000
0.0000
0.0000
0.0000
0.0000
0.0083
0.0000
0.8684
0.0000
CatBoost
0.8576
0.6993
0.9960
0.9356
0.5279
0.9856
0.0000
0.9457
0.9117
0.5101
LightGBM
0,8468
0.6928
0.9938
0,9225
0.5124
0,9846
0.0070
0.9449
0.8859
0,5195
XGBoost
0.8258
0.6793
0.9976
0.9203
0,4817
0.9825
0.0000
0.9409
0.8848
0.4814
BesGraddieBBoosiinng Tree
0.8464
0.6932
0.9938
0.9233
0.5259
0.9828
0.0052
0.9448
0.8855
0.5361
GradientBoosting
0.8397
0.6758
0.9962
0.8942
0.5399
0.9835
0.0251
0.9441
0.8022
0.4733
RaneomForest
0.8372
0.6720
0.9931
0,9141
0.5359
0.9826
0.0000
0.9394
0.8330
0.5016
Searched
DOFEN (ours)
0.6458
0.6972
0.9941
0.9351
0.5637
0.9872
0.0282
0.9447
0.9068
0.5507
Trompt
0.8457
0.6915
0.9956
0.9280
0.5443
0.9873
0.0283
0.9427
0.8948
0.4650
GRANDE
0.7449
0.6426
0.9136
0.8352
0.5457
0.8786
0.0070
0.8270
0.6498
0.6165
PT-Transformer
0.8453
0.6805
0.9973
0.9223
0,5630
0.9844
0.0191
0.9435
0.9171
0.4214
ResNet
0,8342
0.3569
0.9969
0.9172
0.5731
0.9822
0.0120
0.9079
0,4781
MLP
0.8367
0.6754
0.9932
0.9092
0.5776
0.9790
0.0146
0.9436
0.9181
0.4830
SAINT
0.7811
0.6858
0.9940
0.9245
0.5629
0.9849
0.0216
0.9442
0,9224
0.4660
NODE
0,8365
0.6704
0.9877
0.9260
0,5332
0.9730
0.0127
0.9427
0.9148
0.5257
CatBoost
0,8553
0.7062
0,9920
0.9377
0.5353
0.9865
0.0306
0.9450
0.9105
0,4586
LightGBM
0,8468
0.6928
0.9928
0.9339
0.5399
0.9811
0.0329
0.9449
0.8859
0.5167
XGBoost
0.8450
0.6943
0.9976
0.9360
0.5449
0.9861
0.0301
0.9456
0.9072
0.5454
HistGraddinn
0,8464
0.6932
0.9928
0.9265
0.5322
0.9745
0.0299
0.9449
0.8863
0.3912
GradientBoosting
0.8416
0.6889
0.9961
0.9249
0.5457
0.9854
0.0262
0.9450
0.8602
0.5158
Random
0.8386
0.6871
0.9931
0.9242
0.5517
0.9829
0.0308
0.9453
0.8410
0,4806
37

---

Table 38: The performance of medium-sized regression task (numerical features only) (2).
361084 361078 361086
361083
361073
361085
361088
361076 361279
Ranking
Default
DOFEN (ours)
0.8723
0.8099
0.9756
0.4427
0,9885
0.8285
0.8950
0.4139
0.0000
6.84 3,68
Trompt
0.8804
0.8352
0,9788
0.1699
0,9513
0.8096
0.8791
0.3168
0.0083
5.68 + 3.30
GRANDE
0.7918
0.6854
0.8822
0.2149
0.9536
0.7400
0.7986
0.2609
0.0000
11.11 3.70
FT-Transformer
0.8766
0.8235
0.9794
0.1499
0.9313
0.8400
0.8751
0.2648
0.0000
7.24 3.77
ResNet
0.7948
0.7729
0.9772
0.2050
0.6279
0.6979
0.8739
0.2598
0.0000
10.71 2.89
MLP
0.8575
0.8133
0.9789
0.1615
0.8343
0.7922
0,8842
0.2792
0.0000
9.08 1 3.01
SAINT
0.8731
0.8139
0.9788
0.4713
0.9904
0.7859
0.8909
0.3632
0.0449
7.00 3.49
NODE
0.0000
0.0000
0.0000
0.3622
0.0000
0.6828
0.0000
0.0000
0.0336
12.53 3.80
CatBoost
0.8873
0.8472
0.9782
0.5291
0.9863
0.8685
0.9051
0.4500
0.0530
3.34
LightGBM
0.8812
0.8351
0.9785
0,5306
0.9870
0.8143
0.8979
0.4286
0.0480
4.74
XGBoost
0.8743
0.8374
0.9773
0.5487
0,9850
0.8349
0.8955
0.4237
0.0000
7.08 + 3.36
BusGraddenBBoosiinng Tree
0.8816
0.8325
0.9785
0.5186
0,9865
0.8161
0.8964
0.4336
0.0522
5.00 2.64
GradientBoostingTTen
0.8617
0.7874
0.9794
0.4516
0.9349
0.8106
0.8563
0.3763
0.0000
7.74 3.62
RandomForess
0.8689
0.8270
0.9768
0.5460
0.9867
0.8439
0.9011
0.4807
0.0601
6.92 3.81
Searched
DOFEN (ours)
0.8824
0.8395
0.9788
0.4950
0.9928
0.8907
0.9091
0.4757
0.0706
5.26 + 3.53
Trompt
0.8832
0.8187
0.9792
0.4550
0.9958
0.8508
0.8949
0.4091
0.0376
7.26 + 3.42
GRANDE
0.7918
0.6877
0.8840
0.2305
0.9541
0.7425
0.8082
0.2654
0.0000
12.74 4.14 4.12
PT-Transformer
0.8824
0.8341
0,9795
0.4926
0.9949
0.8676
0.8870
0.3675
0.0519
7.16 3.62
ResNet
0,8668
0.8236
0.9793
0.4743
0.9606
0.8237
0.8934
0.3666
0.0273
9.50 3.70
MLP
0.8669
0.8202
0.9796
0.4666
0.9708
0.8418
0.8930
0.3949
0.0126
8.84 3.82
SAINT
0,8804
0.8259
0.9794
0.4958
0.9948
0.7603
0.8937
0.3736
0.0579
7.63 2.91
NODE
0,8762
0.7969
0,9782
0.3743
0.9580
0.7309
0.8857
0.2874
0.0393
11.11 3.57
CatBoost
0,8872
0.8487
0,9793
0.5404
0.9908
0.8692
0.9095
0.4996
0.0736
4.42 4.06
LightGBM
0,8863
0.8539
0.9785
0.5306
0,9870
0.8172
0.9048
0.4286
0.0663
6.53 + 3.59
XGBoost
0.8886
0.8495
0.9787
0.5519
0.9909
0.8620
0.9102
0.5006
0.0814
3.74 + 3.57
BootGrig
0.8819
0.8375
0.9791
0.5186
0.9871
0.8203
0.9010
0.4313
0.0612
7.79 3.12
GradientBoosting
0.8828
0.8400
0,9794
0.5531
0.9896
0.8153
0.9030
0.4710
0.0525
6.37
+ 2.64
Random
0.8712
0.8291
0.9789
0.5618
0.9891
0.8585
0.9087
0.5044
0.0937
6.42 1: 3.65
38

---

Table 39: The performance of medium-sized regression task (heterogeneous features) (1).
361293 361292
361099
361098
361097
361104
361288
361093
361291 361096
Default
DOFEN (ours)
0,0359
0.5258
0.9341
0.9932
0,5750
0.9997
0,5686
0.9837
0.0694
0.9869
Trompt
0.0195
0.4939
0,9393
0.9963
0.5409
0.9996
0.5459
0.9470
0.0364
0.9888
GRANDE
0.0000
0.4356
0.8014
0.9101
0,4853
0.9151
0.5354
0.9723
0.0231
0.8957
FT-Transformer
0.5160
0.9280
0.9960
0.5540
0.9997
0.5480
0.9490
0.0000
0.9872
ResNet
0,0000
0.4993
0.8861
0,9883
0.5470
0.9975
0.4229
0.9244
0.0000
0.9857
MLP
0.0000
0,5105
0.9213
0.9942
0.5546
0.9998
0.5486
0.9497
0.0000
0.9861
SAINT
0.0375
0.5191
0.9375
0.9930
0.5522
0.9990
0.5676
0.9777
0.0591
0.9867
NODE
0.0361
0.0000
0.0000
0.0000
0.0000
0.0000
0.0000
0.9797
0.0566
0.0000
CatBoost
0.0156
0.5347
0.9421
0,9959
0.5633
0.9997
0.5375
0.9801
0.0621
0.9911
LightGBM
0,0234
0,5275
0,9402
0.9938
0.5477
0.9997
0,5183
0.9823
0.0618
0.9901
XGBoost
0.0000
0.4807
0.9393
0.9976
0.4968
0.9997
0.4797
0.9759
0.0000
0.9896
BesGraddieBBoosiinng Tree
0.0290
0.5226
0,9410
0.9939
0.5421
0.9997
0.5289
0.9823
0.0557
0.9908
GradientBoostingTTen
0.0451
0.5066
0.8415
0.9962
0.5717
0.9997
0.5470
0.9827
0.0736
0.9842
RandomForess
0.0000
0.4748
0.9369
0.9929
0,5034
0.9998
0.5407
0.9799
0.0000
0.9878
Searched
DOFEN (ours)
0.0440
0.5351
0.9453
0.9946
0.5755
0.9998
0.5641
0.9846
0.0738
0.9903
Trompt
0.0457
0.5330
0.9397
0.9957
0.5816
0.9997
0.5465
0.9848
0.0744
0.9899
GRANDE
0.0000
0.4386
0.8041
0.9137
0.4947
0.9166
0,5010
0.9725
0.0340
0.8957
PT-Transformer
0.0455
0.5217
0,9390
0.9979
0,5662
0.9998
0.5649
0.9797
0.0680
0.9899
ResNet
0.0382
0.5100
0.9361
0.9962
0.5685
0.9996
0.5712
0.9797
0.0599
nan
MLP
0.0413
0.5157
0.9337
0.9948
0.5572
0.9998
0.5775
0.9801
0.0622
0.9876
SAINT
0,0450
0.5253
0.9409
0.9958
0.5618
0.9997
0.5678
0.9789
0.0656
0.9893
NODE
0.0369
0,5169
0.8428
0.9878
0,5735
0.9998
0,5409
0.9823
0.0580
0.9860
CatBoost
0.0479
0.5350
0,9468
0.9921
0,5815
0.9997
0.5449
0.9847
0.0778
0.9917
LightGBM
0.0431
0.5336
0.9439
0.9941
0.5549
0.9997
0.5370
0.9823
0.0789
0.9907
XGBoost
0.0473
0.5352
0.9466
0.9976
0.5771
0.9998
0.5558
0.9832
0.0775
0.9911
HistGraddienBoossinng
0.0468
0.5267
0.9416
0.9931
0.5772
0.9997
0.5441
0.9814
0.0731
0.9909
GradientBoosting
0.0474
0.5298
0,9416
0.9956
0.5765
0.9998
0.5490
0.9816
0.0746
0.9898
RandomFord
0.0456
0.5004
0.9369
0.9934
0.5755
0.9998
0.5564
0.9807
0.0766
0.9881
39

---

Table 40: The performance of medium-sized regression task ( (2).
361102 361294
361101
361103
361289
361287
361094
Ranking
Default
DOFEN (ours)
0.8838
0.9756
0.4178
0.6647
0.1835
0.0329
0.9996
5.47 + 3.39
Trompt
0.8902
0.9782
0.0961
0.6494
0.1817
0.0445
0.9995
7.29 + 3.71
GRANDE
0.7977
0.8822
0.2421
0.6217
0.0992
0.0000
0.9615
11.88 + 2.95
FT-Transformer
0,8883
0.2472
0.6710
0.0200
0.0000
0.9998
8.00 3.17
ResNet
0.8736
0.9782
0.2434
0.6487
0.0305
0.0000
0.9958
11.00 1..00:2..0
MLP
0,8751
0.9792
0.1580
0.6555
0.0321
0.0000
0.9999
8.06 1 3.79
SAINT
0.8836
0.9777
0,4631
0.6602
0.1712
0.0401
0.9998
6,41 1:2.2
NODE
0.0000
0.0000
0.3719
0.0000
0.0000
0.0370
0.0000
11.47
CatBoost
0.8975
0.9776
0.5463
0.6916
0.1843
0.0313
0.9999
3.76
LightGBM
0.8905
0.9779
0.5448
0.6874
0.1792
0.0199
0.9999
5.18
XGBoost
0.8834
0.9773
0.5699
0.6619
0.1653
0.0000
1.0000
7.47 + 3.94
HistGraa Tree
0.8914
0.9785
0.5389
0.6904
0.1727
0.0302
0.9999
5.00 3.11
GradientBoostingTree
0.8693
0.9794
0,4694
0.6717
0.1861
0.0305
0.9994
5.35 4.27
RandomForest
0.8747
0.9767
0,5619
0.6551
0.1639
0.0233
1.0000
7.94 3.49
Searched
DOFEN (ours)
0.8908
0.9788
0.5102
0.6647
0.1829
0.0531
0.9996
6.65 2.51
Trompt
0.8916
0.9787
0.4804
0.6690
0.1821
0.0503
0.9999
7.00
GRANDE
0.7943
0.8837
0.2621
0.6290
0.0956
0.0000
0.9622
13.88 3.05
FT- TT-Transformer
0.8930
0.9796
0,5281
0.6731
0.1797
0.0496
0.9999
+.28
ResNet
0,8846
0.9793
0.4483
0.6565
0.1798
0.0219
0.9977
10.12
MLP
0.8849
0.9796
0.4737
0.6590
0.1716
0.0256
0.9999
8.94 4.13
SAINT
0.8913
0.9796
0.5195
0.6706
0.1820
0.0525
0.9999
1 2.91
NODE
0,8842
0.9782
0.4972
0.6477
0.0477
0.0423
0.9984
11.00
CatBoost
0.8941
0.9787
0.5665
0.6930
0.1854
0.0574
1.0000
4.18 4.04
LightGBM
0.8935
0.9781
0.5448
0.6874
0.1868
0.0546
1.0000
6.41
XGBoost
0.8965
0.9788
0.5820
0.6909
0.1850
0.0644
1.0000
3.12
HistGraateenBooosinggTeeer
0.8910
0.9791
0.5383
0.6904
0.1857
0.0623
1.0000
6.12
GradientBoostingTTeee
0.8901
0.9794
0.5773
0.6841
0.1858
0.0305
1.0000
5.41 + 3.00
RandomForest
0.8749
0.9787
0.5838
0.6744
0.1827
0.0701
1.0000
6.82 3.77
40

---

Table 41: The performance of large-sized classification task (numerical features only).
361069
361068 361061
361274
Ranking
Default
DOFEN (ours)
0.7306
0.9480
0.8222
0.8018
3.25 + 5.22
Trompt
0.7213
0.9468
0.9004
0.7954
2.88 + 4.35
GRANDE
0.7248
0.9425
0.8315
0.7979
5.00 3.05
FT-Transformer
0.6960
0.9403
0.8983
0.7586
8.25 4.16
ResNet
0.6988
0.9409
0.8801
0.7358
8.50 + 4.04
MLP
nan
nan
nan
nan
nan 1 nan
SAINT
0.7181
0,9436
0.8694
0.7860
6.00 1 1.30
NODE
0.7247
0.9461
0.8886
0.7946
3.75 + 2.92
CatBoost
0.7261
0.9432
0.8377
0.7954
4.38 + 2.77
LightGBM
0.7212
0,9371
0.8071
0.7870
8.00 + 2.30
XGBoost
0.7164
0.9367
0.8361
0.7828
8.50 + 2.61
HistGraadiemBoootinggTee Tree
nan
+ man 11.00 + 4.58
GradientBoosting Tree
0.7103
0.9225
0.7698
0.7718
RandomForest
0.7158
0.9308
0.8767
0.7797
8.50 + 3.29
Searched
DOFEN (ours)
0.7340
0.9479
0.8888
0.8033
3.50 +: 5.76
Trompt
0.7286
0.9436
0.9127
0.7988
4.00 + 4.09
GRANDE
0.7265
0.9454
0.8522
0.7980
6.00 + 3.70
FT- FT-Transformer
0.7299
0,9441
0.9062
0.7962
4.00 + 3.35
ResNet
0.7228
0.9446
0.8935
0.7781
9.25 4.49
MLP
1 man
SAINT
0.7255
0.9440
0.8956
0.7922
8.50 1 2.11
NODE
0.7262
0.9471
0.8982
0.7946
5.25 + 2.39
CatBoost
0.7299
0,9445
0.9015
0.7975
4.00 + 2.77
LightGBM
0.7251
0.9433
0.8964
0.7938
8.50 + 2.11
XGBoost
0.7279
0.9439
0.8956
0.7965
6.50 + 1.52
HissGraddentBoostinggne
nan
nan
nan
nan
nan + nan 9.00 + 3.27
GradientBoosting Tree
0.7247
0.9400
0.7938 0.8978
RandomForest
0.7198
0.9353
0.9059
0.7885
9.50 + 5.36
41

---

Table 42: The performance of large-sized classification task (
361113 361285
Ranking
Default
DOFEN (ours)
0.8691
0.7870
5,00 5.11
Trompt
0,9276
0.7836
2.00 + 5.77
GRANDE
0.8568
0.7785
8.00 3.51
FT-Transformer
0,9317
0.7609
6.00 + 5.03
ResNet
0.8945
0.7653
8.00 3.51
MLP
nan
nan
nan 1 nan
SAINT
0,9123
0.7731
6.50 2.57
NODE
0,9199
0.7774
5.00
CatBoost
0.8827
0.7821
5.50 2.29
LightGBM
0.8476
0.7797
8.00 4.16
XGBoost
0.8781
0.7822
5.50 3.04
HistGradiemtBoostinng Tree
nam + nan
GradientBoottiggTTee
0.7946
0.7519
12.00 6.35
RandomForest
0.9066
0.7767
6,50 1.61
Searched
DOFEN (ours)
0.9116
0.7979
7.00 + 4.16
Trompt
0.9395
0.7844
4.00 + 4.80
GRANDE
0.8914
0.7771
12.00 - 6.35
FT-Transformer
0,9348
0.7890
4.00 + 3.88
ResNet
0.9226
0.7834
8.50 2.65
MLP
nan 1
SAINT
0,9252
0.7796
9.50
NODE
0,9219
0.7800
10.00 = 4.62
CatBoost
0,9368
0.8012
1.50 + 6.08
LightGBM
0,9310
0.7977
4.50 + 2.36
XGBoost
0.9294
0.7987
4.50 + 3.40
HistGradieenBoostinn Tree
nan
nan
nan nan 6.00 + 0.58
GradientBoosting
0,9302
0.7862
RandomForest
0.9327
0.7813
6.50 + 2.52
42

---

Table 43: The performance of large-sized regression task (numerical features only).
361080
361083
361091
Ranking
Default
DOFEN (ours)
0,9469
0.5459
0.3240
3,00 1:00:::0 3.54
Trompt
0.9458
0.3379
0.2498
7.00 1.00++10 1.29
GR GRANDE
0.8272
0.3243
0.2031
9.33 3.40
FT-Transformer
0.9452
0.1198
0.1172
9.67 + 4.27
ResNet
0,9410
0.2469
0.1188
9.67
MLP
nan
nan
nan
nan 1 man
SAINT
0,9445
0.5344
0.2887
5.67 2.53
NODE
0.9453
0.0000
0.2763
7.67 3.10
CatBoost
0,9476
0.5847
0.3020
1.67 4.69
LightGBM
0,9475
0,5607
0.2810
3.335
XGBoost
0.9474
0.6087
0.2512
3.67 + 3.30
BostGra
nan
nan
nan + Tan
Gradientooosting
0.9459
0.2574 0.4635
5.67 0.63
RandomForest
nan nan
nan
non nan
Searched
DOFEN (ours)
0,9490
0.5640
0.3321
2.00 4.72
Trompt
0.9461
0.5242
0.2971
8.33 2.59
GRANDE
0,8289
0.3707
0.2078
11.00 +
FT Fr-Transformer
0.9463
0.5382
0.3049
6.67 167+111 1.50
ResNet
0.9465
0.5277
0.2770
8.33 2.72
MLP
THnn
SAINT
0,9465
0.5491
0.3053
5.33 1.26
NODE
0,9460
0,3967
0.2892
9.779
CatBoost
0,9480
0.5996
0.3130
2.33 3.61
LightGBM
0.9475
0.5607
0.2810
=.10
XGBoost
0.9480
0.6249
0.3070
2.33 3.71
Brostingn
nan
nan
nan
nan nan
GradiennBoostingTTre
0.9471
0.3046 0.6157
4.33 + 2.50
RandomForest
nan
nan
nan
Tann 1 TAn
43

---

Table 44: The performance of large-sized regression task (heterogeneous features).
361104
361095
361096
361101
361103
Ranking
Default
DOFEN (ours)
0.9998
0.6120
0.9923
0.5288
0.6823
4.00 + 2.48
Trompt
0.9996
0.6097
0.9917
0.4035
0.7048
6.20 + 1.86
GRANDE
0.9153
0.4275
0.9011
0.3544
0.6370
10.00 3.99
FT-Transformer
0.9994
0.3514
0.9923
0.4061
0.6761
7.40 2.79
ResNet
0.9895
0.3370
0.9816
0.3971
0.6660
9.80 3.27
MLP
nan
nan
nan
nan
nan
nan + mam
SAINT
0,9997
0.3891
0.9918
0.5480
0.6874
5.80 1 1.79
NODE
0,9997
0,4156
0.9875
0.0198
0.6626
8.80 = 2.79
CatBoost
0,9998
0.6332
0.9928
0.6050
0.7068
2.00 3.44
LightGBM
0.9998
0.6324
0,9916
0,5769
0.7037
4.00 - 2.40
XGBoost
0.9998
0.6345
0.9922
0.6244
0.7060
1.80 + 3.93
HisGGradienBoostting Tree
nan
nan
nan
+
Gradien|Boosting
0.9998
0.6165 0.9857
0.4809
0.6773
6.20 1.74
RandomForest
nan
nan nan
nan
nan
nan nan
Searched
DOFEN (ours)
0.9998
0,6242
0.9935
0.5855
0.6923
4.40 + 2.94
Trompt
0.9998
0.6286
0.9918
0.5479
0.7073
7.00 100022. 2.34
GRANDE
0.9158
0.4350
0.9010
0.3899
0.6421
10.40 = 4.02
FT-Transformer
0,9998
0.3891
0,9924
0.5708
0.7045
7.60 = 2.83
ResNet
0.9998
0.3937
0.9923
0.5336
0.6864
7.80 1 2.90
MLP
nan
nan +
SAINT
0.9998
0.3952
0.9926
0.5659
0.6974
6.40 1.75
NODE
0.9998
0,5908
0.9918
0.4135
0.6668
8.20 2.99
CatBoost
0,9998
0,6363
0,9932
0.6263
0.7116
3.20 = 3.39
LightGBM
0.9998
0.6324
0.9924
0.5769
0.7096
5.20 = 2.35
XGBoost
0.9998
0.6383
0.9932
0.6479
0.7122
1.40 4.00
HistGradientBoosting Tree
nan
nan 0.6301
nan
nan
nan
nan + nan
GradientBoosting Tree
0.9998
0.9918
0.6361
0.7057 nan
4.40 + 3.07
RandomFores
nan
nan nan
nan
YAU + nan
H
More Experiment Settings
H.1 Hardware Used
The following hardware configuration was used for all of our experiments. The hardware selection was based on availability, with neural networks consistently run on GPUs and tree-based models executed on CPUs.
GPUs: NVIDIA GeForce RTX 2080 Ti, NVIDIA DGXI, NVIDIA A100
CPUs: Intel(R) Xeo Silver 4210 CPU, Intel(R) Xeon(R) CPU ES-2698 v4, AMD EPYC605 7742 64-core Processor
H.2 Hyperparameter Search Space
This section details the hyperparameter search space adopted for each model, as referenced in various tables (Tables 45 to 56). We have employed search spaces consistent with those presented in the Tabular Benchmark I for models including XGBoost, GradientBoostinggree RandomForest, FT- FT-rranfformer, SAINT, ResNet, and MLP.
Additionally, we have defined specific search spaces for newer baselines such as CatBoost, LightGBM, Trompt, NODE, and GRANDE. For CatBoost, our search space aligns with the parameters specified by the FT-Transformer study [I]] In the case of LightGBM, we have derived the search space based on recommendations from field practitioners, as cited in (34 K5). For NODE, our approach follows the guidelines provided in TabZilla [15. For GRANDE, we follow the settings provided in the notebook example from the official github of GRANDE.
44

---

Table 46: Hyperparameter search space of XGBoost.
Hyperparameter
Distribution
max _ex_hepth
uniform uniform_nn(11, 11]
num cen_estimators
1000
min child weight
log sg__uiformm int[1, 1c2
subsample learning rate col jol_sample_byleee beveevel
unifrom(0.5, 1]
log unifrom [e-1, 0.7]
uniform 00.5, 1]
col hol_sampe__by_tee gamma lambda
uniform(0.5.
log_uniform 1e- 7]
log uniform []
alpha
log uniformm 8, 1e2
Table 47: Hyperparameter search space of CatBoost.
Hyperparameter
Distribution
max dep_h learning rate
3, 4, 6, 8, 9, 10]
log. log_ sr__unfform/ [e- 1]
iterations
quantile_
uniformmo quantile_uniformi
100, 6000]
temperature
uniform (0. 1]
12 12_leaf_reg __rg lcaf essimatioonnitteaioonn
log_ wni.ormfoomfff 10
iteration
21, 6, 7, 7, 9, 10]
In the context of our model, DOFEN, we have focused our search on the number of m and d. which relate the varied number of Noond and the conditions per rODT. Additionally, drop rate parameter to fine-tune the degree of regularization in our model. It is important to note that the overall search space for DOFEN relatively compact when compared the other baseline models while achieve competitive performance.
have explored the
Table 45: Hyperparameter search space of DOFEN.
Hyperparameter Distribution
d
[3, 6,
[16, 64]
drop_rate
[0.0, 0.1,
Table 48: Hyperparameter search space of LightGBM.
Hyperparameter
Distribution
leaming_rate
uniform (0.001, 1]
max _e__eeephh bagging fraction
2, 4,5, 6, 7, 8, 9, 10, 11] uniform 1.0]
bagging_frequency bagging frequency
[1, 3, 5,
num num_leawe leaves
quantile uniform (30, 150]
cnature_fraction
uniform[0. 00.1, 1.0]
estimators
1000
boosting
[ghdt, rf, dan)
45

---

Table 49: Hyperparameter space of GradientBoostingThe.
Hyperparameter
Distribution
loss
(uuasiiannn d exponentiall fication)) (qquared error, absolute error, huber] (regression)
learning_ rate
log norma (o.01)..og(10)
subsample num_ cetmmmtttttors
uniform(0.35
1000
criterion
friedman mse, squared_erro
max_ mep__
[non,, 2, 3, 4, 5] [2.3] [0.0, 0.01. 0.02, 0.05] [none, 10, 15]
min__amppp samples_sspppppee
min_ _____jiput__eerree decrease
max_leat max__eef_nodes
Table 50: Hyperparameter search space of RandomForest.
Hyperparameter
Distribution
max_ max_depth
[none, 4 250
num estimators
criterion max features
[gini, entropy [sqrt, log2, none, 0.1, 0.2, 0.3, 0.4, 0.6, 0, 0.9]
min samples split min samples leaf
[2, 3] log uniform int[1.5,, 50.5]
bootstrap
[true, false] [0.0, 0.01, 0.02, 0.05]
min_impurity _ecceese
Table 51: Hyperparameter search space of NODE.
Hyperparameter
Distribution
num_layers total_tree__ ____ttee_comr count
[2, 22 8] [1024, 2048]
tree_depth
[6, 8]
gne__output_dimension
[, [rggreers)))) [num_claar
Table 52: Hyperparameter search space of Trompt.
Hyperparameter
Distribution
hidden. dimension
[18, 128) concat, add]
feature peaturejimportances__og
feature_ saatuu__mmootanns__eeee
true, false
feature_ing saarrremmotttcc_rreeeeee
laleg__ediu_a_seeeee connection
true, false
feature importances. sharing dense feature settuidiemeddiigs residual connection minimal batch, ratio
[rue, false
[true, false
[0.1, 0.01]
Table 53: Hyperparameter search space of FT- Transformer. Hyperparameter
Distribution
layers
uniform uniform_int11,0
feature_embedee embedding.nste.e...eeeeeeennneee size
cilfform_jnt(64 512]
residual_die dropput
uniform [0, 0.5]
diention_dropour
uniform (0, 0.5
1PN__roopout
uniform(O. 0.5
WFNNfactoo
uniform(2 [2/3, 8/3]
learning_ rate
log_umi log_uniform] 1e-5, 1e-3]
weight decay
log uniform le 6, 1e 3] [true, false] [headwise, key_valueee [true, false] [256, 512, 1024]
KV compression
LKV compression sharing
learning rate__rtt____eeeeeee rat_rrattehheee scheduler ratt_scchculllrr
batch_sizz
46

---

Table 54: Hyperparameter search space of SAINT.
Hyperparameter
Distribution
num layers
uniform int[1, 2, 6, 12]
num num_heads
[2,4.8]
layer_size
uniform_ _atpmmninnn______ 64, 128]
dropout
[0, 0.1.0.22 0.3, 0.4, 0.5, 0.6, 0.7, log uniform (1- 1e-3]
learning rate
batch size
[128, 256
Table 55: Hyperparameter search space of ResNet.
Hyperparameter
Distribution
num Jayers
uniform int(1, 16)
layer_siae size hidden_factor factor
uniform (iarrm_in((66 int 1024]
uniform[1, 4]
hidden_dreee dropout
0.5]
residual residual_dropout
uniform(0, 0.5
learning_rate rate
log_uni [ng_uniform[Teeee 1e-2
weight_ decay
log rrg_uniform/ (eciiioommee--5660 [1--8, 1e-3
sntegorry_emedddine _ize normalization
cerfform_int644,,0
[batch norm, layer norm] [true, false] [256, 512, 1024]
learning rate scheduler batch size
Table 56: Hyperparameter search space of MLP.
Hyperparameter
Distribution
num layers layer size dropout
uniform ieti_iin[_1___ 8]
uniform int 16, 1024]
[0, 0.5]
learning_rad rate
log_ www.uunifmmmleee 1e-2)
category_embeddee size
uniform_ing i__ (64 512] [true, false]
learning Beatinng_rtt__ee
scheduler rraeheddette batch_siae
[256, 512, 1024]
Table 57: Hyperparameter search space of GRANDE.
Hyperparameter
Distribution
depth n_estimators
[4, [512, 1024, 2048
learning rate weights
log uniform le [ee 4, le
learning rate index
log jeg_unnffrm/p 55 3, 2e-
learning rate values
log wwwformiommmmmmm - 2e
learning_rate, reaf_leaf cosine laine__dccy_ssteee
log_uniform 5e-3, 2e-1
[0, 100, 1000)
loss dropout variables seleated_vvriiables
[crossentropy, crccalccooscenropyyddee ficatti, [mse[(regression) [0.0, 0.25, 0.5)
uniform (0.m. 1.0)
data jubiisssuset___e fraction
uniform 008, 1.0
47

---

NeurIPS Paper Checklist
1. Claims
Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Question:
Answer: [Yes]
Justification: We have verified our idea using experiments as shown in Section 4
The results not only reveal the state-of-the-art performance of DOFEN among deep neural networks on tabular data but also demonstrate the nuance its architecture design, which echos the contribution and scope we mentioned in the abstract and introduction.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
It is fine to include aspirational goals as motivation as long as is clear that these goals attained by the paper.
2. Limitations
Question:
Does the paper discuss the limitations the work performed by the authors?
Answer: [Yes]
Justification: See Section for limitations.
Guidelines:
The answer NA means that the paper has limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect the scope of the claims made, e.g., if the approach was only tested on a few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low or images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full of assumptions and complete (and correct) proof?
48

---

Answer: [NA]
Justification:
This paper focuses on model architecture design and empirical evaluations
based on hypotheses, without making theoretical assumptions providing proofs.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that affects the main claims and/or conclusions of the paper (regardless whether the code and data are provided or not)?
Answer: [Yes]
Justification: See Section 3.2 and Appendix A.2 for model implementation details, Appen- dices A. and H.2 for hyperparameter settings, and Section 4.1 and Appendices B. to B.3
for dataset settings.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how replicate the results, access a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) the contribution is primarily a new algorithm, the paper should make clear how reproduce that algorithm. (b) If the contribution is primarily
model architecture, the paper should describe the architecture clearly and fully.
(c) the contribution is a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should possible for other researchers to have some path to reproducing verifying the results.
49

---

5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, described in supplemental material?
Answer: [Yes]
Justification: The code will be published in the future; currently, provide anonymous version the supplemental material. For datasets, we use an open-source benchmark and strictly follow its official implementation as described in Section 4.1
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://nips cc/
for more details.
While we encourage the release of code and data, we understand that this might be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https
mniiss.cc/publcc//ee
for more details.
The authors should provide instructions data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code is permitted.
6. Experimental Setting/Details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: For dataset settings, see Section 4.1 and Appendices B.1 to B.3 For detailed hyperparameter settings, see Appendices/ A.1 and H22
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the of the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, or as supplemental material.
7. Experiment Statistical Significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [Yes]
we calculate the mean and standard deviation of ranks across datasets to provide the rank for each model with their confidence interval.
Justification: For tables in Appendix G.2
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by emror bars, confi- dence intervals, or statistical significance tests, least for the experiments that support the main claims of the paper.
50

---

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should given (e.g., Normally distributed errors).
should be clear whether the error bar is the standard deviation the standard error of the mean.
It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out range (e.g. negative error rates).
If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables the text.
8. Experiments Compute Resources Question:
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed reproduce the experiments?
Answer: [Yes]
Justification: See Appendix H.T for hardware settings and Appendix C.1 for the FLOPs and inference time of different models.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount compute required for each of the individual experimental runs as well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported the paper (e.g., preliminary or failed experiments that didn't make it into the paper).
9. Code Of Ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurlPS Code of Ethics
Answer: [Yes]
Justification: The research conducted in our paper conforms, in every respect, with the NeurIPS Code of Ethics.
Guidelines:
The answer NA means that the authors have not reviewed the NeurlPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., if there special consid- eration due to laws or regulations in their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [No]
Justification: This paper proposes novel DNN model, with experiments focusing on performance comparisons with other existing DNN and tree-based models, as well as exploring the mechanisms behind this novel method. We acknowledge that the introduction of new model structures can have both positive and negative societal impacts (e.g. fairness considerations): however, these aspects not the primary scope focus of this paper.
51

---

Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g.. disinformation, generating fake profiles, surveillance), fairesss considerations (e.g., dep.oymmly of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification:
This paper poses no such risks.
Guidelines:
The answer NA means that the paper poses such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators or original owners of assets (e.g.. code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification: The public assets accessed in this paper include models and datasets. We have clearly cited each of them and strictly followed their terms of use. For all public models, please refer to their respective websites for the license information. Regarding the Tabular Benchmark, all datasets used are publicly available from OpenML [36] under the CC-BY 4.0 license. Please refer Appendix B.3 for detailed information the datasets used.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package dataset.
52

---

The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset. For scraped data from a particular source (e.g.. website), the copyright and terms of service of that souree should be provided.
assets are released, the license, copyright information, and terms use in package should be provided. For popular datasets, paperswi thcode. com/ datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided. If this information is not available online, the authors are encouraged to reach out to the asset's creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [Yes]
The source code for this paper will be published under the Apache License 2.0 in the future. During the reviewing process, a minimal workable example of DOFEN is provided the supplementary material, along with simple README explaining how to execute the code.
Justification:
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create anonymized URL or include anonymized zip file.
14. Crowdsourcing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?
Answer: [NA]
Justification:
The paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsoureing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
53

---

Justification:
The paper does not involve crowdsoureing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsoureing nor research with
human subjects.
Depending the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurlPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such the institution conducting the review.
54


Reviews:
[{"review_id": "YNWDqiOCKS", "reviewer": "Reviewer_PzWm", "summary": "This paper primarily investigates DNN models for tabular data. Current DNN models have limited performance and do not outperform GBDTs. The authors believe the key lies in the need for sparse feature extraction from tabular data. Existing attention-based models lead to dense selection of columns, while tree-based DNNs can only achieve near-sparsity. The paper introduces a novel deep neural network architecture called DOFEN (Deep Oblivious Forest ENsemble), aiming at enhancing performance on tabular data. Inspired by Oblivious Decision Trees (ODT), DOFEN implements a new two-step process to achieve on-off sparse selection of columns, thereby gaining advantages in feature diversity and mitigating overfitting. The paper evaluates DOFEN on the recognized Tabular Benchmark, demonstrating its excellent performance across various tasks, especially in competitiveness when compared with Gradient Boosting Decision Trees (GBDTs).", "questions": "1. The paper mentions that DOFEN employs randomized steps to enhance the model's generalization ability, but what is the specific mechanism by which these steps affect the training dynamics and ultimate performance of the model?\n2. The performance of DOFEN largely depends on the selection of hyperparameters. Is there an automated strategy for hyperparameter adjustment?\n3. While DOFEN shows excellent performance in experiments, how does it perform in real-world application scenarios, especially when dealing with large-scale datasets or in resource-constrained environments?\n4. Compared to traditional tree models, how interpretable is the decision-making process of DOFEN as a deep learning model, and are there ways to improve the model's interpretability?", "limitations": "see the section of weakness.", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. Innovation: DOFEN integrates the concept of ODT with deep learning techniques, proposing a new neural network architecture, which is an innovation in the processing of tabular data.\n2. Performance: Evaluation results on the Tabular Benchmark show that DOFEN has achieved state-of-the-art performance levels in multiple tasks, demonstrating competitiveness compared to GBDTs.\n3. Robustness and Universality: DOFEN has shown promising performance across various types of datasets, whether they feature numerical attributes or a combination of numerical and categorical features.", "weaknesses": "1. Implementation Complexity: The architecture of DOFEN may be more complex than traditional tree models and some DNN models, which could increase the difficulty of implementation and understanding.\n2. Long Inference Time: Although DOFEN performs well in some aspects, its inference time is relatively long, which may affect its use in applications that require real-time response.\n3. Slow Convergence Speed: The randomized steps in DOFEN may lead to a slower training convergence speed, requiring more training steps to achieve optimal performance.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for their feedback. Following are our responses to each individual comment.\n\n> Weaknesses 1: Implementation Complexity: The architecture of DOFEN may be more complex than traditional tree models and some DNN models, which could increase the difficulty of implementation and understanding.\n\nThe high-level idea of DOFEN is simple. It combines discrete column conditions motivated by ODT and two stages of randomization for ensembles. We will improve the clarity of the equations by replacing right arrows with detailed element-wise operations and adding a Table of Contents (ToC) section before the Appendix section. Regarding the difficulty of implementation, we have provided code within the submission files and will open source our code in the future.\n\n> Weaknesses 2: Long Inference Time: Although DOFEN performs well in some aspects, its inference time is relatively long, which may affect its use in applications that require real-time response.\n\nPlease refer to \"Long Inference Time of DOFEN\" of global rebuttal for our reply.\n\n> Questions 1: The paper mentions that DOFEN employs randomized steps to enhance the model's generalization ability, but what is the specific mechanism by which these steps affect the training dynamics and ultimate performance of the model?\n\nDOFEN employs randomization in two critical steps to enhance its generalization ability:\n1. **Random Selection of Conditions for rODT Construction**: DOFEN randomly selects conditions for the construction of rODTs using a shuffle-then-segment process. This approach aims to create a diverse set of rODTs and achieves a real on-off selection of features, a key characteristic of tree-based models. In order to make the on-off selection differentiable, this randomization step serves as a naive method to achieve it. In Appendix J.3, we compare this randomization with a predefined deterministic strategy. While the predefined deterministic strategy can slightly outperform random selection, it requires training a Catboost model first. Considering differentiability offers benefits for end-to-end training, random selection of conditions remains a viable and promising approach.\n2. **Random Selection of rODTs for rODT Forest Construction**: DOFEN employs a simple bagging strategy  through randomly selecting rODTs for the rODT forest construction. This increases the diversity of the rODT forests, thereby enhancing the performance of forest ensemble. In Appendix J.1, we conduct an ablation study by removing this strategy. The experimental results demonstrate that DOFEN suffers from overfitting (Figure 9) and a huge degradation in performance (Table 19).\n\nThese experiments confirm that both randomized steps in DOFEN significantly enhance the model’s generalization ability and have a critical impact on the ultimate performance. For a more detailed background and motivation, please refer to lines 31 to 55 of the paper. However, these randomizations also result in a slower convergence speed, meaning that DOFEN requires more training steps to reach optimal performance. This limitation is mentioned in Section 5 of the paper.\n\n> Questions 2: The performance of DOFEN largely depends on the selection of hyperparameters. Is there an automated strategy for hyperparameter adjustment?\n\nIt is important to note that extensive hyperparameter tuning is essential for maximizing performance in many machine learning methods. We acknowledge that achieving optimal performance with DOFEN, particularly in medium-sized regression tasks with exclusively numerical features (as illustrated in Figure 4a), necessitates more random search iterations. However, Figure 4a also demonstrates that XGBoost reaches peak performance only after a considerable number of random search iterations.\n\nRegarding the automated hyperparameter adjustment strategy for DOFEN, DOFEN can benefit from all existing optimization strategies such as Bayesian optimization or random search. In this paper, we currently employ random search due to the configurations of the Tabular Benchmark.\n\n> Questions 3: While DOFEN shows excellent performance in experiments, how does it perform in real-world application scenarios, especially when dealing with large-scale datasets or in resource-constrained environments?\n\nFor large-scale datasets, DOFEN has demonstrated promising results in experiments as illustrated in Figure 1c and 1d, and detailed in Appendix K and L. These experiments indicate that DOFEN can deal with large-sized datasets effectively compared with other models.\n\nIn resource-constrained environments, the capacity of models can be downscaled to fit the limited computational resources. To evaluate the performance impact of these capacity adjustments for DOFEN, we conducted a series of scalability experiments detailed in Appendix I. The results reveal that lowering DOFEN’s capacity under its default settings always causes performance degradation, but also increases efficiency due to the reduced parameter size and floating point operations (FLOPs) reported. This indicates that one can always trade off performance for efficiency when facing a resource-constrained environment.\n\n> Questions 4: Compared to traditional tree models, how interpretable is the decision-making process of DOFEN as a deep learning model, and are there ways to improve the model's interpretability?\n\nPlease refer to \"Interpretebility of DOFEN\" of global rebuttal for our reply.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "wyjm1Vx4jx", "reviewer": "Reviewer_K3P3", "summary": "This paper proposes Deep Oblivious Forest ENsemble (DOFEN), a novel neural network-based architecture for tabular deep learning. DOFEN extends Neural Oblivious Decision Ensembles (NODE) by relaxing the interaction level between a sample and the conditions on nodes in oblivious decision trees. In particular, when constructing oblivious decision trees, DOFEN follows a two-stage procedure, where the first stage generates soft conditions and the second stage utilizes a shuffle-then-segment framework to generate multiple condition tables. The resultant combination of condition tables can be regarded as a differentiable counterpart to the vanilla oblivious decision trees, which the authors referred to as \"relaxed oblivious decision trees\". Experimental results on the widely adopted tabular benchmark demonstrate that DOFEN achieves excellent performance, outperforming a number of state-of-the-art tabular deep learning models.", "questions": "- In the code provided in the supplementary material, is CCCModel the implementation of DOFEN? If so, could the authors please point to me which part of the code corresponds to $\\Delta_2$ in DOFEN?\n\n- In the shuffle-then-segment procedure in DOFEN, is the shuffle random for each sample, or is it fixed during training and inference?", "limitations": "The authors have adequately addressed the limitations and potential negative societal impact of their work.", "rating": 4, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "- The idea of using relaxed oblivious decision trees for tabular deep learning is, to the best of my knowledge, novel and technically sound.\n\n- The authors adopted the highly recognized tabular data learning benchmark, which ensures a rigorous evaluation of the proposed approach. Experimental results also suggest that DOFEN achieves very competitive performance.", "weaknesses": "- The paper is not very well-written, and I found it quite hard to follow.\n\n- No ablation study is provided, and thus the readers cannot know how much each component in DOFEN contributes to the final performance. I would suggest that the authors dissect the difference between DOFEN and NODE, and conduct an ablation study to demonstrate how each newly introduced component improves the results.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for their feedback. Following are our responses to each individual comment.\n\n> Weaknesses 1: The paper is not very well-written, and I found it quite hard to follow.\n\nIn future revisions, we will add a Table of Contents (ToC) in front of the Appendix section and update the equations to avoid using right arrows. We hope this will make the literature easier to follow. We welcome any further suggestions.\n\n> Weaknesses 2: No ablation study is provided, and thus the readers cannot know how much each component in DOFEN contributes to the final performance. I would suggest that the authors dissect the difference between DOFEN and NODE, and conduct an ablation study to demonstrate how each newly introduced component improves the results.\n\nWe have conducted several analyses of the components of DOFEN in Appendix J. In Appendix J.1, we studied the importance of the second stage ensemble (forest ensemble) and found that this stage significantly contributes to the final performance, as shown in Table 19. Additionally, incorporating this stage mitigates overfitting, as illustrated in Figure 9.\n\nIn Appendix J.2, we explored another level of ensemble (seed ensemble) to determine whether it helps further improve performance. The results demonstrate a slight performance improvement, as shown in Table 20. In Appendix J.3, we investigated whether there are better condition selection strategies (by default, we use a random shuffle and segment approach; please refer to Figure 2). We tested using columns selected by the CatBoost model and found that it only slightly improve performance, as shown in Table 21. In Appendix J.4, we also explored an alternative strategy to select rODTs other than random sampling, through a sliding window approach. The results show that this deterministic method does not outperform random sampling.\n\nBesides the second stage ensemble, it is challenging to directly ablate one of the components of DOFEN. As a result, we designed several logical alternatives and investigated how they contribute to performance, presenting the results in an analysis section (Appendix J). In future revisions, we will rename the section to \"Analysis of Model Components\" to clarify its intention.\n\nRegarding the difference between DOFEN and NODE, we have described it in detail in Appendix B. Although DOFEN and NODE share a motivation of incorporating ODT into deep neural networks, their architectures are fundamentally different. As a result, we cannot conduct a concise ablation study comparing these two models.\n\n> Questions 1: In the code provided in the supplementary material, is CCCModel the implementation of DOFEN? If so, could the authors please point to me which part of the code corresponds to $\\Delta_2$ in DOFEN?\n\nSorry about the  incorrect naming in the provided code, we will fix this in the open source version in the future. I assume the CCCModel you mentioned is meant to be the OriginalCCC class we provided. The OriginalCCC class is the implementation of DOFEN. The sub-network $\\Delta_2$ mentioned in the paper corresponds to a Sequential module called subtree_scoring in the OriginalCCC class.\n\nAs detailed in the appendix, a $\\Delta_2$ consists of multiple MLP and LayerNorm layers, and we have multiple $\\Delta_2$ networks to evaluate how well a sample aligns with the conditions of different rODTs. To make this operation parallel, we chose to implement $\\Delta_2$ with group convolution and group normalization, where the number of groups is set to the number of rODTs. This means each rODT will operate with its own $\\Delta_2$, and finally, after the $\\Delta_2$ operation, each rODT will have its corresponding score for the later rODT forest construction step.\n\n> Questions 2: In the shuffle-then-segment procedure in DOFEN, is the shuffle random for each sample, or is it fixed during training and inference?\n\nThe shuffling process in the shuffle-then-segment procedure is done once before training and remains fixed during both training and inference. It is crucial to keep the column in each rODT fixed to effectively train the $\\Delta_2$ and rODT embedding vectors. If the tree structure (e.g. columns used by a tree) of each rODT is not consistent during training, the model will struggle to learn the correct scoring and representation. For a more detailed description of this setting, you can refer to lines 174 to 177 of the paper. We will clarify this further in both Figure 2 and its caption in future revisions to make it easier for readers to follow.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "GDC91F2OTK", "reviewer": "Reviewer_EjLV", "summary": "Tabular data is widely used in various fields. This paper studies the current situation that DNN still lags behind Gradient Boosting Decision Trees (GBDTs) in tabular data. This paper introduces Deep Oblivious Forest Ensemble (DOFEN), which is inspired by unbiased decision trees and implements on-off sparse selection of columns. In addition, DOFEN shows excellent performance on the recognized Tabular Benchmark, surpassing other neural network models and similar to GBDTs in multiple tasks. This achievement highlights the robustness and diversity of DOFEN in different tasks and proves its effectiveness on tabular data.", "questions": "1. The DOFEN model mentioned in the paper has a long inference time. What are the specific performance bottlenecks in implementation? Does the author consider or implement specific optimization measures to improve the efficiency of the model?\n2. The randomness introduced in the model is mentioned in the paper. How does this randomness affect the stability and reproducibility of the model? Is there a way to reduce this effect?\n3. In deep learning models, model interpretability is an important consideration. Does the DOFEN model provide an explanation for the decision-making process? How are these explanatory features realized?\n4. The paper mentions some limitations of the model. What are the key assumptions in the model design and experiment? How do these assumptions affect the performance of the model in practical applications?\n5. Does the author test the generalization ability of the DOFEN model on different data distributions or domains ? If yes, please share the results and analysis of these experiments.", "limitations": "1. Compared with other DNN models, DOFEN has a relatively long inference time, which may limit its practicality in application scenarios that require fast response.\n2. Due to the randomization steps introduced in the DOFEN model, the model requires more training steps to achieve optimal performance, which leads to a training process that may be more time-consuming than other models.\n3. The performance of DOFEN may be sensitive to hyperparameter selection, and sufficient hyperparameter adjustment is required to obtain the best performance, which may increase the complexity of model tuning.", "rating": 5, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "First of all, this paper has sufficient technical contributions and corresponding theoretical analysis. I believe that this idea of introducing oblivious decision trees into deep neural network (DNN) architecture will promote the research progress of deep learning in tabular data. Secondly, this article is well written and easy to understand. The explanation of the formula is sufficient. Finally, the paper provides a comprehensive assessment of the DOFEN model, including model stability, efficiency, and hyperparameter effects, which provides valuable insights for further research and model optimization.\n1. A novel deep neural network (DNN) architecture is proposed by DOFEN, which combines the concept of oblivious decision trees (ODT) and realizes the on-off sparse column selection of table data. This is a feature that is not common in existing DNNs.\n2. DOFEN solves the non-differential problem of sparse selection in deep learning models through its innovative two-step process, which is a challenge in previous studies.\n3. The paper not only proposes a new method in theory, but also verifies the effectiveness of the DOFEN model through experiments, showing its application potential on different types of data sets.\n4. Although DOFEN is a complex model, the paper provides detailed model configuration and hyperparameter setting, which is helpful for other researchers or practitioners to implement and apply this model.", "weaknesses": "1. Compared with other DNNs, the inference time of DOFEN may be relatively long, which may affect its practicality in application scenarios that require rapid response.\n2. The randomization step of DOFEN leads to a slower convergence speed, which means that more training steps are needed to achieve optimal performance, which may increase training costs and time.\n3. Although the paper mentions that increasing the number of forests can alleviate the instability caused by randomness, randomness itself may pose a challenge to the stability and reproducibility of the model.\n4. Although the paper mentions that increasing the number of forests can alleviate the instability caused by randomness, randomness itself may pose a challenge to the stability and reproducibility of the model.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for their feedback. Following are our responses to each individual comment.\n\n> Weakness 1: Compared with other DNNs, the inference time of DOFEN may be relatively long, which may affect its practicality in application scenarios that require rapid response.\\\n> Question 1: The DOFEN model mentioned in the paper has a long inference time. What are the specific performance bottlenecks in implementation? Does the author consider or implement specific optimization measures to improve the efficiency of the model?\n\nPlease refer to \"Long Inference Time of DOFEN\" of global rebuttal for our reply.\n\n> Weakness 3: Although the paper mentions that increasing the number of forests can alleviate the instability caused by randomness, randomness itself may pose a challenge to the stability and reproducibility of the model.\\\n> Question 2: The randomness introduced in the model is mentioned in the paper. How does this randomness affect the stability and reproducibility of the model? Is there a way to reduce this effect?\n\nWe understand that readers might be concerned about the instability due to the randomization steps involved in DOFEN. To address this, we conducted the experiment in section 4.3.1 to better understand the significance of this issue and how to mitigate its effects. Table 1 shows that decreasing $N_{forest}$​ gradually raises instability, and it only becomes slightly unacceptable when $N_{forest}$​ is set too low (e.g., ≤10 forests). On the other hand, using the default setting of DOFEN (e.g. 100 forests) ensures adequate performance as well as stability for most datasets. Overall, the experimental results demonstrate that the stability of DOFEN is not an issue in most cases ($N_{forest}>10$).\n\nIn terms of reproducibility, with a fixed random seed and hyper-parameter settings, both the training and inference procedures can always be reproduced. This is not a concern for DOFEN, just as it is for any other model.\n\n> Question 3: In deep learning models, model interpretability is an important consideration. Does the DOFEN model provide an explanation for the decision-making process? How are these explanatory features realized?\n\nPlease refer to \"Interpretability of DOFEN\" of global rebuttal for our reply.\n\n> Question 4: The paper mentions some limitations of the model. What are the key assumptions in the model design and experiment? How do these assumptions affect the performance of the model in practical applications?\n\nThe design of the DOFEN model is based on two key assumptions: that sparse selection of columns is crucial for tabular data and that incorporating ensemble technique will improve performance. Both assumptions are inspired by tree-based models, which have demonstrated strong performance with tabular datasets.\n\nDOFEN achieves sparse column selection by generating conditions for each column (matrix $\\mathbf{M}$) followed by a shuffle-then-segment process, as shown in Equation 5 and Figure 2. This approach not only achieves an on-off sparse column selection but also makes the entire process differentiable.  Afterwards, DOFEN employs a two-level ensemble strategy. The first level involves constructing individual rODT forests by randomly selecting subsets from the rODT pool, as detailed in Section 3.2.3. In the second level, these rODT forests are treated as weak learners and aggregated into a comprehensive forest ensemble through bagging. The effectiveness of the model largely relies on the second-level ensemble, ablating it can lead to overfitting and model performance degradation, as shown in Appendix J.1.\n\nRegarding experimental assumptions, we wish to evaluate the performance of DOFEN on a benchmark incorporating datasets from as many domains as possible. As a result, we choose the Tabular Benchmark, which includes 73 datasets spanning a wide array of domains such as governmental operations, botany, house pricing, astronomy, computing resources, medicine and more. However, we want to acknowledge that while the model performs effectively in a controlled, closed environment, this performance may not directly translate to real-world applications and this challenge is common across many research projects.\n\n> Question 5: Does the author test the generalization ability of the DOFEN model on different data distributions or domains ? If yes, please share the results and analysis of these experiments.\n\nThe generalization capability of the DOFEN model was assessed using the Tabular Benchmark, as described in question 4. For comprehensive descriptions of these datasets, please refer to Appendix A of the Tabular Benchmark. DOFEN demonstrated promising performance on this broad benchmark, suggesting its effectiveness across various data domains. Detailed results and analyses of these experiments are presented in Section 4 and Appendices K and L.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "aMb9BhSK6B", "reviewer": "Reviewer_qMj5", "summary": "The paper introduces DOFEN, a deep learning method for tabular data inspired by oblivious decision trees and random forests. DOFEN combines a novel differentiable oblivious tree model with a novel forest ensembling stage. On a well-established tabular benchmark, DOFEN performs close to boosted trees and mostly outperforms other deep learning baselines.", "questions": "**Questions:**\n- Section 3.2.2.: Is the shuffling process necessary to obtain good performance or would direct random sampling of coordinates be sufficient? Do you use more trees for datasets with more features?\n- In Appendix C you say that you do not use early stopping. Do you still select the best epoch based on the validation set? If no, why not?\n\n**Comments:**\n- l. 222: \"Trompt is the current state-of-the-art tabular DNN.\" I would say this is at least unclear, since for example TabR (https://openreview.net/forum?id=rhgIgTSSxW) claimed to outperform XGBoost on the same benchmark. There are also many other deep learning methods that claim to be SOTA, e.g., GANDALF (https://arxiv.org/abs/2207.08548), GRANDE (https://arxiv.org/abs/2309.17130), BiSHop (new, so not required to cite, https://arxiv.org/abs/2404.03830).\n- For the related work I think GRANDE (https://arxiv.org/abs/2309.17130) is highly related, and they also propose differentiable trees that output weights together with the predictions. \n- I think oblivious decision trees are mentioned before they are explained, the authors might consider explaining them earlier.\n\n**Minor comments:**\n- Eq. 2: The notation is a bit strange, specifically the set notation for ODT(x), the long right arrow, the definition of $j$, and why j is an upper index and not a lower index. An alternative suggestion might be $ODT(x) = H(x_j - b), (j, b) = criterion(context), b \\in \\mathbb{R}, j \\in \\{1, \\dots, N_{col}\\}$.\n- l. 157: It is not clear how the shuffling of the matrix works. Is it equivalent to shuffling after flattening? If yes, then it might be better to write it in this way. But in Figure 2 there is still exactly one color per column, which might indicate that the shuffling works differently?\n- Eq. 6: From the text it seems that $\\Delta_2$ is applied for each tree separately, producing a scalar output. From the equation, it seems that it is applied to the whole shuffled matrix, producing the vector $w$.\n- Eq. 9: The naming with one prime for the vector and two primes for the elements might be confusing. Also, it is not clear here if softmax is applied to each $w_i''$ separately or somehow together.\n- Eq. 8-11: It is unclear to me why you write everything as operations on big vectors, instead of just writing the computation for a single index $i$. \n- Eq. 12: use \\mathrm or \\operatorname to write names in formulas (then it doesn't look like products of single-letter variables)\n- An appendix table of contents at the beginning of the appendix would be nice to have.\n\n**Conclusion:** The paper provides a new method that achieves strong results on an established benchmark on a practically important problem. The results appear solid and comprehensive. Therefore I recommend acceptance.", "limitations": "Limitations are discussed. Broader impacts are not discussed, but the method is general-purpose, so the impact depends on the downstream applications.", "rating": 7, "confidence": 4, "soundness": 4, "presentation": 3, "contribution": 3, "strengths": "- The model is evaluated on a well-established benchmark.\n- The model achieves strong results on the benchmark (not necessarily state-of-the-art in every respect, but still promising).\n- Many additional results are provided in the appendix.\n- The ensembling stage, which is shown to be useful in ablations, could be useful for other deep tree-based models as well.\n- Code is provided.", "weaknesses": "1. I found the description of the method sometimes confusing, especially the mathematical notation (see below). Maybe writing the method as an algorithm could help? I was also missing some details the sub-networks $\\Delta_i$ in the main paper (are they fully-connected?).\n2. Training times are not reported (although inference times are reported) and from what I can infer from the text, training is rather slow.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for their feedback. Following are our responses to each individual comment.\n\n> Weaknesses 1: I found the description of the method sometimes confusing, especially the mathematical notation (see below). Maybe writing the method as an algorithm could help? I was also missing some details the sub-networks $\\Delta_{i}$ in the main paper (are they fully-connected?).\n\nAppendix D and Figures 6-8 detail the components of $\\Delta_1$, $\\Delta_2$, and $\\Delta_3$. $\\Delta_1$ uses embedding layers for numerical and categorical inputs, while $\\Delta_2$ and $\\Delta_3$ employ fully-connected layers, layer normalization, and dropout for hidden representations.\n\n> Weaknesses 2: Training times are not reported (although inference times are reported) and from what I can infer from the text, training is rather slow.\n\nYes, the training of DOFEN is slow, and reporting training times for each model is challenging due to:\n\n1. **Incomplete Data**: Many models, including FT-Transformer, MLP, ResNet, and Trompt, have incomplete or missing training times in their benchmarks or original papers. This inconsistency hinders fair comparison. Additionally, we did not record training times for models like DOFEN and NODE during our experiments.\n2. **Resource Constraints**: To ensure a fair comparison, each model would need to be retrained on the same hardware to control for hardware variability. However, we do not have sufficient resources or time to perform these extensive reruns.\n\n> Questions 1-1: Is the shuffling process necessary to obtain good performance or would direct random sampling of coordinates be sufficient?\n\nThe shuffle-then-segment process is equivalent to random sampling without replacement; it's an implementation alternative. We choose sampling without replacement over sampling with replacement because:\n1. We want to use all generated conditions.\n2. We aim to ensure each rODT uses a different set of conditions, maximizing their diversity.\n\n> Questions 1-2: Do you use more trees for datasets with more features?\n\nYes, we use more trees for datasets with more features. The number of rODTs ($N_{rODT}$) for each dataset, determined by the formula in Table 2, is positively correlated with $N_{col}$ (the total number of features). We hypothesize that more features necessitate more trees to cover diverse feature combinations.\n\nFor the exact number of rODTs used for each dataset, see Table 8 in Appendix G, which also shows the $N_{estimators}$ set for each dataset, correlated with the number of features.\n\n> Questions 2: In Appendix C you say that you do not use early stopping. Do you still select the best epoch based on the validation set? If no, why not?\n\nIn the Tabular Benchmark specification, early stopping is optional and not consistently applied across models. According to Tabular Benchmark, Random Forest and Gradient Boosting Tree do not use a validation set for early stopping.\n\nIn our experiments, we followed each model's original implementation, applying early stopping where intended. For the additional models compared—LightGBM, CatBoost, NODE, and Trompt—we used their default settings, resulting in early stopping only for NODE. For DOFEN, we believe that the randomization steps enhance generalizability, reducing overfitting risk, so we chose not to apply early stopping.\n\n> Comments 1: l. 222: \"Trompt is the current state-of-the-art tabular DNN.\" I would say this is at least unclear, since for example TabR claimed to outperform XGBoost on the same benchmark. There are also many other deep learning methods that claim to be SOTA, e.g., GANDALF, GRANDE, BiSHop (this one is new, so not required to cite).\n\nWe reviewed related works and acknowledged that our previous claim was overstated. Many recent tabular neural network papers recognize the challenge of using different evaluation benchmarks, making direct performance comparisons difficult without conducting experiments.\n\nWhile Trompt, TabR, GANDALF, GRANDE, and BiSHop use the Tabular Benchmark, only Trompt strictly follows its settings. TabR altered the evaluation procedure and filtered datasets (Appendix C.2), GANDALF excluded some datasets due to resource constraints (Section 4.1.1), GRANDE selected datasets from their collection (Section 4), and BiSHop used only one-third of the datasets due to resource constraints.\n\nWe followed the standard settings of the Tabular Benchmark, similar to Trompt, to facilitate easy comparison by future works. In future revisions, we will limit the claim in sentence l.222 to our specific experimental setting, cite these recent works, and attempt comparisons if resources allow. Thank you for your concrete suggestions!\n\n> Comments 2: For the related work I think GRANDE is highly related, and they also propose differentiable trees that output weights together with the predictions.\n\nThis is a highly relevant recent work that we will definitely cite and compare in future revisions. The differences between GRANDE and DOFEN are twofold. First, GRANDE uses a straight-through operator for axis-aligned and differentiable split functions, while DOFEN generates a discrete condition set. Second, GRANDE employs an instance-wise weighted sum for GradTree predictions, whereas DOFEN uses a second stage randomization to select rODTs, compute weights, and form rODT forests, followed by a forest ensemble for the final prediction.\n\n> Comments 3: I think oblivious decision trees are mentioned before they are explained, the authors might consider explaining them earlier.\n\nThank you for your suggestion. We will add an explanatory sentence to introduce ODT in the introduction section.\n\n> Minor comments\n\nDue to the limited length of the rebuttal, we are not able to reply each comment in detail. Please refer to \"The Readability of the Paper\" of global rebuttal for our reply to comments for notations. Besides, we will cite and compare with recent related works mentioned above in future revisions. Thanks you for your concrete advices!", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
We thank all the reviewers for their time and insightful comments which have helped improve our paper. We found all reviewers mentioned the issue of relatively long inference time of DOFEN, and several reviewers mentioned the interpretability of DOFEN and the readability of the paper. Hence, we decide to address them here collectively as follows:

> Long Inference Time of DOFEN

We analyzed the inference time of each DOFEN module in proportion, as shown in Tables A1 and A2 (provided in the additional attachment), which is averaged across 59 medium-sized datasets with default hyperparameters. Table A1 shows that the Forest Construction module consumes the most inference time. In Table A2, more detailed operations reveal that the sub-module $\Delta_2$ in the Forest Construction module, which generates weights for each rODT, has the longest inference time.

The sub-module $\Delta_2$ is designed with multiple MLP and normalization layers, implemented using group convolution and group normalization to parallelize scoring for each rODT. However, the efficiency of group convolution in PyTorch has been problematic and remains unresolved. Specifically, the operation efficiency decreases as the number of groups increases, sometimes making it slower than separate convolutions in CUDA streams (see PyTorch issues 18631, 70954, 73764). The sub-module $\Delta_1$ also uses group convolution to parallelize condition generation across different numerical columns, resulting in slower inference times compared to other operations, though less significant than $\Delta_2$ due to fewer groups being used.

However, we mainly focus on the concept and model structure in this paper, acknowledging that model implementation can be further optimized. For example, attention operations are originally slow due to quadratic complexity, and many recent works have successfully accelerated the speed of attention operations and reduced their memory usage. Hence, we believe there will be better implementations of these group operations with much greater efficiency in the future.

> Interpretability of DOFEN

In tree-based models, feature importance is often determined by the total gain of splits using the feature. Although DOFEN does not possess the same characteristics as tree-based models to use gain-based feature importance, the on-off sparse feature selection nature of DOFEN still makes it possible to interpret features similarly to tree-based models. Specifically, we use a feature importance type called "split" or "weight" provided in LightGBM and XGBoost packages, which counts the number of times a feature is used in the model. 

To obtain DOFEN's feature importance, we first calculate the occurrence of features across different rODTs. We then measure how well a sample aligns with the conditions of an rODT using the sub-module $\Delta_2$, which outputs a vector $w$ (in Equation 2) to represent the importance across all rODTs for each sample. A softmax operation is further applied to the vector $w$ to ensure the importance sums to 1 (which is also done in Equation 9). Finally, we perform a weighted sum between the feature occurrences and the importance of each rODT, resulting in a single vector representing DOFEN's feature importance. 

We tested the reliability of the feature importance introduced for DOFEN on three real-world datasets: the mushroom dataset, the red wine quality dataset, and the white wine quality dataset, following Trompt's experimental design. The results, shown in Tables A3 to A5 (provided in the additional attachment), indicate that the top-3 important features selected by DOFEN match those selected by other tree-based models, with minor ranking differences. This demonstrates DOFEN's ability to reliably identify key features while maintaining interpretability despite its deep learning structure.

> The Readability of the Paper

We will revise equations and descriptions to make it easier for readers to navigate. Below are responses and improvements addressing reviewer qMj5's suggestions.

Suggestions 1: The details about the sub-networks $\Delta_{2}$, Eq. 2,  and shuffling of the matrix $\mathbf{M}$.

- $\Delta_2$ is applied to each tree separately in Eq. 6, and we will rewrite the equation for clarity.

- The notation in Eq. 2 uses long right arrows to represent complex operations. We will consider detailing these processes using algorithms in future revisions. $j$ is used as an upper index to keep $i$ as a lower index, which is common in tabular data representation.

- The shuffling process is equivalent to shuffling after flattening the matrix. We will update the descriptions and figures to better convey this idea.

Suggestions 2: Improve the notation in Eqs. 8-12, and add an appendix table of contents.

- In Eq. 9, the softmax is applied to each $w_i'$ individually, and we will use a different character for $w_i'$ and $w_i''$ to improve clarity.

- In Eqs. 8-11, element-wise operations will be explicitly written for single indices to improve clarity.

- We will use \mathrm or \operatorname to improve the notation in Eq. 12.

- An appendix table of contents will be added in future revisions to help navigate the appendix.

Meta-review: The work presents a novel deep neural network (DNN) architecture inspired by oblivious decision trees for tabular data. The proposed model, DOFEN, demonstrates promising results and outperforms other DNNs on tabular data. While the paper received mixed ratings (i.e., 1 borderline reject, 1 weak accept, and 2 borderline accept), most reviewers acknowledge the effectiveness and novelty of the proposed method, and their primary concerns have been addressed. In addition, after carefully checking the rebuttal and discussions, the results are reproducible as claimed by the authors. However, reviewers also highlighted that the writing and presentation, particularly regarding notations, need further refinement. Additionally, they suggest that more results and analyses, such as training time, be included and clarified. It is especially important to provide clearer details on experimental settings (e.g., evaluation metrics, hyperparameter settings, etc.) to enhance reproducibility. The authors should incorporate these suggestions into the final version of the paper.
Decision: Accept (poster)