Title: DOFEN: Deep Oblivious Forest ENsemble
Venue: NeurIPS 2024
Accepted: yes
Keywords: Tabular Data

Structured Data

Deep Neural Network

Architecture Design
Abstract: Deep Neural Networks (DNNs) have revolutionized artificial intelligence, achieving impressive results on diverse data types, including images, videos, and texts. However, DNNs still lag behind Gradient Boosting Decision Trees (GBDT) on tabular data, a format extensively utilized across various domains. This paper introduces DOFEN, which stands for Deep Oblivious Forest ENsemble. DOFEN is a novel DNN architecture inspired by oblivious decision trees and achieves on-off sparse selection of columns. DOFEN surpasses other DNNs on tabular data, achieving state-of-the-art performance on the well-recognized benchmark: Tabular Benchmark, which includes 73 total datasets spanning a wide array of domains. The code of DOFEN is available at: https://github.com/Sinopac-Digital-Technology-Division/DOFEN

DOFEN: Deep Oblivious Forest ENsemble

Abstract

Deep Neural Networks (DNNs) have revolutionized artificial intelligence, achieving impressive results on diverse data types, including images, videos, and texts. However, DNNs still lag behind Gradient Boosting Decision Trees (GBDT) on tabular data, a format extensively utilized across various domains. This paper introduces DOFEN, which stands for D eep O blivious F orest EN semble. DOFEN is a novel DNN architecture inspired by oblivious decision trees and achieves on-off sparse selection of columns. DOFEN surpasses other DNNs on tabular data, achieving state-of-the-art performance on the well-recognized benchmark: Tabular Benchmark [1], which includes 73 total datasets spanning a wide array of domains. The code of DOFEN is available at: https: //github.com/Sinopac-Digital-Technology-Division/DOFEN .

1 Introduction

Tree-based models, including RandomForest [2], Extra Trees [3], and Gradient Boosting Decision Tree (GBDT) frameworks such as XGBoost [4], LightGBM [5], and CatBoost [6], are widely recognized for their simplicity, efficiency, and remarkable performance with tabular data. This has inspired numerous studies investigating the integration of tree-based algorithms with deep neural networks (DNNs), leading to tree-inspired DNNs such as Deep Forest [7], NODE [8], and TabNet [9]. In another line of tabular DNN research, novel DNN architectures such as SAINT [10], FT-Transformer [11], and Trompt [12] have been proposed. These novel architectures, which are essentially attention-based, demonstrate better performance compared with tree-inspired DNNs but require significantly more time and space. While these tabular DNNs have shown promising performance in specific contexts, recent surveys and benchmarks generally indicate that they do not surpass the performance of GBDTs on tabular data [1, 11, 13-15].

Hence, we begin by questioning what's missing in existing tabular DNNs and identify one key difference: in tree-based models, only a limited number of features are used in the construction of each tree. This concept of "sparse selection of columns" not only increases feature diversity but also helps mitigate overfitting in tree-based models [2-6]. However, existing tabular DNNs are unable to achieve a sparse selection. For example, attention-based models [10-12] use the softmax operation to aggregate column information, resulting in a "dense selection" across columns. Some tree-inspired DNNs [8, 9] have utilized methods like entmax and sparsemax [16, 17] to enhance sparsity, but they can still only achieve near-sparse effects. Therefore, we opt to develop a new approach to achieve this characteristic.

CatBoost

CatBoost

CatBoost

CatBoost

FT-Transformer

FT-Transformer

FT-Transformer

FT-Transformer

GradientBoostingTree

GradientBoostingTree

GradientBoostingTree

GradientBoostingTree

GRANDE

GRANDE

GRANDE

GRANDE

HistGradientBoostingTree

HistGradientBoostingTree

HistGradientBoostingTree

HistGradientBoostingTree

LightGBM

LightGBM

LightGBM

LightGBM

MLP

MLP

MLP

MLP

NODE

NODE

NODE

NODE

RandomForest

RandomForest

RandomForest

RandomForest

ResNet

ResNet

ResNet

ResNet

Trompt

Trompt

Trompt

Trompt

XGBoost

XGBoost

XGBoost

XGBoost

0.5

0.6

0.7

0.8

0.9

1

10

100

Number of random search iterations

Normalized test accuracy of best

model (on valid set) up to this iteration

CatBoost

CatBoost

CatBoost

CatBoost

FT-Transformer

FT-Transformer

FT-Transformer

FT-Transformer

GradientBoostingTree

GradientBoostingTree

GradientBoostingTree

GradientBoostingTree

GRANDE

GRANDE

GRANDE

GRANDE

HistGradientBoostingTree

HistGradientBoostingTree

HistGradientBoostingTree

HistGradientBoostingTree

LightGBM

LightGBM

LightGBM

LightGBM

MLP

MLP

MLP

MLP

NODE

NODE

NODE

NODE

RandomForest

RandomForest

RandomForest

RandomForest

ResNet

ResNet

ResNet

ResNet

Trompt

Trompt

Trompt

Trompt

XGBoost

XGBoost

XGBoost

XGBoost

0.5

0.6

0.7

0.8

0.9

1

10

100

Number of random search iterations

Normalized test R2 score of best

model (on valid set) up to this iteration

CatBoost

CatBoost

CatBoost

CatBoost

FT-Transformer

FT-Transformer

FT-Transformer

FT-Transformer

GradientBoostingTree

GradientBoostingTree

GradientBoostingTree

GradientBoostingTree

GRANDE

GRANDE

GRANDE

GRANDE

LightGBM

LightGBM

LightGBM

LightGBM

NODE

NODE

NODE

NODE

RandomForest

RandomForest

RandomForest

RandomForest

ResNet

ResNet

ResNet

ResNet

Trompt

Trompt

Trompt

Trompt

XGBoost

XGBoost

XGBoost

XGBoost

0.5

0.6

0.7

0.8

0.9

1

10

100

Number of random search iterations

Normalized test accuracy of best

model (on valid set) up to this iteration

CatBoost

CatBoost

CatBoost

CatBoost

FT-Transformer

FT-Transformer

FT-Transformer

FT-Transformer

GradientBoostingTree

GradientBoostingTree

GradientBoostingTree

GradientBoostingTree

GRANDE

GRANDE

GRANDE

GRANDE

LightGBM

LightGBM

LightGBM

LightGBM

NODE

NODE

NODE

NODE

ResNet

ResNet

ResNet

ResNet

Trompt

Trompt

Trompt

Trompt

XGBoost

XGBoost

XGBoost

XGBoost

0.5

0.6

0.7

0.8

0.9

1

10

100

Number of random search iterations

Normalized test R2 score of best

model (on valid set) up to this iteration

(a) Medium, Classification. (b) Medium, Regression.

(c) Large, Classification.

(d) Large, Regression.

For deep learning models, the biggest challenge is that generating a sparse matrix for on-off column selection is non-differentiable. In this study, we propose a novel two-step process to work around this issue: (1) enumerating as many sparse selections of columns as possible, and (2) weighting the importance of these sparse selections, making the weights differentiable and trainable by a DNN model. We name this new tree-inspired DNN DOFEN , an abbreviation for D eep O blivious F orest EN semble, and further demonstrate how DOFEN implements these two steps below:

Condition Generation and rODT Construction. In DOFEN, the step of enumerating sparse selections is further divided into two parts. The first part generates conditions, each involving exactly one column and corresponding to a decision rule of a tree node, as described in Section 3.2.1. The second part combines conditions using a shuffle-then-reshape procedure, detailed in Section 3.2.2. Each resultant combination of conditions can be seen as a differentiable counterpart to the Oblivious Decision Tree (ODT) [18], referred to as a relaxed ODT (rODT) in the context. Consequently, all the combinations collectively form a pool of rODTs.

Two-level rODT Ensemble. To ensure that the weighting of a limited number of rODTs can achieve good predictive performance, the previous step requires a sufficiently large pool. However, assembling all the rODTs in the pool into a single giant forest tends to cause overfitting, as shown in Appendix F.1 and Figure 9. Thus, DOFEN implements the step of importance weighting using a two-level ensemble procedure. The first level involves ensembling only a randomly selected subset of the rODT pool to form individual rODT forests, which is similar to applying dropout [19] to the rODT pool. The second level treats each rODT forest as a weak learner and aggregates them into a forest ensemble. This level is designed to enhance performance and stability, similar to standard ensemble learning. Both level of ensemble is detailed in Algorithm 1 of Section 3.2.3.

To evaluate DOFEN comprehensively and objectively, we have chosen a recent and well-recognized benchmark: the Tabular Benchmark [1]. This benchmark addresses the issue of inconsistent dataset selection, which is prevalent in deep learning research on tabular data. It includes a variety of regression and classification datasets with standardized feature processing for consistency. Additionally, we have conducted detailed analyses focusing on the distinct features of DOFEN, thereby offering insights into its functionalities. In summary, our research makes two key contributions:

Innovative Neural Network Architecture. The DOFEN model is fundamentally inspired by ODTs and incorporates an innovative two-step process to achieve on-off sparse selection of columns. This unique approach enhances performance beyond that of current tree-inspired DNNs and offers differentiability compared to conventional tree-based models.

State-of-the-Art Performance. The DOFEN model exhibits outstanding performance, surpassing that of other neural network models and competing closely with GBDTs on the Tabular Benchmark. This achievement underscores its robustness and versatility across various tasks, as illustrated in Figure 1.

2 Related Work

In this section, we start by exploring ODT and detail our rationale for selecting ODT as the foundational element in our study. We then systematically categorize deep tabular neural networks into two distinct streams: tree-inspired DNN architectures and novel DNN architectures. Through comparing DOFEN with these established models, our goal is to highlight its unique contributions and position it within the broad landscape of deep tabular network research.

Oblivious Decision Tree. The ODT is a variant of the traditional decision tree algorithm [20], which makes a series of feature-based decisions along its root-to-leaf path to deliver a prediction. In the context, a feature-based decision rule, e.g. age > 18 , is called a condition. The traditional decision tree algorithm [18] chooses different conditions on different nodes, while in ODT, all nodes at the same level apply the same condition, resulting in a more uniform decision-making process. This uniformity allows for streamlined and vectorized decision-making, thus enhancing computational efficiency, while it also comes at the cost of capacity [8]. However, studies have shown that ensembles of ODTs can achieve remarkable performance with sufficient capacity [6, 8]. In this research, we integrate ODTs as the foundational element in the DOFEN model and capitalize on the strengths of ODTs while mitigating their limitations through ensemble strategies and deep learning techniques.

Tree-inspired DNN Architectures. Integrating decision tree (DT) algorithms with DNNs has become prominent for handling tabular data. Pioneering works like Deep Forest [7], NODE [8], TabNet [9], GradTree [21] and GRANDE [22] have each introduced unique methodologies.

Deep Forest adapts the random forest algorithm and incorporates multi-grained feature scanning to leverage the representation learning capabilities of DNNs. TabNet models the sequential decisionmaking process of traditional decision trees using a DNN, featuring a distinct encoder-decoder architecture that enables self-supervised learning. GradTree recognizes the importance of hard, axisaligned splits for tabular data and uses a straight-through operator to handle the non-differentiable nature of decision trees, allowing for the end-to-end training of decision trees. NODE and GRANDE share a similar observation and high-level structure to DOFEN, in that they ensemble multiple treelike deep learning base models. NODE uses ODT as a base predictor and employs a DenseNet-like multi-layer ensemble to boost performance. GRANDE, a successor to GradTree, uses DT as a base predictor and introduces advanced instance-wise weighting for ensembling each base model's prediction.

However, DOFEN distinguishes itself from NODE and GRANDE through its unique architectural design. First, DOFEN employs a different approach to transform tree-based models into neural networks. Unlike NODE and GRANDE, which explicitly learn the decision paths (i.e., selecting features and thresholds for each node) and the leaf node values of a tree, DOFEN randomly selects features to form rODTs and uses a neural network to measure how well a sample aligns with the decision rule. Additionally, the leaf node value of an rODT is replaced with an embedding vector for further ensembling. Second, DOFEN introduces a novel two-level ensemble process to enhance model performance and stability. Unlike NODE and GRANDE, which simply perform a weighted sum on base model predictions, DOFEN first constructs multiple rODT forests by randomly aggregating selected rODT embeddings and then applies bagging on the predictions of these rODT forests.

Novel DNN Architectures. Beyond merging decision tree algorithms with DNNs, significant progress has been made in developing novel architectures for tabular data. Notable among these are TabTransformer [23], FT-Transformer [11], SAINT [10], TabPFN [24], and Trompt [12]. These models primarily leverage the transformer architecture [25], utilizing self-attention mechanisms to capture complex feature relationships.

TabTransformer applies transformer blocks specifically to numerical features, while FT-Transformer extends this approach to both numerical and categorical features. SAINT enhances the model further by applying self-attention both column-wise and sample-wise, increasing its capacity. TabPFN, a variant of the Prior Fitted Network (PFN) [26], is particularly effective with smaller datasets. Trompt introduces an innovative approach by incorporating prompt learning techniques from natural language processing [27], aiming to extract deeper insights from the tabular data's columnar structure.

These models have demonstrated impressive performance across various studies and benchmarks. As a result, we choose them as our baselines to offer a comprehensive evaluation for deep learning models on tabular data.

3 DOFEN: Deep Oblivious Forest Ensemble

In this section, we begin with discussion about how DOFEN relax an ODT to be differentiable in Section 3.1, and elaborate on the details of the overall architecture design in Section 3.2. In the following figures and equations, three sub-networks-composites of fundamental neural network layers such as linear layers, layer normalization, and dropout-are simplified into symbols ∆ 1 , ∆ 2 , and ∆ 3 for readability. The detailed configurations of these sub-networks can be found in Appendix A.2.

3.1 ODT Relaxation

An ODT operates on an input vector ⃗ x , where ⃗ x ∈ R N col and N col is the number of columns in a tabular dataset, as described in Equation (1). Although these columns can be either numerical or categorical, we focus on real numbers in Equations (2) and (3) to simplify the notations.

Fundamentally, an ODT of depth d is a decision table consisting of d entries [28], as depicted in Equation (2). Here, I j indicates the index of a selected column, and x I j denotes its column value at depth j . The corresponding threshold is denoted by b j , and H denotes the Heaviside function. In practice, the choice of x I j is decided by a predefined criterion, e.g., entropy or Gini impurity. It is possible for a raw column to be selected multiple times at different depths, each with a varying threshold.

Equation (2) involves non-differentiable calculations, including the Heaviside function and the predefined criterion. Consequently, the key to integrating an ODT within a neural network model lies in making the following operations differentiable: selecting columns, deciding thresholds, and modeling H .

To address these challenges, DOFEN proposes a method to relax an ODT, as shown in Equation (3). In DOFEN, the columns of an ODT at different depths are selected randomly. The thresholds and the Heaviside function for column I j are replaced with a sub-network ∆ 1 I j , which employs the sigmoid activation function to create soft conditions. To avoid confusion, we introduce a new term, relaxed ODT (rODT), in this context. This term distinguishes between the original ODT and the relaxed version proposed in this study, which can be integrated to neural networks.

3.2 DOFEN Model

3.2.1 Condition Generation

This module transforms input vector ⃗ x into multiple soft conditions for subsequent modules. The raw input in tabular data comprises a combination of numerical and categorical columns. In this study, a soft condition is defined as a scalar indicating how well a column adheres to a decision rule.

This transformation process creates a matrix M , as shown in Equation (4), where N cond is a hyperparameter denoting the number of conditions we aim to generate for each column. Notably, each column x i is processed by individual sub-network ∆ 1 i in this context, where i ∈ { 1 , . . . , N col } . This design is derived from the original ODT, where each condition involves only a single column. The sub-network ∆ 1 is an embedding layer for a categorical column or a linear layer for a numerical column. Further details of ∆ 1 can be found in Appendix A.2. As depicted in Figure 2a, three instances of ∆ 1 generate four conditions for each column, resulting in a 3 × 4 matrix.

paired

paired

paired

paired

(a) Condition Generation

(b) Relaxed ODT Construction

(c) Forest Construction

paired

paired

Permutation with

and reshape

3.2.2 Relaxed ODT Construction

This module constructs multiple rODTs. Unlike traditional ODT, which selects columns and their corresponding thresholds based on predefined criteria, DOFEN randomly selects d elements from the N cond × N col conditions in matrix M without replacement to build an rODT with depth d . In our implementation, M is shuffled and reshaped into a matrix O with dimensions N rODT × d , as shown in Equation (5). Here, we use π to represent a bijective function that maps the index of each element in M to a unique position in O (i.e. permutation). The whole process is also illustrated in Figure 2b.

Specifically, N rODT = N cond N col /d . To guarantee that N rODT is an integer, we introduce an intermediate parameter, m , which ensures that N cond is always a multiple of d by formulating N cond = md . In practice, we use m to adjust N cond instead of directly changing N cond .

On the other hand, note that each row in O represents an rODT, which is crucial for subsequent operations. To ensure this consistency and the stability during training, the permutation is done only once during model construction and the configuration is then maintained throughout.

average

sum

sample

instances of (

,

)

without replacement and repeat

times

a shared

(a) Forest Construction

(b) Forest Ensemble

= softmax operation

= weighted sum operation

3.2.3 Two-level Relaxed ODT Ensemble

This module integrates rODTs to construct forests and then assembles multiple forests to conduct a final prediction.

Forest Construction. To construct an rODT forest using the generated rODTs, DOFEN introduces a sub-network and a standalone embedding vector for each rODT, denoted as ∆ 2 i and ⃗ e i respectively, where i ∈ { 1 , . . . , N rODT } . The role of ∆ 2 i is to evaluate how well a sample aligns with the conditions of an rODT, producing a weight scalar w i , as shown in Equation (6) and Figure 2c.

The embedding vector ⃗ e i represents the tree information and is independent of the samples. The embedding vectors are combined into a matrix E , as depicted in Equation (7), where N hidden represents the size of the hidden dimension. Importantly, each tree embedding vector is directly linked to the specific conditions of its corresponding rODT. It is crucial to keep this association consistent throughout each training session to effectively train the tree embedding vectors.

To further construct an rODT forest, N estimator of paired weights and embeddings are sampled from ⃗ w and E . This process is graphically represented in Figure 3a and described in line 3 to 7 of the pseudo-code for the two-level ensemble (Algorithm 1). The weights are processed through a softmax function and the weighted sum of embeddings forms the embedding vector f ⃗ for an rODT forest. The magnitude of these softmaxed weights indicate the importance of the selected rODTs for making predictions. Noted that this process is repeated N forest times to form N forest instances of rODT forests.

Forest Ensemble. To make a prediction, DOFEN applies a shared sub-network ∆ 3 to the embedding of each rODT forest to make individual predictions. The predictions are then averaged for a bagging

ensemble. The process is detailed in line 1, 8, 10, and 12 in Algorithm 1 and is illustrated in Figure 3b. Notice that the output ˆ y i is a scalar for regression tasks and a vector for classification tasks.

During training, DOFEN updates the model parameters by aggregating the loss from each prediction, as shown in line 9 in Algorithm 1. The loss function L is cross-entropy for classification tasks and mean squared error for regression tasks.

Notably, the sampling of weight-embedding pairs allows resampling in each forward pass without disrupting the training. In fact, the two-level rODT ensemble essentially implements a form of bootstrap aggregating (i.e. bagging) of trees. Conventional tree-based models like random forest bootstrap samples to generate a variety of trees, which are then combined to form a forest. In DOFEN, the ⃗ w and E represent a tree pool. From this pool, trees are sampled with replacement to create diverse tree sets, or forests, represented by ⃗ w ′ and E ′ . These forests are then integrated to make the final prediction. The design of this tree bagging method enables the construction of varied base models (in this case, forests rather than individual trees) within a single training session, which is particularly suited to deep learning contexts. Although the randomization may seem chaotic, experiments shows that this approach contributes to the model's stability and generalizability, which is discussed in detail in Section 4.3.1 and Appendix F.1.

4 Experiments

This section presents a comprehensive analysis of our experimental results, organized as follows: The Tabular Benchmark and the baseline models are first introduced in Section 4.1. In Section 4.2, we evaluate DOFEN on the medium-sized Tabular Benchmark, while leaving the results for large-sized benchmark in Appendix G.1. Section 4.3 delves into DOFEN to elucidate the underlying mechanics that drive its performance. Additionally, we discuss DOFEN's computational efficiency in Appendices C.1 to C.3, analyze DOFEN's scalability in Appendix D, and show DOFEN's interpretability in Appendix E.

4.1 Tabular Benchmark Setup

Datasets. We strictly follow the protocols of the Tabular Benchmark as detailed in its official implementation 1 . This includes dataset splits, preprocessing methods, hyperparameter search guidelines, and evaluation metrics. For full details, please refer to the original paper [1]. The Tabular Benchmark categorized datasets into classification and regression, with features being either exclusively numerical or a combination of numerical and categorical (heterogeneous). These datasets are further

CatBoost

CatBoost

CatBoost

CatBoost

FT-Transformer

FT-Transformer

FT-Transformer

FT-Transformer

GradientBoostingTree

GradientBoostingTree

GradientBoostingTree

GradientBoostingTree

GRANDE

GRANDE

GRANDE

GRANDE

HistGradientBoostingTree

HistGradientBoostingTree

HistGradientBoostingTree

HistGradientBoostingTree

LightGBM

LightGBM

LightGBM

LightGBM

MLP

MLP

MLP

MLP

NODE

NODE

NODE

NODE

RandomForest

RandomForest

RandomForest

RandomForest

ResNet

ResNet

ResNet

ResNet

Trompt

Trompt

Trompt

Trompt

XGBoost

XGBoost

XGBoost

XGBoost

0.5

0.6

0.7

0.8

0.9

1

10

100

Number of random search iterations

Normalized test accuracy of best

model (on valid set) up to this iteration

(a) Exclusively Numerical,

Classification

CatBoost

CatBoost

CatBoost

CatBoost

FT-Transformer

FT-Transformer

FT-Transformer

FT-Transformer

GradientBoostingTree

GradientBoostingTree

GradientBoostingTree

GradientBoostingTree

GRANDE

GRANDE

GRANDE

GRANDE

HistGradientBoostingTree

HistGradientBoostingTree

HistGradientBoostingTree

HistGradientBoostingTree

LightGBM

LightGBM

LightGBM

LightGBM

MLP

MLP

MLP

MLP

NODE

NODE

NODE

NODE

RandomForest

RandomForest

RandomForest

RandomForest

ResNet

ResNet

ResNet

ResNet

Trompt

Trompt

Trompt

Trompt

XGBoost

XGBoost

XGBoost

XGBoost

0.5

0.6

0.7

0.8

0.9

1

10

100

Number of random search iterations

Normalized test accuracy of best

model (on valid set) up to this iteration

(b) Heterogeneous,

Classification

CatBoost

CatBoost

CatBoost

CatBoost

FT-Transformer

FT-Transformer

FT-Transformer

FT-Transformer

GradientBoostingTree

GradientBoostingTree

GradientBoostingTree

GradientBoostingTree

GRANDE

GRANDE

GRANDE

GRANDE

HistGradientBoostingTree

HistGradientBoostingTree

HistGradientBoostingTree

HistGradientBoostingTree

LightGBM

LightGBM

LightGBM

LightGBM

MLP

MLP

MLP

MLP

NODE

NODE

NODE

NODE

RandomForest

RandomForest

RandomForest

RandomForest

ResNet

ResNet

ResNet

ResNet

Trompt

Trompt

Trompt

Trompt

XGBoost

XGBoost

XGBoost

XGBoost

0.5

0.6

0.7

0.8

0.9

1

10

100

Number of random search iterations

Normalized test R2 score of best

model (on valid set) up to this iteration

(c) Exclusively Numerical,

Regression

CatBoost

CatBoost

CatBoost

CatBoost

FT-Transformer

FT-Transformer

FT-Transformer

FT-Transformer

GradientBoostingTree

GradientBoostingTree

GradientBoostingTree

GradientBoostingTree

GRANDE

GRANDE

GRANDE

GRANDE

HistGradientBoostingTree

HistGradientBoostingTree

HistGradientBoostingTree

HistGradientBoostingTree

LightGBM

LightGBM

LightGBM

LightGBM

MLP

MLP

MLP

MLP

NODE

NODE

NODE

NODE

RandomForest

RandomForest

RandomForest

RandomForest

ResNet

ResNet

ResNet

ResNet

Trompt

Trompt

Trompt

Trompt

XGBoost

XGBoost

XGBoost

XGBoost

0.5

0.6

0.7

0.8

0.9

1

10

100

Number of random search iterations

Normalized test R2 score of best

model (on valid set) up to this iteration

(d) Heterogeneous,

Regression

classified according to their sample size: medium-sized or large-sized. The dataset counts from Tabular Benchmark are provided in Appendix B.1, and the detailed datasets used in Tabular Benchmark are provided in Appendix B.3.

Model Selection. For model comparison, Tabular Benchmark includes four tree-based models: RandomForest, GradientBoostingTree [29], HGBT [30], and XGBoost; two generic DNN models: MLP and ResNet [11]; and two tabular DNN models: SAINT and FT-Transformer. To ensure a comprehensive comparison, we also included two additional tree-based models: LightGBM and CatBoost, and three tabular DNN models: NODE, Trompt, and GRANDE. LightGBM and CatBoost are selected due to their widespread use across various domains. NODE and GRANDE both share similar motivation and high-level structure with DOFEN, while Trompt represents the current stateof-the-art tabular DNNs when following the origin protocols of the Tabular Benchmark. The default hyperparameter configuration of DOFEN and hyperparameter search space of different models are presented in Appendices A.1 and H.2, and the list of some missing model baselines from Tabular Benchmark is provided in Appendix B.2.

4.2 Performance Evaluation

We analyze the results of medium-sized benchmark on classification and regression tasks separately. The evaluation metrics adhere to the Tabular Benchmark protocols, which use accuracy for classification datasets and the R-squared score for regression datasets. We discuss the overall performance in this section and provide comprehensive results for each dataset in Appendix G.2.

Classification. In Figure 4a, the models can be roughly categorized into three groups: (1) tree-based models and three tabular DNN models: DOFEN, Trompt and GRANDE, (2) three other tabular DNN models, and (3) the two generic DNN models. Prior to DOFEN, Trompt was the sole DNN model comparable to tree-based models. DOFEN not only matches but also surpasses the performance of most tree-based models, establishing a new benchmark for DNN models in tabular data. In Figure 4b, DOFEN and Trompt are again the only two DNN models grouped with tree-based models, yet they are positioned at the bottom of this group.

Regression. In Figure 4c, XGBoost stands out as a distinct category. Meanwhile, CatBoost and DOFEN represent a second level of performance. Notably, XGBoost and DOFEN demonstrate a significant improvement during the hyperparameter optimization, whereas CatBoost maintains strong performance consistently. In Figure 4d, XGBoost and CatBoost continue to hold the top two positions. DOFEN, ending up in sixth place, is overtaken by GradientBoostingTree as well as HGBT, and is comparable with FT-Transformer towards the end of the hyperparameter search process.

The analysis of Figure 4 allows us to draw several conclusions. When compared to DNN models, DOFEN consistently either ranks first or shares the top positions. Additionally, DOFEN exhibits strong competitiveness against tree-based models. In datasets with numerical features, it consistently places within the top three. However, in the context of heterogeneous features, DOFEN's performance is moderate, typically falling in the middle or lower tiers in comparison with tree-based models. This challenge in managing heterogeneous features is a prevalent issue among all DNN models, highlighting an area for potential improvement in future tabular DNN models.

4.3 Additional Analysis

This section is dedicated to a deeper exploration of the DOFEN model. Randomness plays an important role in DOFEN, as both the condition selection of an rODT and rODT selection of a forest involve random processes. A straightforward concern is the stability of DOFEN, which is examined in Section 4.3.1. Moreover, given that the conditions are randomly selected, we investigate whether this randomness leads to redundant trees in Section 4.3.2. In addition to randomness, another distinct feature of DOFEN is the introduction of a higher-level ensemble that combines multiple forests, instead of merely assembling trees into a forest. Appendix F.1 discusses the impact of removing this higher-level ensemble on DOFEN.

All experiments in this section are conducted using the default hyperparameters and medium-sized datasets from the Tabular Benchmark. For evaluation metrics, accuracy is used for classification datasets, while the R-squared score for regression datasets. Except for the cases evaluated on individual datasets, the results represent the averaged metrics across the corresponding datasets.

4.3.1 Model Stability

DOFEN incorporates randomness at two steps: firstly, in the selection of conditions as shown in Equation (5) for rODT construction, and secondly, in the sampling of rODTs as shown in line 3 of Algorithm 1 for a two-level rODT ensemble. This section explores how randomness affects the stability of DOFEN.

We start by analyzing the variation in performance of four datasets, where DOFEN ranks first, as shown in Table 1. The standard deviations are even negligible when N forest = 1 (about 0 . 1% to 1% to mean), except for the delays-zurich dataset. Moreover, with increased N forest, the standard deviations become even smaller (about 0 . 01% to 0 . 1% to mean). These results suggest that the stability of DOFEN is not an issue in most cases ( N forest > 10 ), and using the default setting of DOFEN ( N forest = 100 ) ensures both adequate performance and stability for most datasets. Furthermore, the performance improves as the N forest increases, indicating that the tree bagging of DOFEN not only mitigates instability but also enhances the model's generalizability.

In addition to analyzing the intrinsic instability, we also replace the steps involving randomness with deterministic alternatives to assess the impact of the randomness on DOFEN from a different perspective. For the selection of conditions, we utilize CatBoost to choose columns based on a predefined criterion. The detailed results, presented in Appendix F.3, reveal that the predefined criterion perform only slightly better than the shuffle-then-reshape process. Considering the differentiability and the potential for end-to-end training, random selection of conditions remains a viable and promising option. For sampling rODTs, we implement a sliding window technique to lock in the selected trees for each forest. The results are detailed in Appendix F.4, which suggests that our straightforward approach is comparable to a more sophisticated approach.

1

2

3

4

5

6

7

8

9

10

11

12

13

14

15

16

17

18

19

20

21

22

23

24

25

rODT rank

6

4

2

0

2

average rODT weight

TP

TN

(a) Relaxed ODTs with large weight variation.

840

841

842

843

844

845

846

847

848

849

850

851

852

853

854

855

856

857

858

859

860

861

862

863

864

rODT rank

2.0

1.5

1.0

0.5

0.0

average rODT weight

TP

TN

(b) Relaxed ODTs with small weight variation.

4.3.2 Weights of Individual Relaxed ODT

In DOFEN, an rODT is assigned a weight to predict a sample, as shown in Equation (6). In this section, we analyze a binary classification dataset (covertype) to observe the variation in the weights assigned to individual rODTs, as shown in Figure 5.

Figure 5a shows that, for most rODTs ranked in the top 25 according to their standard deviations of weights, there is a significant difference between the average weights of true positives and those of true negatives. Conversely, Figure 5b shows an opposite trend for rODTs with the smallest standard deviations of weights. These trends are also observed in another dataset, as shown in Appendix F.5. These observations imply that rODTs with larger standard deviations of weights is more crucial role in classifying samples.

In addition, we come up with an idea to examine the performance change after pruning weights with small standard deviations and their corresponding embeddings, since they are not sensitive to samples with different label. The results are provided in Appendix F.6 and suggest that the variation serves as a reliable indicator of the importance of rODTs. Moreover, pruning the less important rODTs not only enhances the model's efficiency but also its performance.

5 Limitation and Conclusion

Limitation. Although DOFEN shows promising results, it still contains two weaknesses. First, the inference time of DOFEN is relatively long compared to other DNN models, as shown in Appendix C.1. However, Appendix C.1 also shows that DOFEN possesses the fewest floating point operations (FLOPs). This inconsistency between inference time and FLOPs is mainly caused by the group convolution operation for calculating weights for each rODT (Appendix C.2), which can be improved in the future implementation of DOFEN. Second, the randomization steps involved in DOFEN result in a slower convergence speed, meaning that DOFEN requires more training steps to reach optimal performance. This is reflected in the relatively larger number of training epochs needed for DOFEN. Therefore, the workaround strategy of differentiable sparse selection proposed in this study is merely a starting point, demonstrating its potential. Finding more efficient strategies will be the future work.

Conclusion. In this work, we proposed DOFEN, a novel tree-inspired DNN for tabular data that achieves on-off sparse selections of columns. DOFEN first constructs sufficiently large number of rODTs and randomly ensembles these rODTs into multiple rODT forests to make prediction. DOFEN was evaluated on the Tabular Benchmark, achieving state-of-the-art results compared to DNN-based models and proving competitive with tree-based ones. Furthermore, we showed that the randomization steps involved in DOFEN do not compromise stability but do yield redundant rODTs. Nevertheless, redundant rODTs can be efficiently removed through our pruning method. In summary, based on DOFEN's outstanding performance, it has the potential to serve as the backbone model for tabular data across various scenarios, including self- and semi-supervised learning, as well as multi-modal training.

References

Grinsztajn, L.; Oyallon, E.; Varoquaux, G. Why do tree-based models still outperform deep learning on typical tabular data? Advances in Neural Information Processing Systems 2022 , 35 , 507-520.

Breiman, L. Random forests. Machine learning 2001 , 45 , 5-32.

Geurts, P.; Ernst, D.; Wehenkel, L. Extremely randomized trees. Machine learning 2006 , 63 , 3-42.

Chen, T.; Guestrin, C. Xgboost: A scalable tree boosting system. Proceedings of the 22nd acm sigkdd international conference on knowledge discovery and data mining. 2016; pp 785-794.

Ke, G.; Meng, Q.; Finley, T.; Wang, T.; Chen, W.; Ma, W.; Ye, Q.; Liu, T.-Y. Lightgbm: A highly efficient gradient boosting decision tree. Advances in neural information processing systems 2017 , 30 .

Prokhorenkova, L.; Gusev, G.; Vorobev, A.; Dorogush, A. V.; Gulin, A. CatBoost: unbiased boosting with categorical features. Advances in neural information processing systems 2018 , 31 .

Zhou, Z.-H.; Feng, J. Deep forest. National science review 2019 , 6 , 74-86.

Popov, S.; Morozov, S.; Babenko, A. Neural oblivious decision ensembles for deep learning on tabular data. arXiv preprint arXiv:1909.06312 2019 ,

Arik, S. Ö.; Pfister, T. Tabnet: Attentive interpretable tabular learning. Proceedings of the AAAI conference on artificial intelligence. 2021; pp 6679-6687.

Somepalli, G.; Goldblum, M.; Schwarzschild, A.; Bruss, C. B.; Goldstein, T. Saint: Improved neural networks for tabular data via row attention and contrastive pre-training. arXiv preprint arXiv:2106.01342 2021 ,

Gorishniy, Y.; Rubachev, I.; Khrulkov, V.; Babenko, A. Revisiting deep learning models for tabular data. Advances in Neural Information Processing Systems 2021 , 34 , 18932-18943.

Chen, K.-Y.; Chiang, P.-H.; Chou, H.-R.; Chen, T.-W.; Chang, T.-H. Trompt: Towards a Better Deep Neural Network for Tabular Data. arXiv preprint arXiv:2305.18446 2023 ,

Shwartz-Ziv, R.; Armon, A. Tabular data: Deep learning is not all you need. Information Fusion 2022 , 81 , 84-90.

Borisov, V.; Leemann, T.; Seßler, K.; Haug, J.; Pawelczyk, M.; Kasneci, G. Deep neural networks and tabular data: A survey. IEEE Transactions on Neural Networks and Learning Systems 2022 ,

McElfresh, D.; Khandagale, S.; Valverde, J.; Ramakrishnan, G.; Goldblum, M.; White, C.; others When Do Neural Nets Outperform Boosted Trees on Tabular Data? arXiv preprint arXiv:2305.02997 2023 ,

Peters, B.; Niculae, V.; Martins, A. F. Sparse Sequence-to-Sequence Models. Proc. ACL. 2019.

Martins, A.; Astudillo, R. From softmax to sparsemax: A sparse model of attention and multi-label classification. International conference on machine learning. 2016; pp 1614-1623.

Kohavi, R. Bottom-up induction of oblivious read-once decision graphs. European Conference on Machine Learning. 1994; pp 154-169.

Srivastava, N.; Hinton, G.; Krizhevsky, A.; Sutskever, I.; Salakhutdinov, R. Dropout: a simple way to prevent neural networks from overfitting. The journal of machine learning research 2014 , 15 , 1929-1958.

Quinlan, J. R. Induction of decision trees. Machine learning 1986 , 1 , 81-106.

Marton, S.; Lüdtke, S.; Bartelt, C.; Stuckenschmidt, H. GradTree: Learning axis-aligned decision trees with gradient descent. Proceedings of the AAAI Conference on Artificial Intelligence. 2024; pp 14323-14331.

Marton, S.; Lüdtke, S.; Bartelt, C.; Stuckenschmidt, H. GRANDE: Gradient-Based Decision Tree Ensembles for Tabular Data. The Twelfth International Conference on Learning Representations. 2024.

Huang, X.; Khetan, A.; Cvitkovic, M.; Karnin, Z. Tabtransformer: Tabular data modeling using contextual embeddings. arXiv preprint arXiv:2012.06678 2020 ,

Hollmann, N.; Müller, S.; Eggensperger, K.; Hutter, F. Tabpfn: A transformer that solves small tabular classification problems in a second. arXiv preprint arXiv:2207.01848 2022 ,

Vaswani, A.; Shazeer, N.; Parmar, N.; Uszkoreit, J.; Jones, L.; Gomez, A. N.; Kaiser, Ł.; Polosukhin, I. Attention is all you need. Advances in neural information processing systems 2017 , 30 .

Müller, S.; Hollmann, N.; Arango, S. P.; Grabocka, J.; Hutter, F. Transformers can do bayesian inference. arXiv preprint arXiv:2112.10510 2021 ,

Radford, A.; Narasimhan, K.; Salimans, T.; Sutskever, I.; others Improving language understanding by generative pre-training. 2018 ,

Lou, Y.; Obukhov, M. Bdt: Gradient boosted decision tables for high accuracy and scoring efficiency. Proceedings of the 23rd ACM SIGKDD international conference on knowledge discovery and data mining. 2017; pp 1893-1901.

Friedman, J. H. Stochastic gradient boosting. Computational statistics & data analysis 2002 , 38 , 367-378.

Pedregosa, F. et al. Scikit-learn: Machine Learning in Python. Journal of Machine Learning Research 2011 , 12 , 2825-2830.

Paszke, A.; Gross, S.; Massa, F.; Lerer, A.; Bradbury, J.; Chanan, G.; Killeen, T.; Lin, Z.; Gimelshein, N.; Antiga, L.; others Pytorch: An imperative style, high-performance deep learning library. Advances in neural information processing systems 2019 , 32 .

Loshchilov, I.; Hutter, F. Decoupled weight decay regularization. arXiv preprint arXiv:1711.05101 2017 ,

fvcore library. https://github.com/facebookresearch/fvcore/ .

Averagemn LGBM with hyperopt tuning. 2019; https://www.kaggle.com/code/donkeys/ lgbm-with-hyperopt-tuning/notebook , [Online; accessed 5-January-2023].

Bahmani, M. Understanding LightGBM Parameters (and How to Tune Them). 2022; https: //neptune.ai/blog/lightgbm-parameters-guide , [Online; accessed 5-January-2023].

Vanschoren, J.; van Rijn, J. N.; Bischl, B.; Torgo, L. OpenML: Networked Science in Machine Learning. SIGKDD Explorations 2013 , 15 , 49-60.

Appendix

Table of Contents

A More DOFEN Settings

A.1 Default Hyperparameters Settings for DOFEN

In this section, we describe the hyperparameters used in our DOFEN model, along with their default values, as shown in Table 2. All notations used here have been previously introduced in Section 3, except for dropout_rate. The dropout_rate is applied in dropout layers, and its usage is detailed in Appendix A.2.

The calculated N estimator for each dataset can be found in Appendix A.3. Additionally, the hyperparameter search spaces for both the DOFEN model and all baseline models are detailed in Appendix H.2.

DOFEN is implemented in Pytorch [31]. For hyperparameters used in model optimization (e.g. optimizer, learning rate, weight decay, etc.), all experiments share the same settings. Specifically, DOFEN uses AdamW optimizer [32] with 1e -3 learning rate and no weight decay. The batch size is set to 256, and DOFEN is trained for 500 epochs without using learning rate scheduling or early stopping.

1 depth of a rODT

A.2 Detailed Model Configurations.

In this appendix, we elucidate the specific configurations of the neural network layer composites, denoted as ∆ 1 , ∆ 2 , and ∆ 3 in the main paper.

∆ 1 - Generate conditions for each column: ∆ 1 is designed to generate conditions for both numerical and categorical data columns, as detailed in Figure 6. For categorical columns in particular, we employ embedding layers. These layers are utilized to transform categorical features into a format that the neural network can effectively process.

∆ 2 and ∆ 3 - Derive weights and make predictions: The layers represented by ∆ 2 and ∆ 3 are responsible for generating weights based on the combination of conditions and making predictions, respectively. The relevant structures and processes are illustrated in Figure 7 and Figure 8.

Key Parameters:

num_categories : This parameter represents the number of distinct categories in a given categorical column.

drop_rate: This hyperparameter defines the extent of dropout operations applied within the network.

A.3 Actual N estimator for each Dataset

The N estimator is calculated through a pre-defined formula as shown in Table 2. In this section, we provide the calculated N estimator for each dataset in Table 3 when using default hyperparameters. Datasets are represented by their OpenML ID as described in Appendix B.3.

Linear(1,

)

Linear (1, cond_per_column)

LayerNorm(

)

Embedding(num_categories,

)

Linear(

,

)

Numerical

Column

Categorical

Column

Linear(

,

)

LayerNorm(

)

Dropout(drop_rate)

ReLU()

Linear(

, 1)

LayerNorm(

)

Dropout(drop_rate)

Linear(

,

)

LayerNorm(

)

Dropout(drop_rate)

ReLU()

Linear(

, 1 or

)

LayerNorm(

)

Dropout(drop_rate)

B More Tabular Benchmark Settings

B.1 Dataset Counts

In this section, we provide the dataset counts for each task for your reference, as presented in Table 4.

B.2 Missing Model Baselines

We found that two baselines, MLP and HGBT, are absent from the evaluation results in the large-sized classification task because they are missing from the official repository. Furthermore, MLP, HGBT, and RandomForest are not included in the large-sized regression task for the same reason.

B.3 Mappings of OpenML Task ID and Dataset Name

In this section, we introduce the mappings between OpenML Task IDs and elaborate on how to download the corresponding datasets using these IDs.

The mappings are provided in Tables 5 to 8. To access the datasets, please follow the links below, which direct you to the OpenML website for each type of dataset. You can then search using the OpenML ID.

Classification datasets with numerical features only: https://www.openml.org/search?type=benchmark&study_type=task&id=337

Classification datasets with heterogeneous features: https://www.openml.org/search?type=benchmark&study_type=task&id=334

Regression datasets with numerical features only: https://www.openml.org/search?type=benchmark&study_type=task&id=336 https://www.openml.org/search?type=benchmark&study_type=task&id=297 (only for task ID 361091)

Regression datasets with heterogeneous features: https://www.openml.org/search?type=benchmark&study_type=task&id=335 https://www.openml.org/search?type=benchmark&study_type=task&id=299 (only for task ID 361095)

C Computational Efficiency Analysis

C.1 Computational Efficiency Analysis

To discuss the computational efficiency, we analyzed the average floating point operations (FLOPs) [33], parameter sizes, and inference time of DOFEN and other baseline models. Our analyses covered both the default and optimal hyperparameter settings, where the optimal hyperparameter delivers the best performance for each model on each dataset. The experiments involving DNN-based models were performed using an NVIDIA GeForce RTX 2080 Ti, while those for the GBDT-based models utilized an AMD EPYC 7742 64-core Processor with 16 threads.

We begin with the comparison between DNN-based and GBDT-based models. This comparison primarily focuses on inference time, as FLOPs and parameter sizes are applicable for evaluating the efficiency of DNN-based models but cannot be applied to GBDTs. Additionally, inference times under the optimal parameters are provided only when those parameters are available. As shown in Tables Table 9 to Table 12, the inference times for all DNN-based models are slower than those for GBDT-based models. This is expected due to the inherent differences between the two types of models.

When compared to other DNN baselines, DOFEN achieves the highest performance, the lowest FLOPs, and the smallest parameter sizes but exhibits the relatively long inference time among all the DNN-based models. This inconsistency between FLOPs and inference time suggests that there is still room for implementation improvements in DOFEN. Hence, we conduct additional experiments to analyze which part of the DOFEN model is the computational bottleneck, as discussed in Appendix C.2, showing that the bottleneck of DOFEN arises from using group operations when constructing rODTs. Although this does not affect DOFEN's article, improvements can be made during future open-source releases.

C.2 Long Inference Time of DOFEN

To find out the computation bottleneck of DOFEN, we analyzed the inference time of each DOFEN module in proportion, as shown in Table 13 and Table 14, which is averaged across 59 medium-sized

datasets with default hyperparameters. Table A1 shows that the Forest Construction module consumes the most inference time. In Table A2, more detailed operations reveal that the sub-module ∆ 2 in the Forest Construction module, which generates weights for each rODT, has the longest inference time.

The sub-module ∆ 2 is designed with multiple MLP and normalization layers, implemented using group convolution and group normalization to parallelize scoring for each rODT. However, the efficiency of group convolution in PyTorch has been problematic and remains unresolved. Specifically, the operation efficiency decreases as the number of groups increases, sometimes making it slower than separate convolutions in CUDA streams (see PyTorch issues 18631, 70954, 73764). The sub-module ∆ 1 also uses group convolution to parallelize condition generation across different numerical columns, resulting in slower inference times compared to other operations, though less significant than ∆ 2 due to fewer groups being used.

However, we mainly focus on the concept and model structure in this paper, acknowledging that model implementation can be further optimized. For example, attention operations are originally slow due to quadratic complexity, and many recent works have successfully accelerated the speed of attention operations and reduced their memory usage. Hence, we believe there will be better implementations of these group operations with much greater efficiency in the future.

C.3 Training Time of DOFEN

To know more about how the slow inference time will affect the training time of DOFEN, we also conducted an experiment to compare the training time of DOFEN with other deep learning methods included in our paper (i.e. Trompt, FT Transformer, and NODE). We measured the training time on medium-sized datasets using both default and optimal hyperparameter settings, where the optimal hyperparameters refers to the settings that deliver the best performance for each model on each dataset.

This experiment was conducted using a single NVIDIA Tesla V100 GPU. During model training, we carefully ensured that no other computational processes were running concurrently to enable a fair comparison. Additionally, we excluded datasets that would cause OOM (Out of Memory) issues during training, resulting in the selection of 50 out of 59 medium-sized datasets.

The average training time across datasets for each model is provided in Table A7. The results show that the training time for DOFEN is relatively long, approximately twice as long as Trompt when

using optimal hyperparameters. This extended training time may be due to the inefficient group operations involved in DOFEN, which consume about 80% of the computation time during the forward pass. For more details, please refer to Appendix C.2. Therefore, improving the efficiency of group operations could reduce both the training and inference time of DOFEN.

D Scalability of DOFEN

To discuss the scalability of DOFEN, we have conducted experiments to investigate its performance given changes in hyperparameters m , d , and the number of MLP layers ( num _ layers ). In detail, changes in m and d affect the number of conditions ( N cond), while alterations in m impact both the total number of rODTs ( N rODT) and the number of rODTs within an rODT forest ( N estimator). For further details on these parameters, please refer to Table 2. The num _ layers hyperparameter, newly introduced, refers to the number of MLP layers in neural networks ∆ 1 , ∆ 2 , and ∆ 3 . A detailed introduction to ∆ 1 , ∆ 2 , and ∆ 3 can be found in Appendix A.2.

Due to limited computational resources, we only conducted this experiment on datasets that would not cause out-of-memory (OOM) issues on our machine across all hyperparameter settings. This selection resulted in 51 out of 59 medium-sized datasets and 10 out of 14 large-sized datasets.

Based on Table 16 to Table 21, we observed that larger values of m and d enhance DOFEN's performance. Notably, improvements are more significant with large-sized datasets than with mediumsized datasets, likely because larger datasets benefit more from increased model capacity. In contrast, Table 20 reveals that an increase in num _ layers generally results in poorer performance. This could be attributed to the substantial growth in parameter size and FLOPs, compared to adjustments in the m and d , potentially leading to overfitting.

E Interpretability of DOFEN

This section aims to demonstrate the interpretability of DOFEN. Specifically, we adopt a feature importance metric akin to the "split" or "weight" importance used in LightGBM and XGBoost, which counts how often a feature is used in the model.

To calculate DOFEN's feature importance of a specific sample, let F ∈ R N rODT × N col be a matrix of feature occurrences across different rODTs. We then use the output of sub-module ∆ 2 , a vector ⃗ w ∈ R N rODT (Equation (6)), to represent the importance across all rODTs for each sample, as this weight ⃗ w is used for constructing rODT forest to perform prediction in DOFEN model. A softmax

operation is further applied to the vector ⃗ w to ensure the importance sums to 1 (also done in line 6 of Algorithm 1). Finally, we perform a weighted sum between the feature occurrences and the importance of each rODT, resulting in a single vector ⃗ t ∈ R N col representing DOFEN's feature importance for a specific sample. To calculate DOFEN's overall feature importance of a dataset, we simply average the feature importance of all samples in training dataset.

We tested the reliability of DOFEN's feature importance on three real-world datasets: the mushroom dataset, the red wine quality dataset, and the white wine quality dataset, following the experimental design used by Trompt. The results of these three datasets are shown in Tables 22 to 24, respectively. The results indicate that the top-3 important features identified by DOFEN align closely with those selected by other tree-based models, with only minor ranking differences. This demonstrates DOFEN's ability to reliably identify key features while maintaining interpretability despite its deep learning architecture. This further indicates that DOFEN may contain similar decision-making process as tree-based model does, as it is a tree-inspired deep neural network.

F More Analysis

F.1 Sampling in Relaxed ODT Forest Ensemble

The forest ensemble in DOFEN is a level higher than common tree-based models. This section attempts to explore its impact on model performance by ablating this higher-level ensemble. The experiment involves using all the constructed rODTs from Equation (5) to form a single forest, without the sampling of weights and embeddings as described in line 6 in Algorithm 1. In practice,

we directly apply a softmax function to the weight vector ⃗ w and calculate the weighted sum of corresponding embeddings E . As a result, there is only one prediction for each sample, unlike N forest predictions as in Algorithm 1.

From the results in Table 25, it can be seen that with the ensemble of forests, the performance of DOFEN is improved across all datasets, independent of the types of tasks and features. The average improvement in classification datasets has reached 0.0363 in accuracy, while in regression datasets has reached 0.3367 in the R-squared score.

To further investigate the drastic drop in performance without using sampling in forest ensembles, we analyze performance at various training checkpoints. As shown in Figure 9, when sampling is not used in forest ensembles, training performance is significantly better compared to testing performance, and testing performance decreases with increasing training epochs, indicating an overfitting issue. Conversely, with an ensemble of multiple forests, both training and testing performance improve concurrently, thus mitigating the overfitting issue.

Based on the performance improvements from introducing a forest ensemble, we have applied an extra level of ensemble that combines multiple DOFEN models. However, the enhancement in performance is negligible. The detailed results can be found in Appendix F.2.

100

200

300

400

500

Number of Training Epochs

0.75

0.80

0.85

0.90

0.95

(a) Classification

100

200

300

400

500

Number of Training Epochs

0.6

0.7

0.8

0.9

R2 Score

(b) Regression

train, w/

test, w/

train, w/o

test, w/o

F.2 Seed Ensemble

In this section, we explore the results of applying an additional layer of bagging ensemble to the DOFEN model, a technique we denote as the seed ensemble. Specifically, we infer the trained DOFEN model using 15 different random seeds. This results in 15 distinct predictions due to the random sampling procedure conducted in the rODT Forest Construction module. Building upon these predictions, we average the 15 different predictions to create another layer of bagging ensemble, which we then present as the final prediction. This experiment is conducted on a medium-sized tabular benchmark. As shown in Table 26, this seed ensemble approach further enhances the performance of the DOFEN model, even with a small N forest. The results further suggest that the DOFEN model can easily benefit from ensemble strategies, thanks to the random sampling procedure.

F.3 An Alternative Strategy for Condition Selection

In the rODT Construction module, we implement a shuffle-then-reshape procedure to construct rODTs as outlined in Equation (5). The shuffle of matrix M serves as a straightforward approach to delivering a diverse set of condition combinations for subsequent segmentation. In this section, we aim to experiment with an alternative strategy for selecting columns to construct rODTs, which we adopt the column selection strategy used by CatBoost to form the rODTs in our DOFEN model, we denote this approach as 'Catboost-Init'. Specifically, we begin by selecting a machine learning algorithm that also employs ODT as the base element, namely, CatBoost. Subsequently, we train the CatBoost model and use the columns it identifies post-training to construct our rODTs. In the context of experimental configurations, to ensure a fair comparision, it is crucial to equate the capacity of CatBoost model with the default settings of DOFEN. To achieve this, we set the depth and number of boosting iterations of Catboost as the depth of an rODT ( d ) and the total number of rODT ( N rODT) in DOFEN, respectively. The CatBoost trained based on these configurations is denoted as 'CatBoost*' in the context.

The results are presented in Table 27. As can be seen, the 'Catboost-Init' approach achieves performance comparable to the 'CatBoost*' approach and generally outperforms the default shuffle approach. This indicates that designing a more sophisticated approach for rODT construction indeed contributes to better performance. However, adopting a selection strategy from a tree-based model results in a two-stage modeling process, which contradicts our goal of designing an end-to-end differentiable DNN model. This intriguing insight leads us to consider a more innovative end-to-end condition selection approach, which we will explore in future work.

F.4 An Alternative Strategy for Weight Selection

In the Two-level rODT Ensemble module, our default method involves random sampling of weights. We contrast this approach with an alternative weight selection strategy, denoted as sliding window selection. The method creates N forest windows, each containing N estimator weights. These windows are then evenly distributed across N rODT weights. Within each window, the weights are collectively treated as a rODT forest. As shown in Table 28, our results indicate that random sampling yields better performance compared to the sliding window selection. This finding substantiates that the random sampling approach already works well compared to a sophisticated alternative. Investigating a more advanced and effective approach is worth considering in future work.

F.5 More Experiments for Section 4.3.2 (Activated rODT for Different Classes)

In this section, we aim to test whether the findings presented in Section 4.3.2 can be replicated on another dataset. Consequently, we have chosen another binary classification dataset (compass-twoyears) and repeated the experiment using the same settings. The experimental results are depicted in Figure 10. As shown in Figure 10a, weights with higher standard deviation exhibit distinct distributions between true positive and true negative samples. Conversely, Figure 10b illustrates an opposite trend. Overall, conducting the experiment on another dataset further validates our findings.

1

2

3

4

5

6

7

8

9

10

11

12

13

14

15

16

17

18

19

20

21

22

23

24

25

rODT rank

3

2

1

0

1

2

average rODT weight

TP

TN

(a) Relaxed ODTs with large weight variation.

152

153

154

155

156

157

158

159

160

161

162

163

164

165

166

167

168

169

170

171

172

173

174

175

176

rODT rank

1.2

1.0

0.8

0.6

0.4

0.2

0.0

0.2

average rODT weight

TP

TN

(b) Relaxed ODTs with small weight variation.

Figure 10: In the compass dataset, the weights w i of rODT are sorted based on the standard deviation calculated across true positive (TP) and true negative (TN) samples in the testing data. Figure 10a shows that the weights of TP samples differ significantly from those of TN samples when the standard deviation of the weights is higher. Conversely, Figure 10b reveals contrasting results for weights with a lower standard deviation.

F.6 Pruning of Relaxed ODT

Following Section 4.3.2, in this section, we aim to examine the performance change after pruning weights with small standard deviations and their corresponding embeddings.

Table 29 shows the performance under different pruning ratios. The column labeled 'by dataset' indicates that we tailored the pruning ratio for each dataset based on its validation data. As shown in Table 29, pruning these rODTs does not negatively affect performance. In fact, a minor degree of pruning can actually enhance performance, with the optimal pruning ratio being 0.02 for classification datasets and 0.1 for regression datasets. Notice that the 'by dataset' approach is better suited to real-world scenarios, even though it does not always yield the best performance.

We then investigate the outcomes when weights with higher standard deviations are pruned. Consequently, we sort the weights and prune them from the higher end. The results, presented in Table 30, show that the performance in both classification and regression tasks monotonically drops as the prune ratio increases. This finding suggests that the standard deviation of weights is a good indicator of their importance in making predictions. It further validates why pruning weights with lower standard deviation does not harm performance and, in some cases, even helps.

In addition, we discuss another, potentially more straightforward, pruning approach. Specifically, we prune the weights w i based on their average value across samples. Similar to the experiments that use standard deviation as the metric for pruning, this time we sort the weights by their average. We then attempt to prune the weights from both the top and bottom ends. The results are provided in Table 31 and Table 32, suggesting that the value of weights is not an effective indicator for pruning. Although there is some improvement in performance at a low ratio, this approach generally diminishes performance with larger ratios, regardless of whether the weights are pruned from the higher or lower end.

G More Evaluation Results on Tabular Benchmark

G.1 Performance Evaluation on Large-sized Benchmark

This section discusses the evaluation results on large-sized classification and regression tasks. Overall, the results demonstrate a similar trend as the medium-sized tabular benchmark. Notably, DOEFN achieves the top ranks in both tasks with numerical features.

CatBoost

CatBoost

CatBoost

CatBoost

CatBoost

CatBoost

CatBoost

FT-Transformer

FT-Transformer

FT-Transformer

FT-Transformer

FT-Transformer

FT-Transformer

FT-Transformer

GradientBoostingTree

GradientBoostingTree

GradientBoostingTree

GradientBoostingTree

GradientBoostingTree

GradientBoostingTree

GradientBoostingTree

GRANDE

GRANDE

GRANDE

GRANDE

GRANDE

GRANDE

GRANDE

LightGBM

LightGBM

LightGBM

LightGBM

LightGBM

LightGBM

LightGBM

NODE

NODE

NODE

NODE

NODE

NODE

NODE

RandomForest

RandomForest

RandomForest

RandomForest

RandomForest

RandomForest

RandomForest

ResNet

ResNet

ResNet

ResNet

ResNet

ResNet

ResNet

Trompt

Trompt

Trompt

Trompt

Trompt

Trompt

Trompt

XGBoost

XGBoost

XGBoost

XGBoost

XGBoost

XGBoost

XGBoost

0.5

0.6

0.7

0.8

0.9

1

10

100

Number of random search iterations

Normalized test accuracy of best

model (on valid set) up to this iteration

(a) Exclusively Numerical

CatBoost

CatBoost

CatBoost

CatBoost

CatBoost

CatBoost

CatBoost

FT-Transformer

FT-Transformer

FT-Transformer

FT-Transformer

FT-Transformer

FT-Transformer

FT-Transformer

GradientBoostingTree

GradientBoostingTree

GradientBoostingTree

GradientBoostingTree

GradientBoostingTree

GradientBoostingTree

GradientBoostingTree

GRANDE

GRANDE

GRANDE

GRANDE

GRANDE

GRANDE

GRANDE

LightGBM

LightGBM

LightGBM

LightGBM

LightGBM

LightGBM

LightGBM

NODE

NODE

NODE

NODE

NODE

NODE

NODE

RandomForest

RandomForest

RandomForest

RandomForest

RandomForest

RandomForest

RandomForest

ResNet

ResNet

ResNet

ResNet

ResNet

ResNet

ResNet

Trompt

Trompt

Trompt

Trompt

Trompt

Trompt

Trompt

XGBoost

XGBoost

XGBoost

XGBoost

XGBoost

XGBoost

XGBoost

0.5

0.6

0.7

0.8

0.9

1

10

100

Number of random search iterations

Normalized test accuracy of best

model (on valid set) up to this iteration

(b) Heterogeneous

Classification. In Figure 11a, DOFEN even surpasses CatBoost to become the top performer. Conversely, in Figure 11b, CatBoost clearly outperforms other models. FT-Transformer, Trompt and DOFEN are the best-performing tabular DNN models, though they rank in the middle among all models. As a result, with the current development of tabular DNN models, their performance in processing numerical features is already on par with or even surpass that of tree-based models, and they are more advantageous for large-sized datasets. However, DOFEN and other DNN models are still less efficient in handling heterogeneous features.

CatBoost

CatBoost

CatBoost

CatBoost

CatBoost

CatBoost

CatBoost

FT-Transformer

FT-Transformer

FT-Transformer

FT-Transformer

FT-Transformer

FT-Transformer

FT-Transformer

GradientBoostingTree

GradientBoostingTree

GradientBoostingTree

GradientBoostingTree

GradientBoostingTree

GradientBoostingTree

GradientBoostingTree

GRANDE

GRANDE

GRANDE

GRANDE

GRANDE

GRANDE

GRANDE

LightGBM

LightGBM

LightGBM

LightGBM

LightGBM

LightGBM

LightGBM

NODE

NODE

NODE

NODE

NODE

NODE

NODE

ResNet

ResNet

ResNet

ResNet

ResNet

ResNet

ResNet

Trompt

Trompt

Trompt

Trompt

Trompt

Trompt

Trompt

XGBoost

XGBoost

XGBoost

XGBoost

XGBoost

XGBoost

XGBoost

0.5

0.6

0.7

0.8

0.9

1

10

100

Number of random search iterations

Normalized test R2 score of best

model (on valid set) up to this iteration

(a) Large; Numerical

CatBoost

CatBoost

CatBoost

CatBoost

CatBoost

CatBoost

CatBoost

FT-Transformer

FT-Transformer

FT-Transformer

FT-Transformer

FT-Transformer

FT-Transformer

FT-Transformer

GradientBoostingTree

GradientBoostingTree

GradientBoostingTree

GradientBoostingTree

GradientBoostingTree

GradientBoostingTree

GradientBoostingTree

GRANDE

GRANDE

GRANDE

GRANDE

GRANDE

GRANDE

GRANDE

LightGBM

LightGBM

LightGBM

LightGBM

LightGBM

LightGBM

LightGBM

NODE

NODE

NODE

NODE

NODE

NODE

NODE

ResNet

ResNet

ResNet

ResNet

ResNet

ResNet

ResNet

Trompt

Trompt

Trompt

Trompt

Trompt

Trompt

Trompt

XGBoost

XGBoost

XGBoost

XGBoost

XGBoost

XGBoost

XGBoost

0.5

0.6

0.7

0.8

0.9

1

10

100

Number of random search iterations

Normalized test R2 score of best

model (on valid set) up to this iteration

(b) Large; Heterogeneous

Regression. In Figure 12a, the leading models remain DOFEN, XGBoost, and CatBoost. DOFEN's proficiency in handling numerical features, further enhanced by the increased data volume, enables it to secure the top position once again. In Figure 12b, DOFEN and Trompt barely maintain their positions within the leading group, yet they still stand out from the other DNN models.

G.2 Detailed Evaluation Results

In the main paper, we have discussed the overall performance of DOFEN. To simplify tables, we map dataset names with their OpenML ID, as described in Appendix B.3. The evaluation results of each task are organized in Table 33. Please refer to the detailed figures and tables for each task of your interest. The evaluation metrics are accuracy for classification tasks and R 2 score for regression tasks, consistent with our main paper. Furthermore, we calculate the mean and standard deviation of ranks across datasets to provide the rank for each model in the tables.

MagicTelescope

MiniBooNE

pol

eye_movements

heloc

Higgs

house_16H

credit

default-of-credit-card-clients

Diabetes130US

electricity

bank-marketing

Bioresponse

california

covertype

1

10

100

1

10

100

1

10

100

1

10

100

1

10

100

1

10

100

1

10

100

1

10

100

1

10

100

1

10

100

1

10

100

1

10

100

1

10

100

1

10

100

1

10

100

1

10

100

0.78

0.80

0.82

0.84

0.82

0.84

0.86

0.875

0.880

0.885

0.890

0.93

0.95

0.97

0.87

0.88

0.89

0.90

0.56

0.57

0.58

0.59

0.60

0.68

0.69

0.70

0.71

0.920

0.925

0.930

0.935

0.940

0.74

0.76

0.78

0.80

0.69

0.70

0.71

0.72

0.70

0.71

0.72

0.85

0.86

0.87

0.790

0.795

0.800

0.805

0.810

0.75

0.76

0.77

0.78

0.575

0.600

0.625

0.650

0.74

0.75

0.76

0.77

0.78

Number of random search iterations

Test accuracy of best model

(on valid set) up to this iteration

CatBoost

FT-Transformer

GradientBoostingTree

GRANDE

HistGradientBoostingTree

LightGBM

MLP

NODE

RandomForest

ResNet

Trompt

XGBoost

electricity

eye_movements

road-safety

compas-two-years

covertype

default-of-credit-card-clients

1

10

100

1

10

100

1

10

100

1

10

100

1

10

100

1

10

100

1

10

100

0.700

0.705

0.710

0.715

0.720

0.80

0.82

0.84

0.86

0.88

0.75

0.76

0.77

0.62

0.64

0.66

0.68

0.600

0.625

0.650

0.62

0.63

0.64

0.65

0.66

0.83

0.85

0.87

Number of random search iterations

Test accuracy of best model

(on valid set) up to this iteration

CatBoost

FT-Transformer

GradientBoostingTree

GRANDE

HistGradientBoostingTree

LightGBM

MLP

NODE

RandomForest

ResNet

Trompt

XGBoost

superconduct

wine_quality

yprop_4_1

MiamiHousing2016

nyc-taxi-green-dec-2016

pol

sulfur

house_16H

house_sales

houses

medical_charges

cpu_act

delays_zurich_transport

diamonds

elevators

abalone

Ailerons

Bike_Sharing_Demand

Brazilian_houses

1

10

100

1

10

100

1

10

100

1

10

100

1

10

100

1

10

100

1

10

100

1

10

100

1

10

100

1

10

100

1

10

100

1

10

100

1

10

100

1

10

100

1

10

100

1

10

100

1

10

100

1

10

100

1

10

100

0.00

0.25

0.50

0.75

1.00

0.7

0.8

0.9

0.00

0.25

0.50

0.75

1.00

0.70

0.75

0.80

0.85

0.90

0.0

0.2

0.4

0.6

0.00

0.25

0.50

0.75

0.00

0.25

0.50

0.75

0.00

0.25

0.50

0.75

1.00

0.000

0.025

0.050

0.075

0.00

0.25

0.50

0.75

0.00

0.01

0.02

0.03

0.00

0.25

0.50

0.75

0.2

0.3

0.4

0.5

0.0

0.1

0.2

0.3

0.4

0.5

0.0

0.2

0.4

0.6

0.00

0.25

0.50

0.75

1.00

0.0

0.2

0.4

0.6

0.00

0.25

0.50

0.75

0.00

0.25

0.50

0.75

Number of random search iterations

Test R2 score of best model

(on valid set) up to this iteration

CatBoost

FT-Transformer

GradientBoostingTree

GRANDE

HistGradientBoostingTree

LightGBM

MLP

NODE

RandomForest

ResNet

Trompt

XGBoost

visualizing_soil

particulate-matter-ukair-2017

seattlecrime6

SGEMM_GPU_kernel_performance

topo_2_1

house_sales

medical_charges

Mercedes_Benz_Greener_Manufacturingnyc-taxi-green-dec-2016

Bike_Sharing_Demand

Brazilian_houses

delays_zurich_transport

diamonds

abalone

Airlines_DepDelay_1M

Allstate_Claims_Severity

analcatdata_supreme

1

10

100

1

10

100

1

10

100

1

10

100

1

10

100

1

10

100

1

10

100

1

10

100

1

10

100

1

10

100

1

10

100

1

10

100

1

10

100

1

10

100

1

10

100

1

10

100

1

10

100

0.94

0.96

0.98

0.00

0.25

0.50

0.75

1.00

0.2

0.4

0.6

0.00

0.02

0.04

0.06

0.0

0.2

0.4

0.00

0.02

0.04

0.06

0.08

0.0

0.2

0.4

0.6

0.00

0.25

0.50

0.75

1.00

0.00

0.01

0.02

0.03

0.04

0.05

0.00

0.25

0.50

0.75

1.00

0.00

0.25

0.50

0.75

1.00

0.00

0.05

0.10

0.15

0.0

0.2

0.4

0.6

0.00

0.25

0.50

0.75

0.00

0.25

0.50

0.75

0.0

0.2

0.4

0.6

0.00

0.25

0.50

0.75

1.00

Number of random search iterations

Test R2 score of best model

(on valid set) up to this iteration

CatBoost

FT-Transformer

GradientBoostingTree

GRANDE

HistGradientBoostingTree

LightGBM

MLP

NODE

RandomForest

ResNet

Trompt

XGBoost

(on valid set) up to this iteration

covertype

Higgs

MiniBooNE

1

10

100

1

10

100

1

10

100

1

10

100

0.93

0.94

0.95

0.75

0.77

0.79

0.70

0.71

0.72

0.73

0.80

0.84

0.88

Number of random search iterations

Test accuracy of best model

CatBoost

FT-Transformer

GradientBoostingTree

GRANDE

LightGBM

NODE

RandomForest

ResNet

Trompt

XGBoost

(on valid set) up to this iteration

covertype

road-safety

1

10

100

1

10

100

0.75

0.76

0.77

0.78

0.79

0.80

0.80

0.85

0.90

Number of random search iterations

Test accuracy of best model

CatBoost

FT-Transformer

GradientBoostingTree

GRANDE

LightGBM

NODE

RandomForest

ResNet

Trompt

XGBoost

(on valid set) up to this iteration

diamonds

nyc-taxi-green-dec-2016

year

1

10

100

1

10

100

1

10

100

0.15

0.20

0.25

0.30

0.0

0.2

0.4

0.6

0.825

0.850

0.875

0.900

0.925

0.950

Number of random search iterations

Test R2 score of best model

CatBoost

FT-Transformer

GradientBoostingTree

GRANDE

LightGBM

NODE

ResNet

Trompt

XGBoost

SGEMM_GPU_kernel_performance

black_friday

diamonds

nyc-taxi-green-dec-2016

particulate-matter-ukair-2017

1

10

100

1

10

100

1

10

100

1

10

100

1

10

100

0.64

0.66

0.68

0.70

0.0

0.2

0.4

0.6

0.900

0.925

0.950

0.975

0.4

0.5

0.6

0.92

0.94

0.96

0.98

1.00

Number of random search iterations

Test R2 score of best model

(on valid set) up to this iteration

CatBoost

FT-Transformer

GradientBoostingTree

GRANDE

LightGBM

NODE

ResNet

Trompt

XGBoost

H More Experiment Settings

H.1 Hardware Used

The following hardware configuration was used for all of our experiments. The hardware selection was based on availability, with neural networks consistently run on GPUs and tree-based models executed on CPUs.

GPUs: NVIDIA GeForce RTX 2080 Ti, NVIDIA DGX1, NVIDIA A100

CPUs: Intel(R) Xeon(R) Silver 4210 CPU, Intel(R) Xeon(R) CPU E5-2698 v4, AMD EPYC605 7742 64-core Processor

H.2 Hyperparameter Search Space

This section details the hyperparameter search space adopted for each model, as referenced in various tables (Tables 45 to 56). We have employed search spaces consistent with those presented in the Tabular Benchmark [1] for models including XGBoost, GradientBoostingTree, RandomForest, FT-Transformer, SAINT, ResNet, and MLP.

Additionally, we have defined specific search spaces for newer baselines such as CatBoost, LightGBM, Trompt, NODE, and GRANDE. For CatBoost, our search space aligns with the parameters specified by the FT-Transformer study [11]. In the case of LightGBM, we have derived the search space based on recommendations from field practitioners, as cited in [34, 35]. For NODE, our approach follows the guidelines provided in TabZilla [15]. For GRANDE, we follow the settings provided in the notebook example from the official github of GRANDE.

In the context of our model, DOFEN, we have focused our search on the number of m and d , which relate to the varied number of N cond and the conditions per rODT. Additionally, we have explored the drop_rate parameter to fine-tune the degree of regularization in our model. It is important to note that the overall search space for DOFEN is relatively compact when compared to the other baseline models while achieve competitive performance.

NeurIPS Paper Checklist

1. Claims

Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?

Answer: [Yes]

Justification: We have verified our idea using experiments as shown in Section 4. The results not only reveal the state-of-the-art performance of DOFEN among deep neural networks on tabular data but also demonstrate the nuance of its architecture design, which echos the contribution and scope we mentioned in the abstract and introduction.

Guidelines:

The answer NA means that the abstract and introduction do not include the claims made in the paper.

The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.

The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.

It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.

2. Limitations

Question: Does the paper discuss the limitations of the work performed by the authors?

Answer: [Yes]

Justification: See Section 5 for limitations.

Guidelines:

The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.

The authors are encouraged to create a separate "Limitations" section in their paper.

The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.

The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.

The authors should reflect on the factors that influence the performance of the approach. For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.

The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.

If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.

While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.

3. Theory Assumptions and Proofs

Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?

Answer: [NA]

Justification: This paper focuses on model architecture design and empirical evaluations based on hypotheses, without making theoretical assumptions or providing proofs.

Guidelines:

The answer NA means that the paper does not include theoretical results.

All the theorems, formulas, and proofs in the paper should be numbered and crossreferenced.

All assumptions should be clearly stated or referenced in the statement of any theorems.

The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition.

Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.

Theorems and Lemmas that the proof relies upon should be properly referenced.

4. Experimental Result Reproducibility

Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?

Answer: [Yes]

Justification: See Section 3.2 and Appendix A.2 for model implementation details, Appendices A.1 and H.2 for hyperparameter settings, and Section 4.1 and Appendices B.1 to B.3 for dataset settings.

Guidelines:

The answer NA means that the paper does not include experiments.

If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.

If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.

Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.

While NeurIPS does not require releasing code, the conference does require all submissions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example

(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm.

(b) If the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.

(c) If the contribution is a new model (e.g., a large language model), then there should either be a way to access this model for reproducing the results or a way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).

(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.

5. Open access to data and code

Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material?

Answer: [Yes]

Justification: The code will be published in the future; currently, we provide an anonymous version in the supplemental material. For datasets, we use an open-source benchmark and strictly follow its official implementation as described in Section 4.1.

Guidelines:

The answer NA means that paper does not include experiments requiring code.

Please see the NeurIPS code and data submission guidelines ( https://nips.cc/ public/guides/CodeSubmissionPolicy ) for more details.

While we encourage the release of code and data, we understand that this might not be possible, so 'No' is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).

The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines ( https: //nips.cc/public/guides/CodeSubmissionPolicy ) for more details.

The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.

The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only a subset of experiments are reproducible, they should state which ones are omitted from the script and why.

At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).

Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted.

6. Experimental Setting/Details

Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?

Answer: [Yes]

Justification: For dataset settings, see Section 4.1 and Appendices B.1 to B.3. For detailed hyperparameter settings, see Appendices A.1 and H.2.

Guidelines:

The answer NA means that the paper does not include experiments.

The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them.

The full details can be provided either with the code, in appendix, or as supplemental material.

7. Experiment Statistical Significance

Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?

Answer: [Yes]

Justification: For tables in Appendix G.2, we calculate the mean and standard deviation of ranks across datasets to provide the rank for each model with their confidence interval.

Guidelines:

The answer NA means that the paper does not include experiments.

The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).

The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)

The assumptions made should be given (e.g., Normally distributed errors).

It should be clear whether the error bar is the standard deviation or the standard error of the mean.

It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have a 96% CI, if the hypothesis of Normality of errors is not verified.

For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).

If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.

8. Experiments Compute Resources

Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?

Answer: [Yes]

Justification: See Appendix H.1 for hardware settings and Appendix C.1 for the FLOPs and inference time of different models.

Guidelines:

The answer NA means that the paper does not include experiments.

The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.

The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.

The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).

9. Code Of Ethics

Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.cc/public/EthicsGuidelines ?

Answer: [Yes]

Justification: The research conducted in our paper conforms, in every respect, with the NeurIPS Code of Ethics.

Guidelines:

The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.

If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics.

The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction).

10. Broader Impacts

Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?

Answer: [No]

Justification: This paper proposes a novel DNN model, with experiments focusing on performance comparisons with other existing DNN and tree-based models, as well as exploring the mechanisms behind this novel method. We acknowledge that the introduction of new model structures can have both positive and negative societal impacts (e.g. fairness considerations); however, these aspects are not the primary scope or focus of this paper.

Guidelines:

The answer NA means that there is no societal impact of the work performed.

If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.

Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.

The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.

The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.

If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).

11. Safeguards

Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?

Answer: [NA]

Justification: This paper poses no such risks.

Guidelines:

The answer NA means that the paper poses no such risks.

Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.

Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.

We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.

12. Licenses for existing assets

Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?

Answer: [Yes]

Justification: The public assets accessed in this paper include models and datasets. We have clearly cited each of them and strictly followed their terms of use. For all public models, please refer to their respective websites for the license information. Regarding the Tabular Benchmark, all datasets used are publicly available from OpenML [36] under the CC-BY 4.0 license. Please refer to Appendix B.3 for detailed information on the datasets used.

Guidelines:

The answer NA means that the paper does not use existing assets.

The authors should cite the original paper that produced the code package or dataset.

The authors should state which version of the asset is used and, if possible, include a URL.

The name of the license (e.g., CC-BY 4.0) should be included for each asset.

For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.

If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset.

For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.

If this information is not available online, the authors are encouraged to reach out to the asset's creators.

13. New Assets

Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?

Answer: [Yes]

Justification: The source code for this paper will be published under the Apache License 2.0 in the future. During the reviewing process, a minimal workable example of DOFEN is provided in the supplementary material, along with a simple README explaining how to execute the code.

Guidelines:

The answer NA means that the paper does not release new assets.

Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.

The paper should discuss whether and how consent was obtained from people whose asset is used.

At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.

14. Crowdsourcing and Research with Human Subjects

Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?

Answer: [NA]

Justification: The paper does not involve crowdsourcing nor research with human subjects.

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper.

According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.

15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects

Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?

Answer: [NA]

Justification: The paper does not involve crowdsourcing nor research with human subjects. Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.

For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper introduces DOFEN, a deep learning method for tabular data inspired by oblivious decision trees and random forests. DOFEN combines a novel differentiable oblivious tree model with a novel forest ensembling stage. On a well-established tabular benchmark, DOFEN performs close to boosted trees and mostly outperforms other deep learning baselines.\n\nSTRENGTHS:\n- The model is evaluated on a well-established benchmark.\n- The model achieves strong results on the benchmark (not necessarily state-of-the-art in every respect, but still promising).\n- Many additional results are provided in the appendix.\n- The ensembling stage, which is shown to be useful in ablations, could be useful for other deep tree-based models as well.\n- Code is provided.\n\nWEAKNESSES:\n1. I found the description of the method sometimes confusing, especially the mathematical notation (see below). Maybe writing the method as an algorithm could help? I was also missing some details the sub-networks $\\Delta_i$ in the main paper (are they fully-connected?).\n2. Training times are not reported (although inference times are reported) and from what I can infer from the text, training is rather slow.\n\nQUESTIONS:\n**Questions:**\n- Section 3.2.2.: Is the shuffling process necessary to obtain good performance or would direct random sampling of coordinates be sufficient? Do you use more trees for datasets with more features?\n- In Appendix C you say that you do not use early stopping. Do you still select the best epoch based on the validation set? If no, why not?\n\n**Comments:**\n- l. 222: \"Trompt is the current state-of-the-art tabular DNN.\" I would say this is at least unclear, since for example TabR (https://openreview.net/forum?id=rhgIgTSSxW) claimed to outperform XGBoost on the same benchmark. There are also many other deep learning methods that claim to be SOTA, e.g., GANDALF (https://arxiv.org/abs/2207.08548), GRANDE (https://arxiv.org/abs/2309.17130), BiSHop (new, so not required to cite, https://arxiv.org/abs/2404.03830).\n- For the related work I think GRANDE (https://arxiv.org/abs/2309.17130) is highly related, and they also propose differentiable trees that output weights together with the predictions. \n- I think oblivious decision trees are mentioned before they are explained, the authors might consider explaining them earlier.\n\n**Minor comments:**\n- Eq. 2: The notation is a bit strange, specifically the set notation for ODT(x), the long right arrow, the definition of $j$, and why j is an upper index and not a lower index. An alternative suggestion might be $ODT(x) = H(x_j - b), (j, b) = criterion(context), b \\in \\mathbb{R}, j \\in \\{1, \\dots, N_{col}\\}$.\n- l. 157: It is not clear how the shuffling of the matrix works. Is it equivalent to shuffling after flattening? If yes, then it might be better to write it in this way. But in Figure 2 there is still exactly one color per column, which might indicate that the shuffling works differently?\n- Eq. 6: From the text it seems that $\\Delta_2$ is applied for each tree separately, producing a scalar output. From the equation, it seems that it is applied to the whole shuffled matrix, producing the vector $w$.\n- Eq. 9: The naming with one prime for the vector and two primes for the elements might be confusing. Also, it is not clear here if softmax is applied to each $w_i''$ separately or somehow together.\n- Eq. 8-11: It is unclear to me why you write everything as operations on big vectors, instead of just writing the computation for a single index $i$. \n- Eq. 12: use \\mathrm or \\operatorname to write names in formulas (then it doesn't look like products of single-letter variables)\n- An appendix table of contents at the beginning of the appendix would be nice to have.\n\n**Conclusion:** The paper provides a new method that achieves strong results on an established benchmark on a practically important problem. The results appear solid and comprehensive. Therefore I recommend acceptance.\n\nLIMITATIONS:\nLimitations are discussed. Broader impacts are not discussed, but the method is general-purpose, so the impact depends on the downstream applications.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 4, "DATE": "2024-06-20"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nTabular data is widely used in various fields. This paper studies the current situation that DNN still lags behind Gradient Boosting Decision Trees (GBDTs) in tabular data. This paper introduces Deep Oblivious Forest Ensemble (DOFEN), which is inspired by unbiased decision trees and implements on-off sparse selection of columns. In addition, DOFEN shows excellent performance on the recognized Tabular Benchmark, surpassing other neural network models and similar to GBDTs in multiple tasks. This achievement highlights the robustness and diversity of DOFEN in different tasks and proves its effectiveness on tabular data.\n\nSTRENGTHS:\nFirst of all, this paper has sufficient technical contributions and corresponding theoretical analysis. I believe that this idea of introducing oblivious decision trees into deep neural network (DNN) architecture will promote the research progress of deep learning in tabular data. Secondly, this article is well written and easy to understand. The explanation of the formula is sufficient. Finally, the paper provides a comprehensive assessment of the DOFEN model, including model stability, efficiency, and hyperparameter effects, which provides valuable insights for further research and model optimization.\n1. A novel deep neural network (DNN) architecture is proposed by DOFEN, which combines the concept of oblivious decision trees (ODT) and realizes the on-off sparse column selection of table data. This is a feature that is not common in existing DNNs.\n2. DOFEN solves the non-differential problem of sparse selection in deep learning models through its innovative two-step process, which is a challenge in previous studies.\n3. The paper not only proposes a new method in theory, but also verifies the effectiveness of the DOFEN model through experiments, showing its application potential on different types of data sets.\n4. Although DOFEN is a complex model, the paper provides detailed model configuration and hyperparameter setting, which is helpful for other researchers or practitioners to implement and apply this model.\n\nWEAKNESSES:\n1. Compared with other DNNs, the inference time of DOFEN may be relatively long, which may affect its practicality in application scenarios that require rapid response.\n2. The randomization step of DOFEN leads to a slower convergence speed, which means that more training steps are needed to achieve optimal performance, which may increase training costs and time.\n3. Although the paper mentions that increasing the number of forests can alleviate the instability caused by randomness, randomness itself may pose a challenge to the stability and reproducibility of the model.\n4. Although the paper mentions that increasing the number of forests can alleviate the instability caused by randomness, randomness itself may pose a challenge to the stability and reproducibility of the model.\n\nQUESTIONS:\n1. The DOFEN model mentioned in the paper has a long inference time. What are the specific performance bottlenecks in implementation? Does the author consider or implement specific optimization measures to improve the efficiency of the model?\n2. The randomness introduced in the model is mentioned in the paper. How does this randomness affect the stability and reproducibility of the model? Is there a way to reduce this effect?\n3. In deep learning models, model interpretability is an important consideration. Does the DOFEN model provide an explanation for the decision-making process? How are these explanatory features realized?\n4. The paper mentions some limitations of the model. What are the key assumptions in the model design and experiment? How do these assumptions affect the performance of the model in practical applications?\n5. Does the author test the generalization ability of the DOFEN model on different data distributions or domains ? If yes, please share the results and analysis of these experiments.\n\nLIMITATIONS:\n1. Compared with other DNN models, DOFEN has a relatively long inference time, which may limit its practicality in application scenarios that require fast response.\n2. Due to the randomization steps introduced in the DOFEN model, the model requires more training steps to achieve optimal performance, which leads to a training process that may be more time-consuming than other models.\n3. The performance of DOFEN may be sensitive to hyperparameter selection, and sufficient hyperparameter adjustment is required to obtain the best performance, which may increase the complexity of model tuning.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2024-07-10"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes Deep Oblivious Forest ENsemble (DOFEN), a novel neural network-based architecture for tabular deep learning. DOFEN extends Neural Oblivious Decision Ensembles (NODE) by relaxing the interaction level between a sample and the conditions on nodes in oblivious decision trees. In particular, when constructing oblivious decision trees, DOFEN follows a two-stage procedure, where the first stage generates soft conditions and the second stage utilizes a shuffle-then-segment framework to generate multiple condition tables. The resultant combination of condition tables can be regarded as a differentiable counterpart to the vanilla oblivious decision trees, which the authors referred to as \"relaxed oblivious decision trees\". Experimental results on the widely adopted tabular benchmark demonstrate that DOFEN achieves excellent performance, outperforming a number of state-of-the-art tabular deep learning models.\n\nSTRENGTHS:\n- The idea of using relaxed oblivious decision trees for tabular deep learning is, to the best of my knowledge, novel and technically sound.\n\n- The authors adopted the highly recognized tabular data learning benchmark, which ensures a rigorous evaluation of the proposed approach. Experimental results also suggest that DOFEN achieves very competitive performance.\n\nWEAKNESSES:\n- The paper is not very well-written, and I found it quite hard to follow.\n\n- No ablation study is provided, and thus the readers cannot know how much each component in DOFEN contributes to the final performance. I would suggest that the authors dissect the difference between DOFEN and NODE, and conduct an ablation study to demonstrate how each newly introduced component improves the results.\n\nQUESTIONS:\n- In the code provided in the supplementary material, is CCCModel the implementation of DOFEN? If so, could the authors please point to me which part of the code corresponds to $\\Delta_2$ in DOFEN?\n\n- In the shuffle-then-segment procedure in DOFEN, is the shuffle random for each sample, or is it fixed during training and inference?\n\nLIMITATIONS:\nThe authors have adequately addressed the limitations and potential negative societal impact of their work.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-12"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper primarily investigates DNN models for tabular data. Current DNN models have limited performance and do not outperform GBDTs. The authors believe the key lies in the need for sparse feature extraction from tabular data. Existing attention-based models lead to dense selection of columns, while tree-based DNNs can only achieve near-sparsity. The paper introduces a novel deep neural network architecture called DOFEN (Deep Oblivious Forest ENsemble), aiming at enhancing performance on tabular data. Inspired by Oblivious Decision Trees (ODT), DOFEN implements a new two-step process to achieve on-off sparse selection of columns, thereby gaining advantages in feature diversity and mitigating overfitting. The paper evaluates DOFEN on the recognized Tabular Benchmark, demonstrating its excellent performance across various tasks, especially in competitiveness when compared with Gradient Boosting Decision Trees (GBDTs).\n\nSTRENGTHS:\n1. Innovation: DOFEN integrates the concept of ODT with deep learning techniques, proposing a new neural network architecture, which is an innovation in the processing of tabular data.\n2. Performance: Evaluation results on the Tabular Benchmark show that DOFEN has achieved state-of-the-art performance levels in multiple tasks, demonstrating competitiveness compared to GBDTs.\n3. Robustness and Universality: DOFEN has shown promising performance across various types of datasets, whether they feature numerical attributes or a combination of numerical and categorical features.\n\nWEAKNESSES:\n1. Implementation Complexity: The architecture of DOFEN may be more complex than traditional tree models and some DNN models, which could increase the difficulty of implementation and understanding.\n2. Long Inference Time: Although DOFEN performs well in some aspects, its inference time is relatively long, which may affect its use in applications that require real-time response.\n3. Slow Convergence Speed: The randomized steps in DOFEN may lead to a slower training convergence speed, requiring more training steps to achieve optimal performance.\n\nQUESTIONS:\n1. The paper mentions that DOFEN employs randomized steps to enhance the model's generalization ability, but what is the specific mechanism by which these steps affect the training dynamics and ultimate performance of the model?\n2. The performance of DOFEN largely depends on the selection of hyperparameters. Is there an automated strategy for hyperparameter adjustment?\n3. While DOFEN shows excellent performance in experiments, how does it perform in real-world application scenarios, especially when dealing with large-scale datasets or in resource-constrained environments?\n4. Compared to traditional tree models, how interpretable is the decision-making process of DOFEN as a deep learning model, and are there ways to improve the model's interpretability?\n\nLIMITATIONS:\nsee the section of weakness.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-15"}