Net-DNF: Effective Deep Modeling of Tabular Data
Authors: Liran Katzir
Gal Elidan
Ran El-Yaniv
Venue: ICLR 2021 Poster (2021)
Keywords: Neural Networks
Architectures
Tabular Data
Predictive Modeling

Abstract: A challenging open question in deep learning is how to handle tabular data. Unlike domains such as image and natural language processing, where deep architectures prevail, there is still no widely accepted neural architecture that dominates tabular data. As a step toward bridging this gap, we present Net-DNF a novel generic architecture whose inductive bias elicits models whose structure corresponds to logical Boolean formulas in disjunctive normal form (DNF) over affine soft-threshold decision terms. Net-DNFs also promote localized decisions that are taken over small subsets of the features. We present an extensive experiments showing that Net-DNFs significantly and consistently outperform fully connected networks over tabular data. With relatively few hyperparameters, Net-DNFs open the door to practical end-to-end handling of tabular data using neural networks. We present ablation studies, which justify the design choices of Net-DNF including the inductive bias elements, namely, Boolean formulation, locality, and feature selection. 


Published as a conference paper at ICLR 2021
NET-DNF: EFFECTIVE DEEP MODELING OF TABULAR DATA
Liran Katzir
Gal Elidan eli @liddnnggoglees
Ran El-Yaniv cani@ca.technionn
1i rrankegoogleecom
ABSTRACT
A challenging open question in deep learning is how to handle tabular data. Unlike domains such image and natural language processing, where deep architectures prevail, there is still no widely accepted neural architecture that dominates tabular data. As a step toward bridging this gap, we present Net-DNF a novel generie archi- tecture whose inductive bias elicits models whose structure corresponds to logical Boolean formulas in disjunctive normal form (DNF) over affine soft-threshold de- cision terms. Net-DNFs also promote localized decisions that are taken small subsets of the features. We present extensive experiments showing that Net-DNFs significantly and consistently outperform fully connected networks over tabular data. With relatively few hyperparameters. Net-DNF's open the door to practical end-to-end handling of tabular data using neural networks. We present ablation studies, which justify the design choices of Net-DNF including the inductive bias elements, namely, Boolean formulation, locality, and feature selection.
INTRODUCTION
key point successfully applying deep neural models is the construction of architecture families that contain inductive bias relevant to the application domain. Architectures such CNNs and RNNs have become the preeminent favorites for modeling images and sequential data, respectively. For example, the inductive bias of CNNs favors locality, well translation and scale invariances. With these properties, CNNs work extremely well on image data, and are capable of generating problem-dependent representations that almost completely overcome the need for expert knowledge. Similarly, the inductive bias promoted by RNNs and LSTMs (and more recent models such as transformers) favors both locality and temporal stationarity.
When considering tabular data, however, neural networks are not the hypothesis class of choice. Most often, the winning class in learning problems involving tabular data is decision forests. In Kaggle competitions, for example, gradient boosting of decision trees (GBDT's) (Chen & Guestrin 2016; Friedman) 2000 Proohhrenkovaa al. 2018 Ke et al., (017) are generally the superior model. While it is quite practical to GBDTs for medium size datasets, it is extremely hard scale these methods to very large datasets. Scaling up the gradient boosting models was addressed by several papers YYe et al. 2009 Tyree et al.. 2011 1 Fu et al. 2019 Vasiloudis al. 2019). The most significant computational disad vantage of GBDTs is the need to store (almost) the entire dataset in memory] Moreover, handling multi-modal data, which involves both tabular and spatial data (e.g.. medical records and images), is problematic. Thus, since GBDTs and neural networks cannot be organically optimized, such multi-modal tasks left with sub-optimal solutions. The creation of purely neural model for tabular data, which can be trained with SGD end-to-end, is therefore prime open objective.
A few works have aimed at constructing neural models for tabular data (see Section 5 Currently, however, there is still no widely accepted end-to-end neural architecture that can handle tabular data and consistently replace fully- colly-connected architectures, or better yet, replace GBDTs. Here we present NeL-DNFs, a family of neural network architectures whose primary inductive bias is an ensemble comprising disjunctive normal form (DNF) formulas over linear separators. This family also promotes (input) feature selection and spatial localization ensemble members. These inductive
This disadvantage is shared among popular GBDT implementations: XGBoost, LightGBM, and CatBonst.

---

Published as a conference paper at ICLR 2021
biases have been included by design to promote conceptually similar elements that inherent in GBDTs and random forests. Appealingly, the Net-DNF architecture can be trained end-to-end using standard gradient-based optimization. Importantly, it consistently and significantly outperforms FCNs on tabular data, and can sometime even outperform GBDTs.
The choice of appropriate inductive bias for specialized hypothesis classes for tabular data is chal- lenging since, clearly, there are many different kinds of such data. Nevertheless, the "universality" of forest methods in handling wide variety of tabular data suggests that it might be beneficial emu- late, using neural networks, the important elements that are part of the tree ensemble representation and algorithms. Concretely, every decision tree is equivalent to some DNF formula over axis -lis-edigned linear separators (see details in Section B This makes DNFs an essential element in any such construction. Secondly, all contemporary forest ensemble methods rely heavily on feature selection. This feature selection is manifested both during the induction of each individual tree, where features are sequentially and greedily selected using information gain or other related heuristics, and by uniform sampling features for each ensemble member. Finally, forest methods include important localization element- - GBDTs with their sequential construction within a boosting approach, where each tree re- weights the instance domain differently and random forests with their reliance bootstrap sampling.
are designed to include precisely these three elements.
After introducing Net-DNF, we include Vapmik- Chenvnnnhers (VC) comparative analysis of DNFs and trees showing that DNFs potentially have advantage over trees when the input dimension is large and vice versa. We then present extensive empirical study. We begin with ablation study over three real-life tabular data prediction tasks that convincingly demonstrates the importance of all three elements included in the Net-DNN design. Second, we analyze our novel feature selection component over controlled synthetic experiments, which indicate that this component is of independent interest. Finally, we compare Net-DNFss to FCNs and GBDTs over several large classification tasks, including two past Kaggle competitions. Our results indicate that Net-DNFs consistently outperform FCNs, and can sometime even outperform GBDT's
2 DISJUNCTIVE NORMAL FORM NETWORKS (NET-DNFS)
In this section we introduce the Net-DNF architecture, which consists of three elements. The main component is block of layers emulating a DNF formula. This block will be referred as a Disjunctive Normal Neural Form (DNNF). The second and third components, respectively, are feature selection module, and localization one. In the remainder of this section we describe each component in detail. Throughout our description we denote by x E IR a column of input feature vectors, by x4, its ith entry, and by (-) the sigmoid function.
2.1 A DISJUNCTIVE NORMAL NEURAL FORM (DNNF) BLOCK
disjunctive normal neural form (DNNF) block is assembled using a two-hidden-layer network. The first layer creates affine "literals" (features) and is trainable. The second layer implements number of soft conjunctions over the literals, and the third output layer is a neural OR gate. Importantly, only the first layer is trainable, while the two other are binary and fixed.
We begin by describing the neural AND and OR gates. For an input vector x, we define soft, differentiable versions of such gates as
d x; + d 1
d AND(x) tanh Σ x; - d 1.5
OR(x)
A
tanh Σ
These definitions are straightforwardly motivated by the precise neural implementation of the corre- sponding binary gates. Notice that by replacing tanh by binary activation and changing the bias constant from 1.5 to 1, we obtain an exact implementation of the corresponding logical gates for binary input vectors (Anthony 2005 Shalev-Shwartz & Ben- David, 2014); see a proof of this state- ment in Appendix A Notably, each unit does not have any trainable parameters. We now define the AND gate in a vector form to project the logical operation over a subset of variables. The projection is controlled by an indicator column vector (a mask) u E {0, 1]d With respect to such projection vector u. we define the corresponding projected gate (ND,(x)) A tanh(u x ||u|]] + 1.5).
2

---

Published as a conference paper at ICLR 2021
Equipped with these definitions, (NNNF(x) (x) R - R with & conjunctions over literals is,
L(x) - tanh(x W +b E R (NNF A OR[[AND, (L(x),,
(1)
...., AND
Equation defines L(x) that generates "neural literals", each of which is the result of a tanh- activation of a (trainable) affine transformation. The (trainable) matrix W E Rd as well as the row vector bias term b R determine the affine transformations for each literal such that each of its columns corresponds to one literal. Equation (2) defines a DNNF. In this equation, the vectors c' E {0, 1mm,, 1 V i S k; are binary indicators such that cj 1 iff the jth literal belongs to the ith conjunction. In our design, each literal belongs to single conjunction. These indicator vectors are defined and fixed according to the number and length of the conjunctions (See Appendix (.2)
2.2 NET-DNFS
The embedding layer of a Net-DNF with n DNNF blocks is a simple concatenation
( A DNNFF (x),, DNNF2(x),, .... ,, (x)]-
(3)
Depending on the application, the final Net-DNF is a composition of an output layer over E(x). For example, for binary classification (logistic output layer), Net-DNF(o)) + (0, 1) is,
Net-DNF(x) A Σ w,DNNF + +b;
(4)
To summarize, a Net-DNF is always four-layer network (including the output layer), and only the first and last layers are learned. Fach DNNF block has two parameters: the number of conjunctions k and the length m of these conjunctions, allowing for variety of Net-DNF architectures. In all our experiments we considered a single Net-DNF architecture that has a fixed diversity of DNNF blocks which includes number of different DNNF groups with different k, each of which has number of conjunction sizes m (see details in Appendix D.2). The number n of DNNFs was treated as hyperparameter, and selected based on validation set as described on Appendix D.I
2.3 FEATURE SELECTION
One key strategy in decision tree training is greedy feature selection, which is performed hierarchically at any split, and allows decision trees to exclude irrelevant features. Additionally, decision tree ensemble algorithms apply random sampling to select subset of the features, which used to promote diversity, and prevent different trees focusing on the same set of dominant features in their greedy selection. line with these strategies, we include in our Net-DNFss conceptually similar feature selection elements: (1) a subset of features uniformly and randomly sampled for each DNNF; (2) a trainable mechanism for feature selection, applied on the resulting random subset. These two elements are combined and implemented in the affine literal generation layer described in Equation D. and applied independently for each DNNF. We now describe these techniques in detail.
Recalling that d is the input dimension, the random selection is made by generating stochastic binary mask m, E {0, 1 (each block has its own mask), such that the probability of any entry being is p (see Appendix DD22 for details on setting this parameter). For given mask m,, this selection can be applied over affine literals using a simple product diag(m,)) W, where W is the matrix of Equation (I). We then construct trainable mask m, Æd which will be applied on the features that are kept by m. We introduce a novel trainable feature selection component that combines binary quantization of the mask together with modified elastic-net regularization. To train a binarized vector we resort to the traiightthhoouge estimator (Hinton) 2012 Hubara et al., 2017). which can be used effectively to train non- con-differentiable step functions such as threshold or sign. The trick is to compute the step function exactly in the forward pass, and utilize a differentiable proxy in the backward pass. We use a version of the straight-through estimator for the sign function (Bengio) et al. 2013).
((x) 4 sign(x); forward pass;
tanh(x), backward pass.
3

---

Published as a conference paper at ICLR 2021
Using the estimator (x)), define a differentiable binary threshold function T(x) - where e E I defines an epsilon neighborhood around zero for which the output of T(x) is zero, and one outside of this neighborhood (in all our experiments, we set 1 and initialize the entries of m, above this threshold). We then apply this selection by (Tmig( W. Given a fixed stochastic selection m,, to train the binarized selection m, employ regularization. Specifically, we consider a modified version of the elastic net regularization, Ri,, m.), which is tailored to our task. The modifications are reflected in two parts. First, the balancing between the L1 and Le regularization is controlled by a trainable parameter E IR. Second, the expressions of the L1 and L2 regularization are replaced by R1 (m, m,), R2 m m,), respectively (defined below). Moreover, since we want to take into account only features that were selected by the random component, the regularization applied on the vector m,s = m, @ my, where @ is element-wise multiplication. The functional form of the modified elastic net regularization is as follows,
mas 2
mes 11
R2 (m m,) &
3e²
R1 (m, m.)
Be
|
|m,
R(m, m,) A 1 - aaaa)
R2 m.) + (a)) (m, m.).
2
The above formulation of R2(-) and R2(-) is motivated as follows. First, normalize both norms by dividing with the effective imput dimension, |m,l1,, which is done to be invariant to the (effective) input size. Second, we define R2 and R₁ absolute emrrr,, which encourages each entry be, on average, approximately equal to the threshold c. The reason is that the vector m, passes through a binary threshold, and though the exact values of its entries are irrelevant. What relevant is whether these values are within epsilon neighborhood of zero or not. Thus, when the values are roughly equal the threshold, and the relevant features were selected. The threshold term is controlled by / (a hyperparameter), which controls the cardinality of m,, where smaller values of B lead to sparser m,, To summarize, feature selection is manifested by both architecture and loss. Architecture relies on the masks m, m, while the loss function uses R(m;; mg).
more likely to converge to a balanced point where the regularization term is low
Finally, the functional form of DNNF block with the feature selection component is obtained by plugging the masks into Equation 2 L(x) A tanh(x" diag(T (m.)) diag(m,) + b) E R".
Additionally, the mean over R ((mm m.) in all DNNFs is added to the loss function as a regularizer.
2.4 SPATIAL LOCALIZATION
The last element we incorporate in the Net-DNF construction is spatial localization. This element encourages each DNNF unit in Net-DNF ensemble to specialize in some focused proximity of the input domain. Localization is a well-known technique in classical machine learning, with various implementations and applications (Jacobs al. 1991 Meir al.) 2000). On the one hand, localization allows construction of low-bias experts. On the other hand, it helps promote diversity, and reduction of the correlation between experts, which can improve the performance of an ensemble (Jacobs) 1997, Derbeko et al. 2002). We incorporate spatial localization by associating a Gaussian kernel loc(x Jr, Σ); with a trainable mean vector μi and trainable diagonal covariance matrix E, for the ith DNNF. Given Net-DNF with n DNNF blocks, the functional form of its embedding layer (Equation with the spatial localization,
loc(x/14, Σ) A le- [x e |E2(x #o) -1 11. R"
sm-llo((x (x (u, Σ) A Softmax (loc(x|4, Ju, Σ) ((T)) 10,
E(x) |sm-loc(x[p, E)1 DNNF, ( E)n (x)),
sm-
where IR is a trainable parameter such that (T)) serves as the trainable temperature in the softmax. The inclusion of an adaptive temperature in this localization mechanism facilitates a data-dependent degree of exclusivity: at high temperatures, only few DNNFs will handle input instance whereas at low temperatures, more DNNFs will effectively participate in the ensemble. Observe that our localization mechanism is fully trainable and does not add any hyperparameters.
4

---

Published as a conference paper at ICLR 2021
3 DNFS AND TREES- A VC ANALYSIS
The basic unit in our construction is a (soft) DNF formula instead of tree. Here we provide a theoretical perspective this design choice. Specifically, we analyze the VC-dimension of Boolean DNF formulas and compare it to that of decision trees. With this analysis we gain some insight into the generalization ability of formulas and trees, and argue numerically that the generalization of DNF can be superior to a tree when the imput dimension is not small (and vice versa).
Throughout this discussion, we consider binary classification problems whose instances are Boolean vectors in {0, 1}".. The first simple observation is that every decision tree has an equivalent DNF formula. Simply, each tree path from the root to a positively labeled leaf can be expressed by conjunction of the conditions over the features appearing along the path to the leaf, and the whole tree can be represented by disjunction of the resulting conjunctions. However, DNFs and decision trees are not equivalent, and we demonstrate that in the lense of VC-dimension. [Sion (1990) presented an exact expression for the VC-dimension of decision trees as function of the tree rank.
Definition (Rank). Consider binary tree T. IfT consists of single node, its rank is defined as O. I consists of root, a left subtree To of rank ro. and right subtree Ti of rank Tı, then
[1 11+ TO (T) = max ri}
if =r else
Clearly, for any decision tree T over n variables, 1 S rank(T) n. Also, it is not hard to see that a binary tree has a rank greater than r iff the complete binary tree of depth r + be embedded into T.
D7.n2
1044
DT.--3
DMF
DMF
Theorem (Simon 1990)). Let DT" denote the class of decision trees of rank at most r on n. Boolean variables. Then it holds that V (CDim(DT)) = -i (7).
dim
DMF.K=16
109
The following theorem, whose proof appears in Ap- pendix B upper bounds the VC-dimension of Boolean DNF formula.
10a
200
400
600
800
1000
Inpur dim
Theorem 2 (DNF VC-dimension bound). Let DNF be the class of DNF formulas with lo conjunc- tions on n Boolean variables. Then it holds that VCDim(DNF5) I 2(n 1)k 11)k log(3k).
Figure 1: VCDim and the upper bound on Vi ('DimmDNFF (log scale) function of the input dimension
is evident that in the case of DNF formulas the upper bound on the VC-dimension grows linearly with the input dimension, whereas in the of decision trees, if the rank is greater than 1, the VC-dimension grows polynomially (with degree at least 2) with the input dimension. In the worst case, this growth is exponential. A direct comparison of these dimensions is not trivial because there is a complex dependency between the rank r of a decision tree, and the number i of the conjunctions of an equivalent DNF formula. Even if we compare large-k DNF formulas to amall-rank trees, is we plot
clear that the VC-dimension of the trees can be significantly larger. For example, in Figure ! the upper bounds on the VC-dimension of large formulas (solid curves), and the exact VC-dimensions of small-rank trees (dashed curves). With the exception of rank-2 trees, the VC-dimension of decision trees dominates the dimension of DNFs, when the input dimension exceeds 100. Trees, however, may have advantage over DNF formulas for low-dimmasional inputs. Since the VC-dimension a qualitative proxy of the sample complexity of a hypothesis class, the above analysis provides theoretical motivation for expressing trees using DNF formulas when the input dimension is not small. Having said that, the disclaimer is that in the present discussion we have only considered binary problems. Moreover, the final hypothesis classes of both Net-DNFs and GBDTs are more complex in structure.
4 EMPIRICAL STUDY
In this section, we present an empirical study that substantiates the design of Net-DNFss and con- vincingly shows its significant advantage over FCN architectures. The datasets used in this study
5

---

Published as a conference paper at ICLR 2021
from Kaggle competitions and OpenML (Vanschoren al. 2014). A summary of these datasets appears in Appendix C All results presented in this work were obtained using a massive grid search for optimizing each model's hyperparameters. A detailed description of the grid search process with additional details can be found in Appendices D.I] D.2 We present the scores for each dataset according to the score function defined in the Kaggle competition we used, log-loss and area under ROC curve (AUC ROC) for multiclass datasets and binary datasets, respectively. All results are the mean of the test scores over five different partitions, and the standard error of the mean is reported
In addition, we also conducted preliminary study of TabNet (Arik& & Plister, 2019) (see Section over our datasets using its Pe PyTorch implementation but failed to produce competitive results.
The merit of the different Net-DNF components. We start with two different ablation studies, where we evaluate the contributions of the three Net-DNF components. In the first study, we start with a vanilla three-hidden-layer FCN and gradually add each component separately. In the second study, we start each experiment with the complete Nct-DNF and leave one component out each time. In each study, we present the results on three real-world datasets, where all results are test log-loss scores (lower is better), out-of-memory (OOM) entries mean that the network was too large to execute on our machine (see Appendix D.2 More technical details can be found in Appendix D.4
Table I: Gradual study (test log-loss scores)
Dataset
Eye Movements
Gesture Phase
Gas Concentrations
# formulas
128
512
2048
128
512
2048
128
512
2048
Exp Fully trained FCN
0,9864
1,0138
OOM
1,3139
1.3368
0OM
0.2423
0.3862
OOM
(+0.0038)
(+00000))
(+00006))
(+00004)
(=10099))
(+b1.0999)
Exp Adding strucuuee
0,9034
0,9336
1.3011
1.1391
1.1812
18633
0.0351
0,0421
0.0778
(20.0058)
(2000058)
(2:00431)
(2:00009)
(200011))
(20
(200.004))
(2:00034)
(20.0000)
Exp Adding feature selection
0.8134
0.8163
0,9652
1.1411
1.1320
1.3015
0.0227
0.0265
0.0516
(100,0142)
(10.0096)
(1000143)
(1:00093)
(1:0.083)
(1:00017))
(1:0.0019)
(110.0012)
(10.0006))
Exp 4: Adding localization
0.7621
0.7125
0.6903
0.9742
0.9120
0.8770
0.0162
0.0149
0.0145
(1.00079)
(1..0077)
(1.0.069)
(1.0.009)
(1.0.123)
(1.00088))
(1000001))
(1.0.0008)
(10.0011)
Consider I In Exp 1 we start with vanilla three- firee-hddeen-lyeer FCN with tanh activation. To make a fair comparison, we defined the widths of the layers according to the widths the NeI-DNF with the corresponding formulas. In Exp 2, we added the DNF structure to the networks from Exp 1 (see Section 2.Tp. In Exp 3 we added the feature selection component (Section 2.3). It is evident that performance is monotonically improving. where the best results are clearly obtained on the complete Net-DNF (Exp 4). A subtle but important observation is that in all of the first three experiments, for all datasets, the trend is that the lower the number of formulas, the better the score. This trend is reversed in Exp 4, where the localization component (Section 2.4p is added, highlighting the importance of using all components of the Net-DNF representation in concert.
Now consider Table 2 In Exp 5 we took the complete Net-DNF (Exp 4) and removed the feature selection component. When considering the Gesture Phase dataset, an interesting phenomenon is observed. In Exp 3 (128 formulas), we can see that the contribution of the feature selection component negligible. but in Exp (2048 formulas) we see the significant contribution of this component. We believe that the reason for this difference lies in the relationship of the feature selection component with the localization component, where this connection intensifies the contribution of the feature selection component. In Exp 6 we took the complete Net (Exp 4) and removed the localization component (identical to Exp 3). We did the same in Exp where we removed the DNF structure. In general, it can be seen that removing each component results in decrease in performance.
An analysis of the feature selection component. Having studied the contribution of the three components to Net-DNF, we now focus the learnable part of the feature selection component (Section 2.3 alone, and examine its effectiveness using a series of synthetic tasks with a varying percentage of irrelevant features. Recall that when considering single DNNF block, the feature
"Our code is available at http://giihub..om//maaaaa /http:://itthbbbc wtwpshuih.oom/rrrmauuar---ka aiitabbnet
Forr example, for the Gas Concentration dataset (see below), TabNet results were slightly inferior to the results obtained for XGBoost (4.89 log-loss for TabNet vs. 2.22 log-loss for XGBoost.

---

Published as a conference paper at ICLR 2021
Table Leave one out study (test log-loss scores)
Dataset formulas
Eye Movements
Gesture Phase
Gas Concentrations
128
512
2048
128
512
2048
128
512
2048
Exp 4: Complete Net-DNF
0,7621
0.7125
0.6903
0.9742
0.9120
0.8770
0.0162
0.0149
0.0145
(#0.000))
(+0.0077)
(200049)
(2:00009)
(2:00113 23)
(=:0.0088)
(:0.0003)
(:000000)
(#0.001))
Exp 5: Leave feature selection out
0,8150
0.8031
0.7969
0.9732
0.9479
0.9438
0.0222
0.0205
0,0200
(10,0016)
(1000046)
(1.0.054)
(1.0.002)
(100.001)
(1000111)
(100,0088)
(10,0021)
(10.0022)
Exp 6: Leave localization out
0.8134
0.8163
0,9652
1.1411
1,1320
1,3015
0.0227
0.0265
0.0516
(+0.0142)
(#0.009))
(+0.0143)
(#:0.0993)
(#:000003)
(+H02117)
(+0000019)
(+0.0002)
(+0.0)3))
Exp Leuve DNF structure out
0,8403
0.8128
0OM
1.1265
1.1101
OOM
0.0488
0.0445
OOM
(+000008)
(d-0.0077)
(d:0.006))
(#:00007)
(+000008)
(#0.0002)
selection is a learnable binary mask that multiplies the input element wise. Here we examine the effect of this mask on a vanilla FCN network (see technical details in Appendix (D.).. The synthetic tasks we use were introduced by Yoon et al. (2019); Chen et al. (2018), where they were used as synthetic experiments to test feature selection. There are six different dataset settings; exact specifications appear in Appendix D.5 For each dataset, generated seven different instances that dilfer in their input size. While increasing the input dimension d, the same logit is used for prediction, so the new features are irrelevant, and as d gets larger, the percentage of relevant features becomes smaller.
We compare the performance of a vanilla FCN on three different cases: (1) oracle (ideal) feature selection (2) (learned) feature selection mask, and (3) feature selection. (See details in Appendix D.5). Consider the graphs in Figure 2 which demonstrate several interesting insights. In all tasks the performance of the vanilla FCN sensitive to irrelevant features, probably due to the representation power of the FCN, which is prone to overfitting. On the other hand, by adding the feature selection component, we obtain near oracle performance on the first three tasks, and a significant improvement on the three others. Moreover, these results support our observation from the ablation studies: that the application of localization together with feature selection increases the latter's contribution. We can see that in Syn1-3 where there is a single interaction, the results are better than in Syn4-6 where the input space is divided into two 'local' sub-spaces with different interactions. These experiments emphasize the importance of the learnable feature selection in itself.
Dataset
Test Metric
Net-DNF
XGBoost
FCN
Otto Group
log-loss
45.600 + 0.445
45.705 1 0.361
47.898 + 0,480
Gesture Phase
log-loss
86.798 1 0.810
81.408
0.806
102.070 = 0.964
Gas Concentrations
log-loss
1.425 = 104
2.219 + 0.219
5.814 = 1.079
Eye Movements
log-loss
68.037 0.651
57.447
0.664
78.797 = 0.674
Santander Transaction
roc auc
88.668 + 0.128
89.682 1 0.165
86,722 0 0.155 0.158
House
95.451 + 0.092
95.525 + 0.138
95.164 + 0.103
Table 3: Mean test results on tabular datasets and standard error of the mean. We present the ROC AUC (higher is better) percentage, and the log-loss (lower is better) with an x x100 factor.
Comparative Evaluation. Finally, we compare the performance of Net-DNT vs. the baselines. Consider Table 3 where we examine the performance of Net-DNE's on six real-life tabular datasets (We add three larger datasets to those we used in the ablation studies). We compare our performance to XGboost Chen & Guestrin (2016), the widely used implementation of GBDTs, and to FCNs. For each model, we optimized its critical hyperparameters. This optimization process required many computational resources: thousands of configurations have been tested for FCNs, hundreds of configurations for XGBoost, and only few dozen for Net-DNF. A detailed description of the grid search we used for each model can be found in Appendix DD3 In Table 3, we see that Net-DNF consistently and significantly outperforms FCN over all the six datasets. While obtaining better than or indistinguishable results from XGBoost over two datasets, on the other datasets, Net-DNN is slightly inferior but in the same ball park as XGBoost.
7

---

Published as a conference paper at ICLR 2021
0.02
Accraay
0.55
0.75
0.50
PCN
motel
FCN
model
0.76
FCN
model
PCN aith unaile muak.
HCN with cear. must
FUN with onat be mosk
0.45
PCN with festure selection
PCN with feature selection
FCN with feature selection
0.72
(a) Synl
(b) Syn2
(c) Syn3
0.80
0.70
0.50
0.75
Acceraty
0.50
0.63
2,000
mosel
0.62
model
model
0.50
FON
FCN
0.60
FCN
PCN with oracle mask
FCN with oracle mask
FUN with oracle mask
PCW with fisttree selection
PG2 with feahurm salection
0.55
FEN with frutuue select
100
200
200
300
(d) Syn4
(c) Syn5
(f) Syn6
Figure 2: The results the six synthetic experiments. For each experiment present the test accuracy (with an error bar of the standard error of the mean) as function of the input dimension d.
5
RELATED WORK
There have been a few attempts to construct neural networks with improved performance tabular data. A recurring idea in some of these works is the explicit use of conventional decision tree induction algorithms, such as ID3 (Quinlan 1979), or conventional forest methods, such GBDT (Friedman 200T) that are trained over the data at hand, and then parameters of the resulting decision trees are explicitly or implicitly "imported" into a neural network using teacher-student distillation (Ke et al. 2018), explicit embedding of tree paths in a specialized network architecture with some kind of DNF structure (Seyedhosseini
Tasdizen 2015), and explicit utilization of forests as the main building block of layers (Feng et 20.1) 8). This reliance on conventional decision tree or forest methods as an integral part of the proposed solution prevents end-to-end neural optimization, as we propose here. This deficiency is not only theoretical nuisance but also makes it hard to use such models on very large datasets and in combination with other neural modules.
A few other recent techniques aimed to cope with tabular data using pure neural optimization we propose here. Yang et al. (2018) considered a method to approximate a single node of a decision tree using soft binning function that transforms continuous features into one-hot features. While this method obtained results comparable to a single decision tree and an FCN (with two hidden layers), it is limited to settings where the number of features is small. [Popov al. (2019) proposed network that combines elements of oblivious decision forests with dense residual networks. While this method achieved better results than GBDTs on several datasets, also FCNs achieved better than or indistinguishable results from GBDTs on most of these cases as well. Arik & Pfister (2019) presented TabNet, neural architecture for tabular data that implements feature selection sequential attention that offers instance-wis feature selection. It is reported that TabNet achieved results that are comparative or superior to GBDTss Both TabNet and Net-DNF rely on sparsity inducing and feature selection, which are implemented in different ways. While TabNet uses an attention mechanism to achieve feature selection, Net-DNF uses DNF formulas and elastic net regularization. Focusing on microbiome data, recent study Shavitt & Segal (2018) presented an elegant regularization technique, which produces extremely sparse networks that are suitable for microbiome tabular datasets. Finally, soft masks for feature selection have been considered before and the advantage of using clastic net regularization in a variable selection task was presented by Zou
& Hastie (2055) Li et al. (2016).
8

---

Published as a conference paper at ICLR 2021
6 CONCLUSIONS
We introduced Net-DNF, a novel neural architecture whose inductive bias revolves around a disjune- tive normal neural form, localization and feature selection. The importance of each of these elements has been demonstrated over real tabular data. The results of the empirical study convincingly indicate that Net-DNFs consistently outperform FCNs over tabular data. While Net-DNF's do not consistently beat XGBoost, results indicate that their performance score is not far behind GBDTs. Thus, Net-DNF offers meaningful step toward effective usability of processing tabular data with neural networks
We have left number of potential incremental improvements and bigger challenges to future work. First, in our work only considered classification problems. We expect Net-DNFs to also be effective in regression problems, and it would also be interesting to consider applications in reinforcement learning over finite discrete spaces. It would be very interesting consider deeper Net-DNF architectures. For example, instead of a single DNNF block, one can construct stack of such blocks allow for more involved feature generation. Another interesting direction would be to consider training Net-DNFs using a gradient boosting procedure similar to that used in XGBoost. Finally, most interesting challenge that remains open is what would constitute the ultimate inductive bias for tabular prediction tasks, which can elicit the best architectural designs for these data. Our successful application DNNFs indicates that soft DNF formulas are quite effective, and are strictly significantly superior to fully connected networks, but we anticipate that further effective biases will be identified, at least for some families tabular tasks.
ACKNOWLEDGMENTS
This research was partially supported by the Israel Science Foundation, grant No. 710/18.
REFERENCES
Martin Anthony. Connections between neural networks and Boolean functions. In Boolean Methods and Models, 2005.
Sercan Ömer Arik and Tomas Pfister. Tabnet: Attentive interpretable tabular learning. CoRR, abs/1908.077 07442,, 2019.
Yoshua Bengio, Nicholas Leonard, and Aaron Courville. Estimating propagating gradients through stochastic neurons for conditional computation. arXiv preprint
2013.
Jianbo Chen, Le Song, Martin Wainwright, and Michael Jordan. Learning explain: information-theoreii perspective model interpretation. arXiv preprint arXiv:1802.07814, 2018.
Tianqi Chen and Carlos Guestrin. Xgboost: A scalable tree boosting system. In Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, pp. 785-794. ACM, 2016.
Philip Derbeko, Ran El- Yaniv, and Ron Meir. Variance optimized bagging. European Conference Machine Learning, pp. 60-72. Springer, 2002.
Ji Feng, Yang Yu, and Zui-Hua Zhou. Multi-layered gradient boosting decision trees. In Advances in Neural Information Processing Systems, pp. 3555 3565, 2018.
Jerome H Friedman. Greedy function approximation: gradient boosting machine. Annals of Statistics, pp. 1232, 2001.
Fangcheng Fu, Jiawei Jiang, Yingxia Shao, and Bin Cui. An experimental evaluation of large scale GBDT systems. Proc. VLDB Endow., 12(11) 2019.
Geoffrey Hinton. Neural networks for machine learning. Coursera, video lectures, 2012. URL https: www. coursera http./ooo.......oooooorrr///eeee netss

---

Published as a conference paper at ICLR 2021
Itay Hubara, Matthieu Courbariaux, Daniel Soudry, Ran El- El-yaniv, and Yoshua Bengio. Quantized neural networks: Training neural networks with low precision weights and activations. The Journal of Machine Learning Research, 18(1):6869-683 6898, 2017.
Robert A Jacobs. Bias/variance analyses of mixtures--f-expprr architectures. Neural computation, (2):369-383, 1997.
Robert A Jacobs, Michael Jordan, Steven J Nowlan, and Geoffrey Hinton. Adaptive mixtures of local experts. Neural Computation, 3(1):79-87, 1991.
Guolin Ke, Qi Meng, Thomas Finley, Taifeng Wang, Wei Chen, Weidong Ma. Qiwei Ye, and Tie- Yan Liu. Lightgbm: A highly efficient gradient boosting decision tree. In Advances in neural information processing systems, pp. 3146-3154 2017.
Guolin Ke, Jia Zhang, Zhenhui Xu, Jiang Bian, and Tie- Yan Liu. Tabnn: A universal neural network solution for tabular data. https://openreview..eed 2018.
Yifeng Li, Chih- Yu Chen, and Wyeth W Wasserman. Deep feature selection: theory and application to identify enhancers and promoters. Journal of Computational Biology, 23(5):322-336, 2016.
Ron Meir, Ran El-Yaniv, and Shai Ben-David. Localized boosting. In COLT, pp. 190-199. Citeseer, 2000.
Sergei Popov, Stanislav Morozov, and Artem Babenko. Neural oblivious decision ensembles for deep learning on tabular data. arXiv preprint arXiv:1909.06312, 2019.
Liudmila Prokhorenkova, Gleb Gusev, Aleksandr Vorobev, Anna Veronika Dorogush, and Andrey Gulin. Catboost: unbiased boosting with categorical features. In Advances in neural information processing systems, pp. 6638-6648, 2018.
J Ross Quinlan. Discovering rules by induction from large collections of examples. Expert Systems in the Micro electronics Age, 1979.
Mojtaba Seyedhosseini and Tolga Tasdizen. Disjunctive normal random forests. Pattern Recognition, 48(3):976-983, 2015.
Shai Shalev-Shwartz and Shai Ben-David. Understanding machine learning: From theory to algorithms. Cambridge university press, 2014.
Ira Shavitt and Eran Segal. Regularization learning networks: Deep learning for tabular datasets, In Advances in Neural Information Processing Systems, pp. 1386-1396, 2018.
Hans Ulrich Simon. On the number of examples and stages needed for learning decision trees. In Proceedings of the Third Annual Workshop on Computational Learning Theory, COLT '90, pp. 303-313 San Francisco, CA, USA, 1990. Morgan Kaufmann Publishers Inc. ISBN 1558601465.
Stephen Tyree Kilian Q. Weinberger, Kunal Agrawal, and Jennifer Paykin. Parallel boosted regression trees for web search ranking. In Sadagopan Srinivasan, Krithi Ramamritham, Arun Kumar, M. P. Ravindra, Elisa Bertino, and Ravi Kumar (eds.), Proceedings of the 20th International Conference on World Wide Web, WWW 20 Hyderabad, India, March 28- April 1, 2011, pp. 387-396. ACM, 2011.
Joaquin Vanschoren, Jan N Van Rijn, Bernd Bischl, and Luis Torgo Openml: networked science in machine learning. ACM SIGKDD Explorations Newsletter, 15(2):49 00, 2014.
Theodore Vasiloudis. Hyunsu Cho, and Henrik Boström. Block-distributed gradient boosted trees. In Benjamin Piwowarski, Max Chevalier, Éric Gaussier, Yoelle Maarek, Jian-Yur Nie, and Falk Scholer (eds.), Proceedings of the 42nd International ACM SIGIR Conference on Research and Development in Information Retrieval, SIGIR 2019, Paris, France, July 21-25, 2019, pp. 1025- 1028. ACM, 2019.
Yongxin Yang, Irene Garcia Morillo, and Timothy M Hospedales. Deep neural decision trees. arXiv preprint arxiv: 1806 06988, 2018.
10

---

Published as a conference paper at ICLR 2021
Jerry Ye, Jyh-Hemg Chow, Jiang Chen, and Zhaohui Zheng. Stochastic gradient boosted distributed decision trees. In David Wai-Lok Cheung, Il- Yeol Song, Wesley W. Chu, Xiaohua Hu, and Jimmy J. Lin (eds.), Proceedings of the 18th ACM Conference Information and Knowledge Management,
CIKM 2009, Hong Kong, China, November 2-6, 2009, pp. 2061-2064. ACM, 2009.
Jinsung Yoon, James Jordon, and Mihaela van der Schaar. INVASE: instance-wise variable selection using neural networks. In 7th International Conference on Learning Representations, ICLR (Poster) 2019, New Orleans, LA, USA, May 6-9, 2019, OpenReview.net, 2019. URL https: //openreviee ew net/ oett:Forrrmma-BBuu_rooAA roAckk
Hui Zou and Trevor Hastie. Regularization and variable selection via the elastic net. Journal of the royal statistical society: series B (statistical methodology), 67(2):301-320, 2005.

---

Published as a conference paper at ICLR 2021
A OR AND AND GATES
The (soft) neural OR and AND gates were defined as
d
d d + 1.5
(x(x)
P tanh Σ
x + - dx
AND(x) - tanh Σx;
x;
By replacing the lanh activation with sign activation, and setting the bias term (instead of 1.5),
we obtain exact binary gates,
d Σ
x; + d -1) Σ 1=1
x; - d +1)
(
sign
AND(x) _ sign
i=1
Consider binary vector x E {+1]d. We prove that
AND(x) = A X
1=1
where, in the definition of the logical "and", is equivalent to 0. for any 1 1sis d, x; 1, then Ag x; = Conversely, have,
AND(x) Σ d
i=1
and the application of the sign activation yields 1. In the case of the soft neural AND gate, we get t1)h(1) 0.76; therefore, we set the bias term to 1.5 to get an output closer to (tanh 0.9).
Otherwise, there exists at least one index 1 S j S d, such that x, = -1, and A A = -1. In this case,
AND(x) = Σ - + Σx -x- -1+ 1) -
i=1
and by applying the sign activation we obtain 1. This proves that the AND(x) neuron equivalent to logical "AND" gate in the binary case. A very similar proof shows that
d
OR(x) =
V xx 1=1
B PROOF OF THEOREM
We bound the VC-dimension of DNF formula in two steps. First, derive an upper bound on the VC-dimension of a single conjunction, and then extend it to disjunction of k conjunctions. We use the following simple lemma.
Lemma 1. For every two hypothesis classes, H' H. it holds that VCDim (H)) VCDim(H))
Proof. Let d = VCDim(H'). By definition, there exist d points that can be shattered by H'. Therefore, there exist 2d hypotheses (M/2"1 in H', which shatter these points. By assumption, (M))21 C H, so VCDim(H) 2 d.
For any conjunction on Boolean variables (regardless of the number of literals), is possible to construct an equivalent decision tree of rank 1. The construction is straightforward. conjunction, the decision tree consists of single main branch of l internal decision nodes connected sequentially. Each left child in this tree corresponds to decision "1", and each right child corresponds to decision "0". The root is indexed 1 and contains the literal 21- For S l. internal node i contains the decision literal æ; and its left child is node + (whose decision literal is 14+1). 1). See the example in Figure
A x; the
12

---

Published as a conference paper at ICLR 2021
It follows that the hypothesis class of conjunctions is contained in the class of rank-1 decision trees. Therefore, by Lemma [ and Theorem + 1.
[ the VC-dimension of conjunctions is bounded above by
We now derive the upper bound on the VC-dimension of disjunction of & conjunctions. Let C' be the class of conjunctions, and let D& (C) be the class of disjunction of & conjunctions. Clearly, D& (C) a k-fold union of the class C, namely,
Dx(C) &
{Ualsed)
By Lemma 3.2.3 in (Blummer et al. 1989). if = VCDim(C), then for all & 2 VCDim(Dud 2dk log(3k). Therefore, for the class DNF of DNF formulas with le con- junctions on n Boolean variables, we have
VCDi S 2(n + 1)k log(3k).
10
mm
(nck-1
XI iask
2em 2 IRL
(renk
(rezk
Figure 3: An example of a decision tree with rank 1, which is equivalent to the conjunction zg A #1 A
I2 Z3 A ZA.
C TABULAR DATASET DESCRIPTION
We use datasets (See Table 4) that differ in several aspects such as the number of features (from 16 up 200), the number of classes (from 2 up to 9), and the number of samples (from 10k up 200k). To keep things simple, we selected datasets with no missing values, and that do not require preprocessing. All models were trained the raw data without any feature data engineering and without any kind of data balancing or weighting, Only feature wise standardization was applied.
Datasel
features
classes
sumples
source
link
Orto Group
61.9%
Kaggle
kaggle com/c/otto proup- product-c wwspeeteeessicctnnoooooollloo whw.aeiooeee..llllooooeeeeeeeeee
Gesture Phase
32
5
9.8k
OpenML
openml. frouml.orggd/4533
Gas C'oncentrations
OpenML
Recuml.orgg//1441
Eye Movements
26
10.9k
OpenML
org///140
Santander Transaction
200
2008
Kaggle
kaggle.
House
22.7k
OpenML
openml
Table 4: A description of the tabular datasets
13

---

Published as a conference paper at ICLR 2021
D EXPERIMENTAL PROTOCOL
D.1 DATA PARTITION AND GRID SEARCH PROCEDURE
All experiments in our work, using both synthetic and real datasets, were done through a grid search process. Each dataset first randomly divided into five folds in way that preserved the original distribution. Then, based on these five folds, we created five partitions of the dataset as follows. Each fold is used the test in of the partitions, while the other folds used the training and validation sets. This way, each partition was 20% test, 10% validation, and 70% training. This division was done once f and the same partitions were used for all models. Based on these partitions, the following grid search process was repeated three times with three different seed (with the exact same five partitions described before).
(eid)((wih
Algorithm 1: Grid Search Procedure
Input: model, configurations_isn Jist
results list []
for i= to n partitionn do val_scores_ __ss__iittte = []
list [
train, val, test read_ iitu(parritionindeeee for e in configurations_ian do
gained_moder
val_ val_data=val, configuration=c)
trained model.load weights_ sarsssrrmm_____________e
best_epoch()
val_score trained_ trained_model. (edee.ppeedic((daa=vva))
test_score = trained Lacd___odee_pppp model.predict (dtt=teet))
val sal_scores_ list.append(( val_ssoree
test [es__score__isstapppessesee http_//sissaappeeeessssssoooo end
best_val_index
test res = test feces_sscrreese sccores_lsstbeesssseessssssse list[best_ val ]
'isults_littappeedddeee
end
mean = mean(results__ list)
sem standard _error of the mean(resulss Jist)
Return: mean, sem
The final mean and sem that presents in all experiments the average across the three seeds. Additionally, as can be seen from Algorithm| the model that was trained on the training set (70%) the one that is used evaluate performance simple. The loading wights command is relevant for the neural network models. While for the XGBoost, the framework handles the optimal number of estimators prediction time (accordingly to early stopping training time).
the test set (20%). This done keep things
D.2 TRAINING PROTOCOL
The Net-DNF and the FCN were implemented using Tesnorflow. To make fair comparison, for both models, we used the same batch size 2048, and the same learning rate scheduler (reduce on plateau) that monitors the training loss. We set a maximum of 1000 epochs and used the same early stopping protocol (30 epochs) that monitors the validation score. Moreover, for both of them, we used the same loss function (softmax-crose -cosss-e for multi-class datasets and aigmoidccrossentrop for binary datasets) and the same optimizer (Adam with default parameters).
*ee used seed number
'We used seed numbers 1, 2, 3.
Foor details,
'For Net-DNF when using 3072 formulas, we set the batch size to 1024 on the Santander Transaction and Gas datasets and when using 2048 formulas, we set the hatch size to 1024 on the Santander Transaction dataset. This done due memory issues.
14

---

Published as a conference paper at ICLR 2021
For Net-DNF used initial learning rate of 0.05. For FCN, added the initial learning rate to the grid search with values of {0.05, 0.005, 0.0005).
For XGBoost set the maximal number of estimators to be 2500, and used early stopping of 50 estimators that monitors the validation score.
All models trained GPUs Titan Xp 12GB RAM.
Additionally, in the case of Net-DNF. we took a caametry-breakkinn approach between the different DNNFs. This is reflected by the DNNF group being divided equally into four subgroups where, for each subgroup, the number of conjunctions is equal to one the following values [6, 9, 12, 15], and the group of conjunctions of each DNNF was divided equally into three subgroups where, for each subgroup, the conjunction length is equal to one of the following values [2, 4, 6]. The same approach was used for the parameter p of the random mask. The DNNF group was divided equally into five subgroups where, for each subgroup, p is equal to one of the following values [0.1, 0.3, 0.5, 0.7, 0.9]. In all experiments we used the same values.
D.3 GRID PARAMETERS TABULAR DATASETS
D.3.1 NET-DNF
Net-DNF (42 configs)
hyperparameter
values
number of formulas
(64, 128, 212, 1024, 2048, 3072] 1,7,
feature selection beta
D.3.2 XGBooST
XGBoost (864 configs)
by perraarreete number of estimators
values
(2500)
learning rate max depth
(0.001, 0.005.00,,,,0 0.01, 0.05, 0.1, 3,5, 11, 13, 15)
colsample sub sample
{0.25, 0.5, 0.75, 1.)
0..25,0.5, 0.75,
To summarize, we performed a crude but broad selection (among 42 hyper-parameter configurations) for our Net-DNE. Results were quite strong. avoided further fine tuning. To ensure extra fairness w.r.t the baselines, we provided them with significantly more paprr-prrreeter tuning resources (864 configurations for XGBoost, and 3300 configurations for FCNs).
D.3.3 FULLY CONNECTED NETWORKS
The FCN networks are constructed using Dense-RELU--
blocks with Le regularization. The network's blocks are defined the following way. Given depth and width parameters, we examine two different configurations: (1) the same width is used for the entire network (c.g., if the width 512 and the depth is four, then the network blocks are [512, 512, 512. 512]), and (2) the width parameter defines the width of the first block, and the subsequent blocks reduced by factor of 2 (e.g., if the width 512 and the depth is four, then the network blocks are [512, 256, 128, 64]). On top of the last block we add simple linear layer that reduce the dimension into the output dimension. The dropout and Le values are the same for all blocks.
15

---

Published as a conference paper at ICLR 2021
FCN (3300 configs)
hyperparameter
values
depth
(1.2.3, 4,
width
[128, 512, 1024, 2048)
Lg lambda
(10 10-4 10-6 1,00,,00,,
dropout
(0. 0.25, 0.5, 0.75)
initial learning rate
(0.05, 0.005, 0.0005)
D.4 ABLATION STUDY
All ablation studies experiments were conducted using the grid search process as described in D. In all experiments, used the same training details as described on DD22for for Net-DNF. Where the only difference between the different experiments is the addition or removal of the components.
The single hyperparameter that was fine-tuned using the grid search is the 'feature selection beta' on the range {1.6, 1.3, 1., 0.7, 0.4, 0.1). in experiments in which the feature selection component is involved. In the other cases, only one configuration was tested in the grid search process for specific number of formulas.
D.5 FEATURE SELECTION ANALYSIS
The input features x E Rd distribution with no correlation across the features, Bernoulli random variable with P(y = 1|x) [+lopi((x)' where logit(x) is varied to create the different synthetic datasets (x, refers to the ith entry):
datasets were generated from a d-dimensional Gaussian
I). The label y is sampled
Synl: logit(x) = exp(x1x2)
2. Syn2: logit(x) crp(Ei= x² 4)
3. Syn3: logit(x) = -in (2x,) + 2|xx| + + exp( 2.4
4. Syn4: if X11 < 0, logit follows Synl, else, logit follows Syn2
5. Syn5: if X11 < 0, logit follows Synl, else, logit follows Syn3
6. Syn6: if X11 < 0, logit follows Syn2, else, logit follows Syn3
We compare the performance of basic FCN three different cases: (1) oracle (ideal) feature selection where the input feature vector is multiplied element-wise with an input oracle mask, whose ith entry equals iff the ith feature is relevant (e.g., on Syn features I and 2 relevant, and on Syn4, features 1-6, and 11 are relevant), (2) our (learned) feature selection mask where the input feature vector multiplied element-wise with the mask m,, i.e., the entries of the mask m, (see Section 2.3) are all fixed to 1, and (3) no feature selection.
From each dataset, generated seven different instances that differ their input size, d E [11,50] 100, 150, 200, 250, 300]. Where when the input dimension d increases, the same logit function is used. Each instance contains 10k samples that were partitioned as described Section D.1 We treated each instance as an independent dataset, and the grid search process that is described in Section D.I was done for each one.
The FCN that we used has two dense hidden layers [64, 32] with a RELU activation. To keep things simple, we have not used drouput or any kind of regularization. The same training protocol was used for all three models. We used the same learning rate scheduler, early stopping protocol, loss function and optimizer as appear in Section D. We use a batch size of 256, and an initial learning rate of 0.001. The only hyperparameter that was fine-tuned is the 'feature selection beta' in the case of 'FCN with feature selection' on the range ( (1.3, 1., 0.7, 0.43. For the two other models, only a single configuration was tested in the grid search process.
We noticed that in this scenario, a large learning rate or large batch size leads to a decline in the performance of the 'PCN with the feature selection". While the simple FCN and the "FCN with oracle mask' remains approximately the same.
16


Reviews:
[{"review_id": "pNnNSLtmbgZ", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The authors propose a end-to-end deep learning model called Net-DNF to handle tabular data. The architecture of Net-DNF has four layers: the first layer is a dense layer (learnable weights) with tanh activation eq(1). The second layer (DNNF) is formed by binary conjunctions over literals eq(2). The third layer is an embedding formed by n DNNF blocks  eq(3). the last layer is a linear transformation of the embedding with a sigmoid activation eq(4). The authors also propose a feature selection method based on a trainable binarized selection with a modified L1 and L2 regularization. In the experimental analysis, Net-DNF outperforms fully connected networks. \n\npros:\n\n- A novel approach to handle tabular data using deep learning with an integrated feature selection\n- The VC dimension bound gives theoretical motivation  for expressing decision trees as DNF formulas.\n\ncons:\n\n- Classical ML approaches are outperforming Net-DNF\n- The experimental analyses are not convincing: The datasets are a bit limited and performance metrics are rather limited to log-loss which is difficult to interpret\n\n\nremarks:\n\n- It is unclear whether adding the Net-DNF layers, in the ablation study, gives improvement due to simply an increase of model capacity or the specific architecture of Net-DNF.\n- In Exp 7, why there is OOM despite removing additional layer ?\n- The authors did not discuss a similar recent work: TabNet (https://arxiv.org/abs/1908.07442) \n- The authors need to include additional datasets similar to TabNet experiments\n- in $R(m_t, m_s)$ I think there should not be a division by 2 ", "extra_scores": {}, "extra_text": {}}, {"review_id": "LiMjsRKMUUI", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper proposes a neural architecture that emulates the characteristics of decision-tree variants, in the hope of mirroring their successes on tabular data. The architecture consists of three components: DNNF blocks, feature-selection masks, and spacial-localization weightings of DNNF blocks. I find these components and their coherent combination into the Net-DNF structure to be novel.\n\nHowever, their empirical evaluations do not place the performance of Net-DNF in the context of existing work. It is good that the authors have compared Net-DNF against the obvious baselines of XGBoost and FCN, but they have neglected to demonstrate where Net-DNF stands in relation to other tabular-inspired approaches (mentioned in the related work). For example, why did the authors not compare against Popov et al. (2019; whose code is open-sourced) and Shavitt & Segal (2018)?\n\nI feel that the authors have left out a line of work that uses a deep learning approach for tabular data. The reference below is a representative publication. (NB: The VAE approach can be evaluated on prediction tasks like Net-DNF, and it works on multi-modal data that Net-DNF purports to handle.) Could the authors comment on where Net-DNF stands relative to this line of work (and include it in their related work section)?\n\nNazabal, Alfredo, et al. \"Handling incomplete heterogeneous data using vaes.\" Pattern Recognition (2020)\n\nThe exposition of the paper is lucid throughout.\n\nQUESTIONS\n\n* Page 1, para 2, \"Moreover,...multi-modal data,...is problematic\":\nHow does Net-DNF handle multi-modal data (e.g., \"medical records and images\")? Would it simply encode an image as a vector of bits and stack it in input x?\n\n* Page 1, para 2, \"GBDTs...hard to scale\":\nHow scalable is Net-DNF? What is its size complexity?\n\n* Page 6, Table 1 and 2.\n\"# formulas\" corresponds to the number of DNNFs in an Net-DNF, but what does it correspond to in an FCN? How many parameters are in each system? Wouldn't the number of parameters provide a fairer comparison of the capacities of the systems?\n\n* Page 8, Figure 2.\nThe authors claim that space localization will improve the green line's performance (feature selection) on Syn4-6. Why don't the authors provide empirical results to show this? Seems like a straightforward data point to add.\n\nNITS\n\n* Page 2, first line: inherent GBDTs -> inherent in GBDTs\n     \n* Page 6, In is evident -> it is evident\n", "extra_scores": {}, "extra_text": {}}, {"review_id": "9pDA55ZUfu1", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 7, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The paper attempts to propose a neural network-based algorithm on tabular data, or non-distributed representation to address the unique challenges in tabular data, which are non-existing in the distributed data (images, language, etc). The primary motivation of this paper is the simulation of DNF, which simulates the Boolean formulas in decision making. The DNF form can better capture the non-distributional nature of tabular data and somehow simulates with the decision-tree algorithms in a more soft-threshold way. The general idea makes sense to me and sounds novel compared to the existing literature. However, I still have some doubts and questions about the paper:\n1) In Sec 2.1, this soft approximation is a very critical part of the paper. Why do you decide to modify the constant from 1.0 to 1.5, you suggested in the Appendix that \"the AND gate will be closer to 1\", but the OR gate will be affected, right? Then OR(0,0,0,1) = tanh(0.5) = 0.4, which is pushed away from 1. I believe there are some trade-offs for this hyperparameter, how do you decide it should be 1.5 rather than other numbers, do you have supporting evidence for your decision? \n2) In your feature selection part, the $w_t$ is a learnable parameter to control the sparsity of the selected feature. You propose to achieve this by using the elastic net regularization. Is the trainable mask $w_t$ shared among all the DNNF blocks or is it specific to a certain DNNF block? \n3) In your experiment table 1, some results from FCN are shown to cause OOM. I kind of get the idea that DNNF uses a more compact parameterization, but not entirely sure how it quantitatively compare with FCN. What is the parameter complexity for DNNF? Could you give a more formal explanation about why it's more compact?\n4) In table 3, some of the experimental results are better than XGBOOST and some are worse. I can understand that the current DNNF has not yet achieved the same performance as XGBOOST as XGBOOST has always been a go-to algorithm for tabular data. But could you explain why DNNF is much better than GAS Concentrations, is it because of some properties of this dataset? And do you have a more high-level suggestion as to \"under what circumstances is DNNF likely to out-perform XGBOOST\", what could be the reason for that? Is it because of the better generalization of the neural network?\n\nTypo: \n2.4 Spacial Localization -> Spatial Localization\n", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper proposes an end-to-end architecture, Net-DNF,  for handling tabular data. This is a novel approach in the relatively under-explored domain of application of neural networks; the paper also presents justification of the design choices via ablation studies. The paper is clearly written, and empirical results are convincing.

Decision: Accept (Poster)