Net-DNF: Effective Deep Modeling of Tabular Data

1 INTRODUCTION . A key point in successfully applying deep neural models is the construction of architecture families that contain inductive bias relevant to the application domain . Architectures such as CNNs and RNNs have become the preeminent favorites for modeling images and sequential data , respectively . For example , the inductive bias of CNNs favors locality , as well as translation and scale invariances . With these properties , CNNs work extremely well on image data , and are capable of generating problem-dependent representations that almost completely overcome the need for expert knowledge . Similarly , the inductive bias promoted by RNNs and LSTMs ( and more recent models such as transformers ) favors both locality and temporal stationarity . When considering tabular data , however , neural networks are not the hypothesis class of choice . Most often , the winning class in learning problems involving tabular data is decision forests . In Kaggle competitions , for example , gradient boosting of decision trees ( GBDTs ) ( Chen & Guestrin , 2016 ; Friedman , 2001 ; Prokhorenkova et al. , 2018 ; Ke et al. , 2017 ) are generally the superior model . While it is quite practical to use GBDTs for medium size datasets , it is extremely hard to scale these methods to very large datasets . Scaling up the gradient boosting models was addressed by several papers ( Ye et al. , 2009 ; Tyree et al. , 2011 ; Fu et al. , 2019 ; Vasiloudis et al. , 2019 ) . The most significant computational disadvantage of GBDTs is the need to store ( almost ) the entire dataset in memory1 . Moreover , handling multi-modal data , which involves both tabular and spatial data ( e.g. , medical records and images ) , is problematic . Thus , since GBDTs and neural networks can not be organically optimized , such multi-modal tasks are left with sub-optimal solutions . The creation of a purely neural model for tabular data , which can be trained with SGD end-to-end , is therefore a prime open objective . A few works have aimed at constructing neural models for tabular data ( see Section 5 ) . Currently , however , there is still no widely accepted end-to-end neural architecture that can handle tabular data and consistently replace fully-connected architectures , or better yet , replace GBDTs . Here we present Net-DNFs , a family of neural network architectures whose primary inductive bias is an ensemble comprising a disjunctive normal form ( DNF ) formulas over linear separators . This family also promotes ( input ) feature selection and spatial localization of ensemble members . These inductive 1This disadvantage is shared among popular GBDT implementations : XGBoost , LightGBM , and CatBoost . biases have been included by design to promote conceptually similar elements that are inherent in GBDTs and random forests . Appealingly , the Net-DNF architecture can be trained end-to-end using standard gradient-based optimization . Importantly , it consistently and significantly outperforms FCNs on tabular data , and can sometime even outperform GBDTs . The choice of appropriate inductive bias for specialized hypothesis classes for tabular data is challenging since , clearly , there are many different kinds of such data . Nevertheless , the “ universality ” of forest methods in handling a wide variety of tabular data suggests that it might be beneficial to emulate , using neural networks , the important elements that are part of the tree ensemble representation and algorithms . Concretely , every decision tree is equivalent to some DNF formula over axis-aligned linear separators ( see details in Section 3 ) . This makes DNFs an essential element in any such construction . Secondly , all contemporary forest ensemble methods rely heavily on feature selection . This feature selection is manifested both during the induction of each individual tree , where features are sequentially and greedily selected using information gain or other related heuristics , and by uniform sampling features for each ensemble member . Finally , forest methods include an important localization element – GBDTs with their sequential construction within a boosting approach , where each tree re-weights the instance domain differently – and random forests with their reliance on bootstrap sampling . Net-DNFs are designed to include precisely these three elements . After introducing Net-DNF , we include a Vapnik-Chervonenkins ( VC ) comparative analysis of DNFs and trees showing that DNFs potentially have advantage over trees when the input dimension is large and vice versa . We then present an extensive empirical study . We begin with an ablation study over three real-life tabular data prediction tasks that convincingly demonstrates the importance of all three elements included in the Net-DNF design . Second , we analyze our novel feature selection component over controlled synthetic experiments , which indicate that this component is of independent interest . Finally , we compare Net-DNFs to FCNs and GBDTs over several large classification tasks , including two past Kaggle competitions . Our results indicate that Net-DNFs consistently outperform FCNs , and can sometime even outperform GBDTs . 2 DISJUNCTIVE NORMAL FORM NETWORKS ( NET-DNFS ) . In this section we introduce the Net-DNF architecture , which consists of three elements . The main component is a block of layers emulating a DNF formula . This block will be referred to as a Disjunctive Normal Neural Form ( DNNF ) . The second and third components , respectively , are a feature selection module , and a localization one . In the remainder of this section we describe each component in detail . Throughout our description we denote by x ∈ Rd a column of input feature vectors , by xi , its ith entry , and by σ ( · ) the sigmoid function . 2.1 A DISJUNCTIVE NORMAL NEURAL FORM ( DNNF ) BLOCK . A disjunctive normal neural form ( DNNF ) block is assembled using a two-hidden-layer network . The first layer creates affine “ literals ” ( features ) and is trainable . The second layer implements a number of soft conjunctions over the literals , and the third output layer is a neural OR gate . Importantly , only the first layer is trainable , while the two other are binary and fixed . We begin by describing the neural AND and OR gates . For an input vector x , we define soft , differentiable versions of such gates as OR ( x ) , tanh ( d∑ i=1 xi + d− 1.5 ) , AND ( x ) , tanh ( d∑ i=1 xi − d+ 1.5 ) . These definitions are straightforwardly motivated by the precise neural implementation of the corresponding binary gates . Notice that by replacing tanh by a binary activation and changing the bias constant from 1.5 to 1 , we obtain an exact implementation of the corresponding logical gates for binary input vectors ( Anthony , 2005 ; Shalev-Shwartz & Ben-David , 2014 ) ; see a proof of this statement in Appendix A . Notably , each unit does not have any trainable parameters . We now define the AND gate in a vector form to project the logical operation over a subset of variables . The projection is controlled by an indicator column vector ( a mask ) u ∈ { 0 , 1 } d. With respect to such a projection vector u , we define the corresponding projected gate as ANDu ( x ) , tanh ( uTx− ||u||1 + 1.5 ) . Equipped with these definitions , a DNNF ( x ) : Rd → R with k conjunctions over m literals is , L ( x ) , tanh ( xTW + b ) ∈ Rm ( 1 ) DNNF ( x ) , OR ( [ ANDc1 ( L ( x ) ) , ANDc2 ( L ( x ) ) , . . . , ANDck ( L ( x ) ) ] ) . ( 2 ) Equation ( 1 ) defines L ( x ) that generates m “ neural literals ” , each of which is the result of a tanhactivation of a ( trainable ) affine transformation . The ( trainable ) matrix W ∈ Rd×m , as well as the row vector bias term b ∈ Rm , determine the affine transformations for each literal such that each of its columns corresponds to one literal . Equation ( 2 ) defines a DNNF . In this equation , the vectors ci ∈ { 0 , 1 } m , 1 ≤ i ≤ k , are binary indicators such that cij = 1 iff the jth literal belongs to the ith conjunction . In our design , each literal belongs to a single conjunction . These indicator vectors are defined and fixed according to the number and length of the conjunctions ( See Appendix D.2 ) . 2.2 NET-DNFS . The embedding layer of a Net-DNF with n DNNF blocks is a simple concatenation E ( x ) , [ DNNF1 ( x ) , DNNF2 ( x ) , . . . , DNNFn ( x ) ] . ( 3 ) Depending on the application , the final Net-DNF is a composition of an output layer over E ( x ) . For example , for binary classification ( logistic output layer ) , Net-DNF ( x ) : Rd → ( 0 , 1 ) is , Net-DNF ( x ) , σ ( n∑ i=1 wiDNNFi ( x ) + bi ) . ( 4 ) To summarize , a Net-DNF is always a four-layer network ( including the output layer ) , and only the first and last layers are learned . Each DNNF block has two parameters : the number of conjunctions k and the length m of these conjunctions , allowing for a variety of Net-DNF architectures . In all our experiments we considered a single Net-DNF architecture that has a fixed diversity of DNNF blocks which includes a number of different DNNF groups with different k , each of which has a number of conjunction sizes m ( see details in Appendix D.2 ) . The number n of DNNFs was treated as a hyperparameter , and selected based on a validation set as described on Appendix D.1 . 2.3 FEATURE SELECTION . One key strategy in decision tree training is greedy feature selection , which is performed hierarchically at any split , and allows decision trees to exclude irrelevant features . Additionally , decision tree ensemble algorithms apply random sampling to select a subset of the features , which is used to promote diversity , and prevent different trees focusing on the same set of dominant features in their greedy selection . In line with these strategies , we include in our Net-DNFs conceptually similar feature selection elements : ( 1 ) a subset of features uniformly and randomly sampled for each DNNF ; ( 2 ) a trainable mechanism for feature selection , applied on the resulting random subset . These two elements are combined and implemented in the affine literal generation layer described in Equation ( 1 ) , and applied independently for each DNNF . We now describe these techniques in detail . Recalling that d is the input dimension , the random selection is made by generating a stochastic binary mask , ms ∈ { 0 , 1 } d ( each block has its own mask ) , such that the probability of any entry being 1 is p ( see Appendix D.2 for details on setting this parameter ) . For a given mask ms , this selection can be applied over affine literals using a simple product diag ( ms ) W , where W is the matrix of Equation ( 1 ) . We then construct a trainable mask mt ∈ Rd , which will be applied on the features that are kept by ms . We introduce a novel trainable feature selection component that combines binary quantization of the mask together with modified elastic-net regularization . To train a binarized vector we resort to the straight-through estimator ( Hinton , 2012 ; Hubara et al. , 2017 ) , which can be used effectively to train non-differentiable step functions such as a threshold or sign . The trick is to compute the step function exactly in the forward pass , and utilize a differentiable proxy in the backward pass . We use a version of the straight-through estimator for the sign function ( Bengio et al. , 2013 ) , Φ ( x ) , { sign ( x ) , forward pass ; tanh ( x ) , backward pass . Using the estimator Φ ( x ) , we define a differentiable binary threshold function T ( x ) = 12Φ ( |x|− ) + 1 2 , where ∈ R defines an epsilon neighborhood around zero for which the output of T ( x ) is zero , and one outside of this neighborhood ( in all our experiments , we set = 1 and initialize the entries of mt above this threshold ) . We then apply this selection by diag ( T ( mt ) ) W . Given a fixed stochastic selection ms , to train the binarized selection mt we employ regularization . Specifically , we consider a modified version of the elastic net regularization , R ( mt , ms ) , which is tailored to our task . The modifications are reflected in two parts . First , the balancing between the L1 and L2 regularization is controlled by a trainable parameter α ∈ R. Second , the expressions of the L1 and L2 regularization are replaced by R1 ( mt , ms ) , R2 ( mt , ms ) , respectively ( defined below ) . Moreover , since we want to take into account only features that were selected by the random component , the regularization is applied on the vector mts = mt ms , where is element-wise multiplication . The functional form of the modified elastic net regularization is as follows , R2 ( mt , ms ) , ∣∣∣∣ ||mts||22||ms||1 − β 2 ∣∣∣∣ , R1 ( mt , ms ) , ∣∣∣∣ ||mts||1||ms||1 − β ∣∣∣∣ R ( mt , ms ) , 1− σ ( α ) 2 R2 ( mt , ms ) + σ ( α ) R1 ( mt , ms ) . The above formulation of R2 ( · ) and R1 ( · ) is motivated as follows . First , we normalize both norms by dividing with the effective input dimension , ||ms||1 , which is done to be invariant to the ( effective ) input size . Second , we define R2 and R1 as absolute errors , which encourages each entry to be , on average , approximately equal to the threshold . The reason is that the vector mt passes through a binary threshold , and though the exact values of its entries are irrelevant . What is relevant is whether these values are within epsilon neighborhood of zero or not . Thus , when the values are roughly equal to the threshold , it is more likely to converge to a balanced point where the regularization term is low and the relevant features were selected . The threshold term is controlled by β ( a hyperparameter ) , which controls the cardinality of mt , where smaller values of β lead to sparser mt . To summarize , feature selection is manifested by both architecture and loss . Architecture relies on the masks mt , ms , while the loss function uses R ( mt , ms ) . Finally , the functional form of a DNNF block with the feature selection component is obtained by plugging the masks into Equation ( 2 ) , L ( x ) , tanh ( xT diag ( T ( mt ) ) diag ( ms ) W + b ) ∈ Rm . Additionally , the mean over R ( mt , ms ) in all DNNFs is added to the loss function as a regularizer .

Summary: The authors propose a end-to-end deep learning model called Net-DNF to handle tabular data. The architecture of Net-DNF has four layers: the first layer is a dense layer (learnable weights) with tanh activation eq(1). The second layer (DNNF) is formed by binary conjunctions over literals eq(2). The third layer is an embedding formed by n DNNF blocks  eq(3). the last layer is a linear transformation of the embedding with a sigmoid activation eq(4). The authors also propose a feature selection method based on a trainable binarized selection with a modified L1 and L2 regularization. In the experimental analysis, Net-DNF outperforms fully connected networks.