Well-tuned Simple Nets Excel on Tabular Datasets

1 Introduction . In contrast to the mainstream in deep learning ( DL ) , in this paper , we focus on tabular data , a domain that we feel is understudied in DL . Nevertheless , it is of great relevance for many practical applications , such as climate science , medicine , manufacturing , finance , recommender systems , etc . During the last decade , traditional machine learning methods , such as Gradient-Boosted Decision Trees ( GBDT ) [ 5 ] , dominated tabular data applications due to their superior performance , and the success story DL has had for raw data ( e.g. , images , speech , and text ) stopped short of tabular data . Even in recent years , the existing literature still gives mixed messages on the state-of-the-art status of deep learning for tabular data . While some recent neural network methods [ 1 , 46 ] claim to outperform GBDT , others confirm that GBDT are still the most accurate method on tabular data [ 48 , 26 ] . The extensive experiments on 40 datasets we report indeed confirm that recent neural networks [ 1 , 46 , 11 ] do not outperform GBDT when the hyperparameters of all methods are thoroughly tuned . We hypothesize that the key to improving the performance of neural networks on tabular data lies in exploiting the recent DL advances on regularization techniques ( reviewed in Section 3 ) , such as data augmentation , decoupled weight decay , residual blocks and model averaging ( e.g. , dropout or snapshot ensembles ) , or on learning dynamics ( e.g. , look-ahead optimizer or stochastic weight averaging ) . 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . Indeed , we find that even plain Multilayer Perceptrons ( MLPs ) achieve state-of-the-art results when regularized by multiple modern regularization techniques applied jointly and simultaneously . Applying multiple regularizers jointly is already a common standard for practitioners , who routinely mix regularization techniques ( e.g . Dropout with early stopping and weight decay ) . However , the deeper question of “ Which subset of regularizers gives the largest generalization performance on a particular dataset among dozens of available methods ? ” remains unanswered , as practitioners currently combine regularizers via inefficient trial-and-error procedures . In this paper , we provide a simple , yet principled answer to that question , by posing the selection of the optimal subset of regularization techniques and their inherent hyperparameters , as a joint search for the best combination of MLP regularizers for each dataset among a pool of 13 modern regularization techniques and their subsidiary hyperparameters ( Section 4 ) . From an empirical perspective , this paper is the first to provide compelling evidence that wellregularized neural networks ( even simple MLPs ! ) indeed surpass the current state-of-the-art models in tabular datasets , including recent neural network architectures and GBDT ( Section 6 ) . In fact , the performance improvements are quite pronounced and highly significant.1 We believe this finding to potentially have far-reaching implications , and to open up a garden of delights of new applications on tabular datasets for DL . Our contributions are as follows : 1 . We demonstrate that modern DL regularizers ( developed for DL applications on raw data , such as images , speech , or text ) also substantially improve the performance of deep multilayer perceptrons on tabular data . 2 . We propose a simple , yet principled , paradigm for selecting the optimal subset of regularization techniques and their subsidiary hyperparameters ( so-called regularization cocktails ) . 3 . We demonstrate that these regularization cocktails enable even simple MLPs to outperform both recent neural network architectures , as well as traditional strong ML methods , such as GBDT , on tabular data . Specifically , we are the first to show neural networks to significantly ( and substantially ) outperform XGBoost in a fair , large-scale experimental study . 2 Related Work on Deep Learning for Tabular Data . Recently , various neural architectures have been proposed for improving the performance of neural networks on tabular data . TabNet [ 1 ] introduced a sequential attention mechanism for capturing salient features . Neural oblivious decision ensembles ( NODE [ 46 ] ) blend the concept of hierarchical decisions into neural networks . Self-normalizing neural networks [ 29 ] have neuron activations that converge to zero mean and unit variance , which in turn , induces strong regularization and allows for high-level representations . Regularization learning networks train a regularization strength on every neural weight by posing the problem as a large-scale hyperparameter tuning scheme [ 48 ] . The recent NET-DNF technique introduces a novel inductive bias in the neural structure corresponding to logical Boolean formulas in disjunctive normal forms [ 26 ] . An approach that is often mistaken as deep learning for tabular data is AutoGluon Tabular [ 11 ] , which builds ensembles of basic neural networks together with other traditional ML techniques , with its key contribution being a strong stacking approach . We emphasize that some of these publications claim to outperform Gradient Boosted Decision Trees ( GDBT ) [ 1 , 46 ] , while other papers explicitly stress that the neural networks tested do not outperform GBDT on tabular datasets [ 48 , 26 ] . In contrast , we do not propose a new kind of neural architecture , but a novel paradigm for learning a combination of regularization methods . 3 An Overview of Regularization Methods for Deep Learning . Weight decay : The most classical approaches of regularization focused on minimizing the norms of the parameter values , e.g. , either the L1 [ 51 ] , the L2 [ 52 ] , or a combination of L1 and L2 known as the Elastic Net [ 63 ] . A recent work fixes the malpractice of adding the decay penalty term before 1In that sense , this paper adds to the growing body of literature on demonstrating the sensitivity of modern ML methods to hyperparameter settings [ 19 ] , demonstrating that proper hyperparameter tuning can yield substantial improvements of modern ML methods [ 6 ] , and demonstrating that even simple architectures can obtain state-of-the-art performance with proper hyperparameter settings [ 38 ] . momentum-based adaptive learning rate steps ( e.g. , in common implementations of Adam [ 27 ] ) , by decoupling the regularization from the loss and applying it after the learning rate computation [ 36 ] . Data Augmentation : Among the augmentation regularizers , Cut-Out [ 10 ] proposes to mask a subset of input features ( e.g. , pixel patches for images ) for ensuring that the predictions remain invariant to distortions in the input space . Along similar lines , Mix-Up [ 60 ] generates new instances as a linear span of pairs of training examples , while Cut-Mix [ 58 ] suggests super-positions of instance pairs with mutually-exclusive pixel masks . A recent technique , called Aug-Mix [ 20 ] , generates instances by sampling chains of augmentation operations . On the other hand , the direction of reinforcement learning ( RL ) for augmentation policies was elaborated by Auto-Augment [ 7 ] , followed by a technique that speeds up the training of the RL policy [ 34 ] . Recently , these complex and expensive methods were superseded by simple and cheap methods that yield similar performance ( RandAugment [ 8 ] ) or even improve on it ( TrivialAugment [ 41 ] ) . Last but not least , adversarial attack strategies ( e.g. , FGSM [ 17 ] ) generate synthetic examples with minimal perturbations , which are employed in training robust models [ 37 ] . Ensemble methods : Ensembled machine learning models have been shown to reduce variance and act as regularizers [ 45 ] . A popular ensemble neural network with shared weights among its base models is Dropout [ 49 ] , which was extended to a variational version with a Gaussian posterior of the model parameters [ 28 ] . As a follow-up , Mix-Out [ 32 ] extends Dropout by statistically fusing the parameters of two base models . Furthermore , so-called snapshot ensembles [ 21 ] can be created using models from intermediate convergence points of stochastic gradient descent with restarts [ 35 ] . In addition to these efficient ensembling approaches , ensembling independent classifiers trained in separate training runs can yield strong performance ( especially for uncertainty quantification ) , be it based on independent training runs only differing in random seeds ( deep ensembles [ 31 ] ) , training runs differing in hyperparameter settings ( hyperdeep ensembles , [ 55 ] ) , or training runs with different neural architectures ( neural ensemble search [ 59 ] ) . Structural and Linearization : In terms of structural regularization , ResNet adds skip connections across layers [ 18 ] , while the Inception model computes latent representations by aggregating diverse convolutional filter sizes [ 50 ] . A recent trend adds a dosage of linearization to deep models , where skip connections transfer embeddings from previous less non-linear layers [ 18 , 22 ] . Along similar lines , the Shake-Shake regularization deploys skip connections in parallel convolutional blocks and aggregates the parallel representations through affine combinations [ 15 ] , while Shake-Drop extends this mechanism to a larger number of CNN architectures [ 56 ] . Implicit : The last family of regularizers broadly encapsulates methods that do not directly propose novel regularization techniques but have an implicit regularization effect as a virtue of their ‘ modus operandi ’ [ 2 ] . The simplest such implicit regularization is Early Stopping [ 57 ] , which limits overfitting by tracking validation performance over time and stopping training when validation performance no longer improves . Another implicit regularization method is Batch Normalization , which improves generalization by reducing internal covariate shift [ 24 ] . The scaled exponential linear units ( SELU ) represent an alternative to batch-normalization through self-normalizing activation functions [ 30 ] . On the other hand , stabilizing the convergence of the training routine is another implicit regularization , for instance by introducing learning rate scheduling schemes [ 35 ] . The recent strategy of stochastic weight averaging relies on averaging parameter values from the local optima encountered along the sequence of optimization steps [ 25 ] , while another approach conducts updates in the direction of a few ‘ lookahead ’ steps [ 61 ] . 4 Regularization Cocktails for Multilayer Perceptrons . 4.1 Problem Definition . A training set is composed of features X ( Train ) and targets y ( Train ) , while the test dataset is denoted by X ( Test ) , y ( Test ) . A parametrized function f , i.e. , a neural network , approximates the targets as ŷ = f ( X ; θ ) , where the parameters θ are trained to minimize a differentiable loss function L as arg minθ L ( y ( Train ) , f ( X ( Train ) ; θ ) ) . To generalize into minimizing L ( y ( Test ) , f ( X ( Test ) ; θ ) , the parameters of f are controlled with a regularization technique Ω that avoids overfitting to the peculiarities of the training data . With a slight abuse of notation we denote f ( X ; Ω ( θ ; λ ) ) to be the predictions of the model f whose parameters θ are optimized under the regime of the regularization method Ω ( · ; λ ) , where λ ∈ Λ represents the hyperparameters of Ω . The training data is further divided into two subsets as training and validation splits , the later denoted by X ( Val ) , y ( Val ) , such that λ can be tuned on the validation loss via the following hyperparameter optimization objective : λ∗ ∈ arg min λ∈Λ L ( y ( Val ) , f ( X ( Val ) ; θ∗λ ) ) , ( 1 ) s.t . θ∗λ ∈ arg min θ L ( y ( Train ) , f ( X ( Train ) ; Ω ( θ ; λ ) ) . After finding the optimal ( or in practice at least a well-performing ) configuration λ∗ , we re-fit θ on the entire training dataset , i.e. , X ( Train ) ∪X ( Val ) and y ( Train ) ∪ y ( Val ) . While the search for optimal hyperparameters λ is an active field of research in the realm of AutoML [ 23 ] , still the choice of the regularizer Ω mostly remains an ad-hoc practice , where practitioners select a few combinations among popular regularizers ( Dropout , L2 , Batch Normalization , etc. ) . In contrast to prior studies , we hypothesize that the optimal regularizer is a cocktail mixture of a large set of regularization methods , all being simultaneously applied with different strengths ( i.e. , dataset-specific hyperparameters ) . Given a set of K regularizers { ( Ω ( k ) ( · ; λ ( k ) ) } K k=1 : = { Ω ( 1 ) ( · ; λ ( 1 ) ) , . . . , Ω ( K ) ( · ; λ ( K ) ) } , each with its own hyperparameters λ ( k ) ∈ Λ ( k ) , ∀k ∈ { 1 , . . . , K } , the problem of finding the optimal cocktail of regularizers is : λ∗ ∈ arg min λ : = ( λ ( 1 ) , ... , λ ( K ) ) ∈ ( Λ ( 1 ) , ... , Λ ( K ) ) L ( y ( Val ) , f ( X ( Val ) ; θ∗λ ) ) ( 2 ) s.t . : θ∗λ ∈ arg min θ L ( y ( Train ) , f ( X ( Train ) ; { Ω ( k ) ( θ , λ ( k ) ) } K k=1 ) ) The intuitive interpretation of Equation 2 is searching for the optimal hyperparameters λ ( i.e. , strengths ) of the cocktail ’ s regularizers using the validation set , given that the optimal prediction model parameters θ are trained under the regime of all the regularizers being applied jointly . We stress that , for each regularizer , the hyperparameters λ ( k ) include a conditional hyperparameter controlling whether the k-th regularizer is applied or skipped . The best cocktail might comprise only a subset of regularizers .

Summary: The paper supplies an empirical existence proof that an "appropriately regularized" multi-layer perceptron (MLP)-like model can outperform gradient-boosted decision trees (GBDT) on tabular data. The paper's proposed approach defines a set of parameterized techniques which includes ensembling, architectural changes, and data augmentation, and performs search over combinations from this set. The extensive experimental comparison across 40 tabular datasets shows that MLP-like models can outperform GBDTs and other neural network architectures.