Regularization Cocktails for Tabular Datasets

1 INTRODUCTION . In most supervised learning application domains , the available data for training predictive models is both limited and noisy with respect to the target variable . Therefore , it is paramount to regularize machine learning models for generalizing the predictive performance on future unseen data . The concept of regularization is well-studied and constitutes one of the pillar components of machine learning . Throughout this work we use the term “ regularization ” for all methods that explicitly or implicitly take measures that reduce the overfitting phenomenon ; we categorize these non-exhaustively into weight decay , data augmentation , model averaging , structure and linearization , and implicit regularization families ( detailed in Section 2 ) . In this paper , we propose a new principled strategy that highlights the need for automatically learning the optimal combination of regularizers , denoted as regularization cocktails , via a hyperparameter optimization procedure . Truth be told , combining regularization methods is of course far from being a novel practice per se . As a matter of fact , most modern deep learning models use combinations of a number of regularizers . For instance , EfficientNet ( Tan & Q.Le , 2019 ) mixes components of structural regularization and linearization via ResNet-style skip connections ( He et al. , 2016 ) , learning rate scheduling , DropOut ensembling ( Srivastava et al. , 2014 ) and AutoAugment data augmentation ( Cubuk et al. , 2019 ) . However , even though each of those regularizers is motivated in isolation , the reasoning behind a specific combination of regularizers is largely based on accuracy-driven manual trial-and-error iterations , mostly on image classification benchmarks like CIFAR ( Krizhevsky et al. , 2009 ) and ImageNet ( Deng et al. , 2009 ) . Unfortunately , the manual search for combinations of regularizers is sub-optimal , unsustainable , and in essence consists of an example of manual hyperparameter tuning , which in turn is easily outperformed by automated algorithms ( Snoek et al. , 2012 ; Thornton et al. , 2013 ; Feurer et al. , 2015 ; Olson & Moore , 2016 ; Jin et al. , 2019 ; Erickson et al. , 2020 ; Zimmer et al. , 2020 ) . Following the spirit of AutoML ( Hutter et al. , 2018 ) , we , therefore , propose a strategy for learning the optimal dataset-specific regularization cocktail by means of a modern hyperparameter optimization ( HPO ) method . To the best of our knowledge , there exists no study providing empirical evidence that a mixture of numerous regularizers outperforms individual regularizers ; this paper fills this gap . More precisely , the research hypothesis of this paper is that a properly mixed regularization cocktail outperforms every individual regularizer in it , in terms of accuracy under the same run-time budget , and that the best cocktail to use depends on the dataset . To validate this hypothesis , we executed a large-scale experimental study employing 40 diverse tabular datasets and 13 prominent regularizers with a thorough hyperparameter tuning for all regularizers . We focus on tabular datasets because , in contrast to large image datasets , a thorough hyper-parameter search procedure is feasible . Neural networks are high variance models for tabular datasets , therefore improved regularization schemes can provide a relatively higher generalization gain on tabular datasets compared to other data types . Thereby , we make the followings contributions : 1 . We demonstrate the empirical accuracy gains of regularization cocktails in a systematic manner via a large-scale experimental study on tabular datasets ; 2 . We challenge the status-quo practices of designing universal dataset-agnostic regularizers , by showing that an optimal regularization cocktail is highly dataset-dependent ; 3 . We demonstrate that regularization cocktails achieve state-of-the-art classification accuracy performance on tabular datasets and outperform Gradient-Boosted Decision Trees ( GBDT ) with a statistically-significant margin ; 4 . As an overarching contribution , this paper provides previously-lacking in-depth empirical evidence to better understand the importance of combining different mechanisms for regularization , one of the most fundamental concepts in machine learning . 2 RELATED WORK . Weight decay : The classical approaches of regularization focused on minimizing the norms of the parameter values , concretely either the L1 ( Tibshirani , 1996 ) , the L2 ( Tikhonov , 1943 ) , or a combination of L1 and L2 known as the Elastic Net ( Zou & Hastie , 2005 ) . A recent work fixes the malpractice of adding the decay penalty term before momentum-based adaptive learning rate steps ( e.g. , in common implementations of Adam ( Kingma & Ba , 2015 ) ) , by decoupling the regularization from the loss and applying it after the learning rate computation ( Loshchilov & Hutter , 2019 ) . Data Augmentation : A different treatment of the overfitting phenomenon relies on enriching the training dataset via instance augmentation . The literature on data augmentation is vast , especially for image data , ranging from basic image manipulations ( e.g. , geometric transformations , or mixing images ) up to parametric augmentation strategies such as adversarial and controller-based methods ( Shorten & Khoshgoftaar , 2019 ) . For example , Cut-Out ( Devries & Taylor , 2017 ) proposes to mask a subset of input features ( e.g. , pixel patches for images ) for ensuring that the predictions remain invariant to distortions in the input space . Along similar lines , Mix-Up ( Zhang et al. , 2018 ) generates new instances as a linear span of pairs of training examples , while Cut-Mix ( Yun et al. , 2019 ) suggests super-positions of instance pairs with mutually-exclusive pixel masks . A recent technique , called Aug-Mix ( Hendrycks et al. , 2020 ) , generates instances by sampling chains of augmentation operations . On the other hand , the direction of reinforcement learning ( RL ) for augmentation policies was elaborated by Auto-Augment ( Cubuk et al. , 2019 ) , followed by a technique that speeds up the training of the RL policy ( S.Lim et al. , 2019 ) . Last but not least , adversarial attack strategies ( e.g. , FGSM ( Goodfellow et al. , 2015 ) ) generate synthetic examples with minimal perturbations , which are employed in training robust models ( Madry et al. , 2018 ) . Model Averaging : Ensembled machine learning models have been shown to reduce variance and act as regularizers ( Polikar , 2012 ) . A popular ensemble neural network with shared weights among its base models is Drop-Out ( Srivastava et al. , 2014 ) , which was extended to a variational version with a Gaussian posterior of the model parameters ( Kingma et al. , 2015 ) . A follow-up work that is known as Mix-Out ( Lee et al. , 2020 ) extends Drop-Out by statistically fusing the parameters of two base models . Furthermore , ensembles can be created using models from the local optima discovered along a single convergence procedure ( Huang et al. , 2016 ) . Structural and Linearization : One strategy of regularizing deep learning models is to discover dedicated neural structures that generalize on particular tasks , such as image classification or Natural Language Processing ( NLP ) . In that context , ResNet adds skip connections across layers ( He et al. , 2016 ) , while the Inception model computes latent representations by aggregating diverse convolutional filter sizes ( Szegedy et al. , 2017 ) . The attention mechanism gave rise to the popular Transformer architecture in the realm of NLP ( Vaswani et al. , 2017 ) . Recently , EfficientNet is an architecture that easily scales deep convolutional neural networks by controlling only a few hyperparameters ( Tan & Q.Le , 2019 ) . Besides the aforementioned manually-designed architectures , the stream of Neural Architecture Search ( Elsken et al. , 2019 ) focuses on exploring neural connectivity graphs for finding the optimal architectures via reinforcement learning ( Zoph & Le , 2017 ) , black-box search ( Real et al. , 2019 ) or differentiable solvers ( Liu et al. , 2019 ) . A recent trend adds a dosage of linearization to deep models , where skip connections transfer embeddings from previous less non-linear layers ( He et al. , 2016 ; Huang et al. , 2017 ) . Along similar lines , the Shake-Shake regularization deploys skip connections in parallel convolutional blocks and aggregates the parallel representations through affine combinations ( Gastaldi , 2017 ) , while Shake-Drop extends this mechanism to a larger number of CNN architectures ( Yamada et al. , 2018 ) . Implicit : The last family of regularizers broadly encapsulates methods which do not directly propose novel regularization techniques but have an implicit regularization effect as a virtue of their ‘ modus operandi ’ ( Arora et al. , 2019 ) . For instance , Batch Normalization improves generalization by reducing the internal covariate shifts ( Ioffe & Szegedy , 2015 ) , while early stopping of the optimization procedure also yields a similar generalization effect ( Yao et al. , 2007 ) . On the other hand , stabilizing the convergence of the training routine is another implicit regularization , for instance by introducing learning rate scheduling schemes ( Loshchilov & Hutter , 2017 ) . The recent strategy of stochastic weight averaging relies on averaging parameter values from the local optima encountered along the sequence of optimization steps ( Izmailov et al. , 2018 ) , while another approach conducts updates in the direction of a few ‘ lookahead ’ steps ( Zhang et al. , 2019 ) . Positioning in the realm of AutoML : In contrast to the prior literature , we do not propose a new individual regularization method , but empirically identify the superiority of learning regularization cocktails among a set of existing regularizers from the aforementioned categories . We train datasetspecific cocktails as a hyperparameter optimization ( HPO ) task ( Feurer & Hutter , 2019 ) . In that regard , our work is positioned in the realm of AutoML and is a special case of a combined algorithm selection and hyperparameter optimization ( Thornton et al. , 2013 ) . We learn the regularization cocktails and optimize the joint hyperparameter configuration space by means of BOHB ( Falkner et al. , 2018b ) , which is a variation of Hyperband ( Li et al. , 2017 ) with model-based surrogates and is one of the current state of the art approaches for efficient HPO . 3 MIXING THE REGULARIZATION COCKTAIL . 3.1 THE RECIPE AS PROBLEM STATEMENT . A training set is composed of features X ( Train ) and targets y ( Train ) , while the test dataset is denoted by X ( Test ) , y ( Test ) . A parameterized function approximates the targets as ŷ = f ( X ; θ ) whereas the parameters θ are trained to minimize a differentiable loss function L as : θ∗ ∈ arg min θ L ( y ( Train ) , f ( X ( Train ) ; θ ) ) . ( 1 ) To generalize into minimizing L ( y ( Test ) , f ( X ( Test ) ; θ ) , the parameters of f are controlled with a regularization technique Ω that avoids overfitting to the peculiarities of the training data . With a slight abuse of notation we denote f ( X ; Ω ( θ ; λ ) ) to be the predictions of the model f whose parameters θ are optimized under the regime of the regularization method Ω ( · ; λ ) , where λ ∈ Λ represent the hyperparameters of Ω . The training data is further divided into two subsets as training and validation splits,1 the later denoted by X ( Val ) , y ( Val ) , such that λ can be tuned on the validation loss via the following hyperparameter optimization objective : λ∗ ∈ arg min λ∈Λ L ( y ( Val ) , f ( X ( Val ) ; Ω ( θ∗ ; λ ) ) ) , ( 2 ) s.t . θ∗ ∈ arg min θ L ( y ( Train ) , f ( X ( Train ) ; Ω ( θ ; λ ) ) . ( 3 ) 1For simplicity , we only discuss hold-out validation scheme here , but in principle any other validation scheme , such as cross validation and bootstrap sampling , would be possible . While the search for optimal hyperparameters λ is an active field of research in the realm of AutoML ( Hutter et al. , 2018 ) , still the choice of the regularizer Ω mostly remains an ad-hoc practice , where practitioners select few combinations among popular regularizers ( Dropout , L2 , Batch Normalization , etc. ) . In contrast to prior studies , we hypothesize that the optimal regularizer is a cocktail mixture of a large set of regularization methods , all being simultaneously applied with different strengths ( i.e. , dataset-specific hyperparameters ) . Given a set of K regularizers { ( Ω ( k ) ( · ; λ ( k ) ) } K k=1 : = { Ω ( 1 ) ( · ; λ ( 1 ) ) , . . . , Ω ( K ) ( · ; λ ( K ) ) } , each with its own hyperparameters λ ( k ) ∈ Λ ( k ) , ∀k ∈ { 1 , . . . , K } , the problem of finding the optimal cocktail of regularizers is : λ∗ ∈ arg min λ ( 1 ) ∈Λ ( k ) , ... , λ ( K ) ∈Λ ( k ) L ( y ( Val ) , f ( X ( Val ) ; { Ω ( k ) ( θ∗ , λ ( k ) ) } K k=1 ) ) ( 4 ) s.t . θ∗ ∈ arg min θ L ( y ( Train ) , f ( X ( Train ) ; { Ω ( k ) ( θ , λ ( k ) ) } K k=1 ) ) ( 5 ) The intuitive interpretation of Equations 4-5 is searching for the optimal hyperparameters λ ( i.e. , strengths ) of the cocktail ’ s regularizers using the validation set ( Equation 4 ) , given that the optimal prediction model parameters θ are trained under the regime of all the regularizers being applied jointly ( Equation 5 ) . We stress that the hyperparameters λ ( k ) include a conditional hyperparameter controlling whether the k-th regularizer is applied at all , or skipped . Therefore , the best cocktail might consist of combinations of a subset of regularizers .

Summary: This work takes a step towards understanding the effect of automated selection of regularisation techniques and analyses the results across 42 structured datasets. It defines a search space over 13 regularisation techniques and employs one flavour of Bayesian Optimisation + Hyperband approach to find an optimal combination of regularisers. It concludes by substantiating three claims with corresponding experiments.