Regularization Cocktails for Tabular Datasets

1 INTRODUCTION . In most supervised learning application domains , the available data for training predictive models is both limited and noisy with respect to the target variable . Therefore , it is paramount to regularize machine learning models for generalizing the predictive performance on future unseen data . The concept of regularization is well-studied and constitutes one of the pillar components of machine learning . Throughout this work we use the term “ regularization ” for all methods that explicitly or implicitly take measures that reduce the overfitting phenomenon ; we categorize these non-exhaustively into weight decay , data augmentation , model averaging , structure and linearization , and implicit regularization families ( detailed in Section 2 ) . In this paper , we propose a new principled strategy that highlights the need for automatically learning the optimal combination of regularizers , denoted as regularization cocktails , via a hyperparameter optimization procedure . Truth be told , combining regularization methods is of course far from being a novel practice per se . As a matter of fact , most modern deep learning models use combinations of a number of regularizers . For instance , EfficientNet ( Tan & Q.Le , 2019 ) mixes components of structural regularization and linearization via ResNet-style skip connections ( He et al. , 2016 ) , learning rate scheduling , DropOut ensembling ( Srivastava et al. , 2014 ) and AutoAugment data augmentation ( Cubuk et al. , 2019 ) . However , even though each of those regularizers is motivated in isolation , the reasoning behind a specific combination of regularizers is largely based on accuracy-driven manual trial-and-error iterations , mostly on image classification benchmarks like CIFAR ( Krizhevsky et al. , 2009 ) and ImageNet ( Deng et al. , 2009 ) . Unfortunately , the manual search for combinations of regularizers is sub-optimal , unsustainable , and in essence consists of an example of manual hyperparameter tuning , which in turn is easily outperformed by automated algorithms ( Snoek et al. , 2012 ; Thornton et al. , 2013 ; Feurer et al. , 2015 ; Olson & Moore , 2016 ; Jin et al. , 2019 ; Erickson et al. , 2020 ; Zimmer et al. , 2020 ) . Following the spirit of AutoML ( Hutter et al. , 2018 ) , we , therefore , propose a strategy for learning the optimal dataset-specific regularization cocktail by means of a modern hyperparameter optimization ( HPO ) method . To the best of our knowledge , there exists no study providing empirical evidence that a mixture of numerous regularizers outperforms individual regularizers ; this paper fills this gap . More precisely , the research hypothesis of this paper is that a properly mixed regularization cocktail outperforms every individual regularizer in it , in terms of accuracy under the same run-time budget , and that the best cocktail to use depends on the dataset . To validate this hypothesis , we executed a large-scale experimental study employing 40 diverse tabular datasets and 13 prominent regularizers with a thorough hyperparameter tuning for all regularizers . We focus on tabular datasets because , in contrast to large image datasets , a thorough hyper-parameter search procedure is feasible . Neural networks are high variance models for tabular datasets , therefore improved regularization schemes can provide a relatively higher generalization gain on tabular datasets compared to other data types . Thereby , we make the followings contributions : 1 . We demonstrate the empirical accuracy gains of regularization cocktails in a systematic manner via a large-scale experimental study on tabular datasets ; 2 . We challenge the status-quo practices of designing universal dataset-agnostic regularizers , by showing that an optimal regularization cocktail is highly dataset-dependent ; 3 . We demonstrate that regularization cocktails achieve state-of-the-art classification accuracy performance on tabular datasets and outperform Gradient-Boosted Decision Trees ( GBDT ) with a statistically-significant margin ; 4 . As an overarching contribution , this paper provides previously-lacking in-depth empirical evidence to better understand the importance of combining different mechanisms for regularization , one of the most fundamental concepts in machine learning . 2 RELATED WORK . Weight decay : The classical approaches of regularization focused on minimizing the norms of the parameter values , concretely either the L1 ( Tibshirani , 1996 ) , the L2 ( Tikhonov , 1943 ) , or a combination of L1 and L2 known as the Elastic Net ( Zou & Hastie , 2005 ) . A recent work fixes the malpractice of adding the decay penalty term before momentum-based adaptive learning rate steps ( e.g. , in common implementations of Adam ( Kingma & Ba , 2015 ) ) , by decoupling the regularization from the loss and applying it after the learning rate computation ( Loshchilov & Hutter , 2019 ) . Data Augmentation : A different treatment of the overfitting phenomenon relies on enriching the training dataset via instance augmentation . The literature on data augmentation is vast , especially for image data , ranging from basic image manipulations ( e.g. , geometric transformations , or mixing images ) up to parametric augmentation strategies such as adversarial and controller-based methods ( Shorten & Khoshgoftaar , 2019 ) . For example , Cut-Out ( Devries & Taylor , 2017 ) proposes to mask a subset of input features ( e.g. , pixel patches for images ) for ensuring that the predictions remain invariant to distortions in the input space . Along similar lines , Mix-Up ( Zhang et al. , 2018 ) generates new instances as a linear span of pairs of training examples , while Cut-Mix ( Yun et al. , 2019 ) suggests super-positions of instance pairs with mutually-exclusive pixel masks . A recent technique , called Aug-Mix ( Hendrycks et al. , 2020 ) , generates instances by sampling chains of augmentation operations . On the other hand , the direction of reinforcement learning ( RL ) for augmentation policies was elaborated by Auto-Augment ( Cubuk et al. , 2019 ) , followed by a technique that speeds up the training of the RL policy ( S.Lim et al. , 2019 ) . Last but not least , adversarial attack strategies ( e.g. , FGSM ( Goodfellow et al. , 2015 ) ) generate synthetic examples with minimal perturbations , which are employed in training robust models ( Madry et al. , 2018 ) . Model Averaging : Ensembled machine learning models have been shown to reduce variance and act as regularizers ( Polikar , 2012 ) . A popular ensemble neural network with shared weights among its base models is Drop-Out ( Srivastava et al. , 2014 ) , which was extended to a variational version with a Gaussian posterior of the model parameters ( Kingma et al. , 2015 ) . A follow-up work that is known as Mix-Out ( Lee et al. , 2020 ) extends Drop-Out by statistically fusing the parameters of two base models . Furthermore , ensembles can be created using models from the local optima discovered along a single convergence procedure ( Huang et al. , 2016 ) . Structural and Linearization : One strategy of regularizing deep learning models is to discover dedicated neural structures that generalize on particular tasks , such as image classification or Natural Language Processing ( NLP ) . In that context , ResNet adds skip connections across layers ( He et al. , 2016 ) , while the Inception model computes latent representations by aggregating diverse convolutional filter sizes ( Szegedy et al. , 2017 ) . The attention mechanism gave rise to the popular Transformer architecture in the realm of NLP ( Vaswani et al. , 2017 ) . Recently , EfficientNet is an architecture that easily scales deep convolutional neural networks by controlling only a few hyperparameters ( Tan & Q.Le , 2019 ) . Besides the aforementioned manually-designed architectures , the stream of Neural Architecture Search ( Elsken et al. , 2019 ) focuses on exploring neural connectivity graphs for finding the optimal architectures via reinforcement learning ( Zoph & Le , 2017 ) , black-box search ( Real et al. , 2019 ) or differentiable solvers ( Liu et al. , 2019 ) . A recent trend adds a dosage of linearization to deep models , where skip connections transfer embeddings from previous less non-linear layers ( He et al. , 2016 ; Huang et al. , 2017 ) . Along similar lines , the Shake-Shake regularization deploys skip connections in parallel convolutional blocks and aggregates the parallel representations through affine combinations ( Gastaldi , 2017 ) , while Shake-Drop extends this mechanism to a larger number of CNN architectures ( Yamada et al. , 2018 ) . Implicit : The last family of regularizers broadly encapsulates methods which do not directly propose novel regularization techniques but have an implicit regularization effect as a virtue of their ‘ modus operandi ’ ( Arora et al. , 2019 ) . For instance , Batch Normalization improves generalization by reducing the internal covariate shifts ( Ioffe & Szegedy , 2015 ) , while early stopping of the optimization procedure also yields a similar generalization effect ( Yao et al. , 2007 ) . On the other hand , stabilizing the convergence of the training routine is another implicit regularization , for instance by introducing learning rate scheduling schemes ( Loshchilov & Hutter , 2017 ) . The recent strategy of stochastic weight averaging relies on averaging parameter values from the local optima encountered along the sequence of optimization steps ( Izmailov et al. , 2018 ) , while another approach conducts updates in the direction of a few ‘ lookahead ’ steps ( Zhang et al. , 2019 ) . Positioning in the realm of AutoML : In contrast to the prior literature , we do not propose a new individual regularization method , but empirically identify the superiority of learning regularization cocktails among a set of existing regularizers from the aforementioned categories . We train datasetspecific cocktails as a hyperparameter optimization ( HPO ) task ( Feurer & Hutter , 2019 ) . In that regard , our work is positioned in the realm of AutoML and is a special case of a combined algorithm selection and hyperparameter optimization ( Thornton et al. , 2013 ) . We learn the regularization cocktails and optimize the joint hyperparameter configuration space by means of BOHB ( Falkner et al. , 2018b ) , which is a variation of Hyperband ( Li et al. , 2017 ) with model-based surrogates and is one of the current state of the art approaches for efficient HPO . 3 MIXING THE REGULARIZATION COCKTAIL . 3.1 THE RECIPE AS PROBLEM STATEMENT . A training set is composed of features X ( Train ) and targets y ( Train ) , while the test dataset is denoted by X ( Test ) , y ( Test ) . A parameterized function approximates the targets as ŷ = f ( X ; θ ) whereas the parameters θ are trained to minimize a differentiable loss function L as : θ∗ ∈ arg min θ L ( y ( Train ) , f ( X ( Train ) ; θ ) ) . ( 1 ) To generalize into minimizing L ( y ( Test ) , f ( X ( Test ) ; θ ) , the parameters of f are controlled with a regularization technique Ω that avoids overfitting to the peculiarities of the training data . With a slight abuse of notation we denote f ( X ; Ω ( θ ; λ ) ) to be the predictions of the model f whose parameters θ are optimized under the regime of the regularization method Ω ( · ; λ ) , where λ ∈ Λ represent the hyperparameters of Ω . The training data is further divided into two subsets as training and validation splits,1 the later denoted by X ( Val ) , y ( Val ) , such that λ can be tuned on the validation loss via the following hyperparameter optimization objective : λ∗ ∈ arg min λ∈Λ L ( y ( Val ) , f ( X ( Val ) ; Ω ( θ∗ ; λ ) ) ) , ( 2 ) s.t . θ∗ ∈ arg min θ L ( y ( Train ) , f ( X ( Train ) ; Ω ( θ ; λ ) ) . ( 3 ) 1For simplicity , we only discuss hold-out validation scheme here , but in principle any other validation scheme , such as cross validation and bootstrap sampling , would be possible . While the search for optimal hyperparameters λ is an active field of research in the realm of AutoML ( Hutter et al. , 2018 ) , still the choice of the regularizer Ω mostly remains an ad-hoc practice , where practitioners select few combinations among popular regularizers ( Dropout , L2 , Batch Normalization , etc. ) . In contrast to prior studies , we hypothesize that the optimal regularizer is a cocktail mixture of a large set of regularization methods , all being simultaneously applied with different strengths ( i.e. , dataset-specific hyperparameters ) . Given a set of K regularizers { ( Ω ( k ) ( · ; λ ( k ) ) } K k=1 : = { Ω ( 1 ) ( · ; λ ( 1 ) ) , . . . , Ω ( K ) ( · ; λ ( K ) ) } , each with its own hyperparameters λ ( k ) ∈ Λ ( k ) , ∀k ∈ { 1 , . . . , K } , the problem of finding the optimal cocktail of regularizers is : λ∗ ∈ arg min λ ( 1 ) ∈Λ ( k ) , ... , λ ( K ) ∈Λ ( k ) L ( y ( Val ) , f ( X ( Val ) ; { Ω ( k ) ( θ∗ , λ ( k ) ) } K k=1 ) ) ( 4 ) s.t . θ∗ ∈ arg min θ L ( y ( Train ) , f ( X ( Train ) ; { Ω ( k ) ( θ , λ ( k ) ) } K k=1 ) ) ( 5 ) The intuitive interpretation of Equations 4-5 is searching for the optimal hyperparameters λ ( i.e. , strengths ) of the cocktail ’ s regularizers using the validation set ( Equation 4 ) , given that the optimal prediction model parameters θ are trained under the regime of all the regularizers being applied jointly ( Equation 5 ) . We stress that the hyperparameters λ ( k ) include a conditional hyperparameter controlling whether the k-th regularizer is applied at all , or skipped . Therefore , the best cocktail might consist of combinations of a subset of regularizers .

Summary: This paper provides an empirical study of combining different regularizers. Fourteen regularizers including batch norm, weight decay, etc. are considered. The authors use BOHB (Falkner et al. 2018) to optimize for whether each regularizer is active, and additional regularizer-specific hyperparameters. Using 40 tabular datasets, they show mixtures nearly always outperform tuning a single regularizer, and that the benefits of regularization improves for smaller datasets.