iFlood: A Stable and Effective Regularizer

1 INTRODUCTION . Though overparameterized neural networks have achieved success on a wide range of tasks and applications , it is worth noting that their capacities are sufficient to memorize the entire training data ( Zhang et al. , 2017a ; Arpit et al. , 2017 ) , which often leads to an intolerable generalization gap , or in other words , overfitting . To prevent overfitting , many methods have been proposed to regularize how machine learning models fit the training data via introducing additional constraints to control the capacity in effect . For example , L1- or L2- regularizer ( i.e. , Weight Decay ) ( Hanson & Pratt , 1989 ) , Early Stopping ( Yao et al. , 2007 ) , Dropout ( Srivastava et al. , 2014 ) , Label Smoothing ( Szegedy et al. , 2016 ) , Confident Penalty ( Pereyra et al. , 2017 ) , etc ( Zhang et al. , 2017b ; Izmailov et al. , 2018 ; Zheng et al. , 2021 ; Foret et al. , 2021 ; Yang et al. , 2020 ) . Recently , a new method named Flooding ( Ishida et al. , 2020 ) is proposed , which controls the extent to which machine learning models fit the training data via directly encouraging the averaged training loss to stay at a given level rather than achieving ( near- ) zero loss . Formally , Flooding suggests the following loss function LFlooding to be minimized : LFlooding = ∣L − b∣ + b , ( 1 ) where L denotes the averaged training loss that is defined over a total of N training instances as L = 1 N ∑ N i=1Li , and b ≥ 0 is a hyper-parameter called “ the flood level ” to control the training loss . Designed in such a way , whenever L is below b , the gradients will be negated to increase it , so that L will stay around b and avoid being ( near- ) zero . Ishida et al . ( 2020 ) suggests to implement Flooding by minimizing LFlooding with mini-batch SGD , where L is estimated over a mini-batch of instances rather than the full data . Although they have pointed out that the objective being optimized by SGD is an upper bound of its desired one , we notice that this gap increases as the batch size decreases , which makes the discrepancy between the objective and implementation of Flooding more serious . By investigating how machine learning models learned with Flooding behave on the training instances and generalize on the testing ones , we uncover the instability issue of Flooding , where it can lead to different solutions , and the solutions are inconsistent in their generalization abilities and their behaviors over individuals . Further , we point out that the reason for the instability issue is that Flooding can only guarantee “ global convergence ” — Flooding encourages the averaged training loss to be sufficiently close to b , while having no requirement on the individual losses . Both the discrepancy and instability issues can be attributed to the design of its loss function , which motivates us to propose a new regularizer in this paper , called individual Flood ( denoted as iFlood ) . The proposed regularizer iFlood defines a loss function LiFlood as : LiFlood = 1 N N ∑ i=1 ( ∣Li − b∣ + b ) , ( 2 ) where b ≥ 0 is the flood level , N is the size of training sample and Li is the loss function defined upon the i-th instance . When b = 0 , L , LFlooding , and LiFlood are equivalent . With b > 0 , iFlood encourages the model to better fit the under-fitted instances while suppressing the confidence of over-fitted ones . Although the modification is simple , the proposed new regularizer has the following merits : ( 1 ) The design of the loss function of iFlood ensures that it can be optimized by SGD without discrepancy between its objective and implementation . ( 2 ) Compared with Flooding , the models learned with iFlood can achieve “ local convergence ” , that is , the individual losses ( i.e. , Lis ) rather than the averaged loss ( i.e. , L ) are encouraged to be sufficiently close to the specified level b , which ensures that the learned models behave consistently over individual instances and produce stable generalization performance . ( 3 ) Meanwhile , we theoretically show that the design of iFlood can be intrinsically connected with removing the noise or bias in the training data , making iFlood suitable for a variety of applications to improve the generalization abilities . ( 4 ) Moreover , we theoretically compare iFlood with some related works ( Szegedy et al. , 2016 ; Pereyra et al. , 2017 ) , showing that iFlood discounts the over-confident predictions with less inductive bias . We conduct extensive experiments on both image classification and language understanding tasks to compare the performance improvements gained by different regularizers , demonstrating the effectiveness of the proposed iFlood . Further , we evaluate the stability1 of iFlood from several measurements , such as total variation distance and gradient norm . All the experimental results show that , with the “ local convergence ” suggested by iFlood , the learned models stably converge to solutions with better generalization ability . 2 PRELIMINARY . For ease of discussion , we introduce some notations at first . Without loss of generality , we consider a typical classification problem : Given a training dataset D = { ( xi , yi ) ∣x ∈ X , y ∈ Y , i = 1 , . . . , N } where X stands for the instance domain , Y stands for the set of labels , and each instance ( xi , yi ) is independently drawn from an underlying joint distribution Pr ( X , Y ) , we aim to learn a function f ∶ X → ∆ ( Y ) ( i.e. , a mapping from the instance domain to the space of probability distributions over the labels ) to minimize the generalization error E ( x , y ) ∼Pr ( X , Y ) [ 1y≠argmax y′ f ( x ) y′ ] , where f ( x ) y′ denotes the probability of taking the class y′ . The function f is often learned by minimizing certain loss function l ( y , f ( x ) ) ( e.g. , Cross-Entropy loss ) . For simplicity , we denote the loss over the i-th training instance as Li and that over the whole training dataset as L = 1 N ∑ N i=1Li . Note that , besides the classification problem , the regularization methods discussed in this paper are applicable to other machine learning tasks , e.g. , regression . 3 INDIVIDUAL FLOOD ( IFLOOD ) . In this section , we first compare iFlood with Flooding from various aspects to demonstrate the advantages of the proposed new regularizer iFlood . Then we provide theoretical analysis about the effect of iFlood for removing the noise or bias in the training data , and connect iFlood with existing regularization methods , such as Label Smoothing ( Szegedy et al. , 2016 ) and Confident Penalty ( Pereyra et al. , 2017 ) . 1In this paper , “ stability ” has a different meaning from that used in learning theory ( Mohri et al. , 2018 ) . 3.1 LOCAL CONVERGENCE . Let us revisit how Flooding works first . By applying Flooding , once the original averaged loss L has approached the flood level , it goes below and above b repeatedly until its convergence . Such process is called “ flooding ” in Ishida et al . ( 2020 ) . The loss function of Flooding encourages the averaged loss L to approach to b , thus Flooding pursues the “ global convergence ” ( i.e. , ∣L − b∣ ≈ 0 ) . To learn the model parameters θ by minimizing LFlooding defined in Eq . ( 1 ) , the gradient of LFlooding w.r.t . θ is in the same direction as that of L when L ≥ b , and in the opposite direction when L < b . To be specific , ∇θLFlooding = ⎧⎪⎪⎪⎪⎪⎪⎪ ⎨ ⎪⎪⎪⎪⎪⎪⎪⎩ ∇θL = 1 N N ∑ i=1 ∇θLi , if L ≥ b , −∇θL = 1 N N ∑ i=1 ( −1 ) ⋅ ∇θLi , if L < b . ( 3 ) From Eq . ( 3 ) , we know that whether the original gradient of an instance ( i.e. , ∇θLi ) should be negated is determined by the fact that whether the averaged loss L is below or above the flood level b . However , in most practical cases and meanwhile the experiments of Flooding , deep neural networks are trained with mini-batch SGD where L is estimated from a sampled mini-batch of instances at each step . Thus , whether the gradients should be negated is determined by the averaged training loss estimated over a mini-batch of instances rather than the full data . In this way , the objective being actually optimized by mini-batch SGD is an upper bound of LFlooding , as pointed out by Ishida et al . ( 2020 ) . This leads to a discrepancy between the loss function of Flooding and its implementation , which introduces randomness to the choice of whether to negate the gradients and makes the flooding process become random . In this study , we further notice that , due to the property of absolute operator , such discrepancy increases along with the decreasing of batch size . For iFlood , according to Eq . ( 2 ) , in a training dataset consisting of N instances , the gradient of LiFlood w.r.t . the model parameters θ can be given as ∇θLiFlood = 1N ∑ N i=1∇θ ∣Li − b∣ . Therefore , for each individual , it is guaranteed to contribute 1 N ∇θLi to the aggregated gradient , when Li ≥ b ; or to contribute the negation of that , when Li < b . By applying iFlood , the model parameters are encouraged to walk along the contour of L = b like what Flooding does , but , in contrast to Flooding , this is achieved by demanding each individual loss Li , ∀i ∈ { 1 , . . . , N } to be close to b . Intuitively speaking , iFlood encourages the model to better fit the under-fitted instances while suppressing the confidence of over-fitted ones , according to the given flood level b . In a word , iFlood pursues the “ local convergence ” ( i.e. , ∣Li − b∣ ≈ 0 , ∀i ∈ { 1 , . . . , N } ) . We will analyze the advantage of “ local convergence ” over “ global convergence ” from the aspect of stability in Section 3.2 . For now , the additive property of Eq . ( 2 ) ensures the gradients calculated in each SGD an unbiased estimation , which seemingly eliminates the discrepancy between the design and implementation . Note that when optimized by SGD with batch size of 1 , Flooding coincides with iFlood . However , it is unreasonable to set batch size to 1 for training machine learning models in most cases , which is empirically confirmed in Appendix C. For practical batch sizes , the designs of the loss function of iFlood and Flooding are different , as aforementioned , and we will empirically demonstrate those differences in Section 4 .

Summary: This paper proposed a stable and effective regularizer to prevent overfitting. Specifically, this paper proposes individual Flood. Different from Flood which constrains training loss on mini-batch level, iFlood gives instance-level constraints on training loss. This paper also theoretically shows that the design of iFlood can be intrinsically connected with removing the noise or bias in training data.