iFlood: A Stable and Effective Regularizer

1 INTRODUCTION . Though overparameterized neural networks have achieved success on a wide range of tasks and applications , it is worth noting that their capacities are sufficient to memorize the entire training data ( Zhang et al. , 2017a ; Arpit et al. , 2017 ) , which often leads to an intolerable generalization gap , or in other words , overfitting . To prevent overfitting , many methods have been proposed to regularize how machine learning models fit the training data via introducing additional constraints to control the capacity in effect . For example , L1- or L2- regularizer ( i.e. , Weight Decay ) ( Hanson & Pratt , 1989 ) , Early Stopping ( Yao et al. , 2007 ) , Dropout ( Srivastava et al. , 2014 ) , Label Smoothing ( Szegedy et al. , 2016 ) , Confident Penalty ( Pereyra et al. , 2017 ) , etc ( Zhang et al. , 2017b ; Izmailov et al. , 2018 ; Zheng et al. , 2021 ; Foret et al. , 2021 ; Yang et al. , 2020 ) . Recently , a new method named Flooding ( Ishida et al. , 2020 ) is proposed , which controls the extent to which machine learning models fit the training data via directly encouraging the averaged training loss to stay at a given level rather than achieving ( near- ) zero loss . Formally , Flooding suggests the following loss function LFlooding to be minimized : LFlooding = ∣L − b∣ + b , ( 1 ) where L denotes the averaged training loss that is defined over a total of N training instances as L = 1 N ∑ N i=1Li , and b ≥ 0 is a hyper-parameter called “ the flood level ” to control the training loss . Designed in such a way , whenever L is below b , the gradients will be negated to increase it , so that L will stay around b and avoid being ( near- ) zero . Ishida et al . ( 2020 ) suggests to implement Flooding by minimizing LFlooding with mini-batch SGD , where L is estimated over a mini-batch of instances rather than the full data . Although they have pointed out that the objective being optimized by SGD is an upper bound of its desired one , we notice that this gap increases as the batch size decreases , which makes the discrepancy between the objective and implementation of Flooding more serious . By investigating how machine learning models learned with Flooding behave on the training instances and generalize on the testing ones , we uncover the instability issue of Flooding , where it can lead to different solutions , and the solutions are inconsistent in their generalization abilities and their behaviors over individuals . Further , we point out that the reason for the instability issue is that Flooding can only guarantee “ global convergence ” — Flooding encourages the averaged training loss to be sufficiently close to b , while having no requirement on the individual losses . Both the discrepancy and instability issues can be attributed to the design of its loss function , which motivates us to propose a new regularizer in this paper , called individual Flood ( denoted as iFlood ) . The proposed regularizer iFlood defines a loss function LiFlood as : LiFlood = 1 N N ∑ i=1 ( ∣Li − b∣ + b ) , ( 2 ) where b ≥ 0 is the flood level , N is the size of training sample and Li is the loss function defined upon the i-th instance . When b = 0 , L , LFlooding , and LiFlood are equivalent . With b > 0 , iFlood encourages the model to better fit the under-fitted instances while suppressing the confidence of over-fitted ones . Although the modification is simple , the proposed new regularizer has the following merits : ( 1 ) The design of the loss function of iFlood ensures that it can be optimized by SGD without discrepancy between its objective and implementation . ( 2 ) Compared with Flooding , the models learned with iFlood can achieve “ local convergence ” , that is , the individual losses ( i.e. , Lis ) rather than the averaged loss ( i.e. , L ) are encouraged to be sufficiently close to the specified level b , which ensures that the learned models behave consistently over individual instances and produce stable generalization performance . ( 3 ) Meanwhile , we theoretically show that the design of iFlood can be intrinsically connected with removing the noise or bias in the training data , making iFlood suitable for a variety of applications to improve the generalization abilities . ( 4 ) Moreover , we theoretically compare iFlood with some related works ( Szegedy et al. , 2016 ; Pereyra et al. , 2017 ) , showing that iFlood discounts the over-confident predictions with less inductive bias . We conduct extensive experiments on both image classification and language understanding tasks to compare the performance improvements gained by different regularizers , demonstrating the effectiveness of the proposed iFlood . Further , we evaluate the stability1 of iFlood from several measurements , such as total variation distance and gradient norm . All the experimental results show that , with the “ local convergence ” suggested by iFlood , the learned models stably converge to solutions with better generalization ability . 2 PRELIMINARY . For ease of discussion , we introduce some notations at first . Without loss of generality , we consider a typical classification problem : Given a training dataset D = { ( xi , yi ) ∣x ∈ X , y ∈ Y , i = 1 , . . . , N } where X stands for the instance domain , Y stands for the set of labels , and each instance ( xi , yi ) is independently drawn from an underlying joint distribution Pr ( X , Y ) , we aim to learn a function f ∶ X → ∆ ( Y ) ( i.e. , a mapping from the instance domain to the space of probability distributions over the labels ) to minimize the generalization error E ( x , y ) ∼Pr ( X , Y ) [ 1y≠argmax y′ f ( x ) y′ ] , where f ( x ) y′ denotes the probability of taking the class y′ . The function f is often learned by minimizing certain loss function l ( y , f ( x ) ) ( e.g. , Cross-Entropy loss ) . For simplicity , we denote the loss over the i-th training instance as Li and that over the whole training dataset as L = 1 N ∑ N i=1Li . Note that , besides the classification problem , the regularization methods discussed in this paper are applicable to other machine learning tasks , e.g. , regression . 3 INDIVIDUAL FLOOD ( IFLOOD ) . In this section , we first compare iFlood with Flooding from various aspects to demonstrate the advantages of the proposed new regularizer iFlood . Then we provide theoretical analysis about the effect of iFlood for removing the noise or bias in the training data , and connect iFlood with existing regularization methods , such as Label Smoothing ( Szegedy et al. , 2016 ) and Confident Penalty ( Pereyra et al. , 2017 ) . 1In this paper , “ stability ” has a different meaning from that used in learning theory ( Mohri et al. , 2018 ) . 3.1 LOCAL CONVERGENCE . Let us revisit how Flooding works first . By applying Flooding , once the original averaged loss L has approached the flood level , it goes below and above b repeatedly until its convergence . Such process is called “ flooding ” in Ishida et al . ( 2020 ) . The loss function of Flooding encourages the averaged loss L to approach to b , thus Flooding pursues the “ global convergence ” ( i.e. , ∣L − b∣ ≈ 0 ) . To learn the model parameters θ by minimizing LFlooding defined in Eq . ( 1 ) , the gradient of LFlooding w.r.t . θ is in the same direction as that of L when L ≥ b , and in the opposite direction when L < b . To be specific , ∇θLFlooding = ⎧⎪⎪⎪⎪⎪⎪⎪ ⎨ ⎪⎪⎪⎪⎪⎪⎪⎩ ∇θL = 1 N N ∑ i=1 ∇θLi , if L ≥ b , −∇θL = 1 N N ∑ i=1 ( −1 ) ⋅ ∇θLi , if L < b . ( 3 ) From Eq . ( 3 ) , we know that whether the original gradient of an instance ( i.e. , ∇θLi ) should be negated is determined by the fact that whether the averaged loss L is below or above the flood level b . However , in most practical cases and meanwhile the experiments of Flooding , deep neural networks are trained with mini-batch SGD where L is estimated from a sampled mini-batch of instances at each step . Thus , whether the gradients should be negated is determined by the averaged training loss estimated over a mini-batch of instances rather than the full data . In this way , the objective being actually optimized by mini-batch SGD is an upper bound of LFlooding , as pointed out by Ishida et al . ( 2020 ) . This leads to a discrepancy between the loss function of Flooding and its implementation , which introduces randomness to the choice of whether to negate the gradients and makes the flooding process become random . In this study , we further notice that , due to the property of absolute operator , such discrepancy increases along with the decreasing of batch size . For iFlood , according to Eq . ( 2 ) , in a training dataset consisting of N instances , the gradient of LiFlood w.r.t . the model parameters θ can be given as ∇θLiFlood = 1N ∑ N i=1∇θ ∣Li − b∣ . Therefore , for each individual , it is guaranteed to contribute 1 N ∇θLi to the aggregated gradient , when Li ≥ b ; or to contribute the negation of that , when Li < b . By applying iFlood , the model parameters are encouraged to walk along the contour of L = b like what Flooding does , but , in contrast to Flooding , this is achieved by demanding each individual loss Li , ∀i ∈ { 1 , . . . , N } to be close to b . Intuitively speaking , iFlood encourages the model to better fit the under-fitted instances while suppressing the confidence of over-fitted ones , according to the given flood level b . In a word , iFlood pursues the “ local convergence ” ( i.e. , ∣Li − b∣ ≈ 0 , ∀i ∈ { 1 , . . . , N } ) . We will analyze the advantage of “ local convergence ” over “ global convergence ” from the aspect of stability in Section 3.2 . For now , the additive property of Eq . ( 2 ) ensures the gradients calculated in each SGD an unbiased estimation , which seemingly eliminates the discrepancy between the design and implementation . Note that when optimized by SGD with batch size of 1 , Flooding coincides with iFlood . However , it is unreasonable to set batch size to 1 for training machine learning models in most cases , which is empirically confirmed in Appendix C. For practical batch sizes , the designs of the loss function of iFlood and Flooding are different , as aforementioned , and we will empirically demonstrate those differences in Section 4 .

Summary: This paper extends the idea of Flooding to a sample-specific level and call it iFlooding. The extension is intuitively important and the authors also offered several analytical discussions to show its importance beyond the intuition. The empirical results are fairly relevant and strong.