Implicit Rugosity Regularization via Data Augmentation

1 INTRODUCTION . Deep ( neural ) networks are being profitably applied in a large and growing number of areas , from signal processing to computer vision and artificial intelligence . The expressive power of these networks has been demonstrated both in theory and practice ( Cybenko , 1989 ; Barron , 1994 ; Telgarsky , 2015 ; Yarotsky , 2017 ; Hanin & Sellke , 2017 ; Daubechies et al. , 2019 ) . In fact , it has been shown that deep networks can even perfectly fit pure noise ( Zhang et al. , 2016 ) . Surprisingly , highly overparameterized deep networks—where the number of network parameters exceeds the number of training data points—can be trained for a range of different classification and regression tasks and perform extremely well on unobserved data . Understanding why these networks generalize so well has been the subject of great interest in recent years . But , to date , classical approaches to bound the generalization error have failed to provide much insight into deep networks . The ability of overparameterized deep networks to overfit noise while generalizing well suggests the existence of some kind of regularization in the learning process . Sometimes the regularization is explicit , as in the case of Tikhonov or ` 1 regularization ( Krogh & Hertz , 1992 ; Bishop , 1995 ; Han et al. , 2015 ; Bartlett et al. , 2017 ) . By contrast , we are becoming increasingly aware that various heuristic techniques—such as dropout ( Baldi & Sadowski , 2013 ; Wager et al. , 2013 ; Srivastava et al. , 2014 ) , batch normalization ( Ioffe & Szegedy , 2015 ; Salimans & Kingma , 2016 ; Bjorck et al. , 2018 ) , and even stochastic gradient descent ( Hardt et al. , 2015 ; Bousquet & Elisseeff , 2002 ; Yao et al. , 2007 ) —that are used in deep neural networks provide implicit regularization , having effects that are not as transparent . In this work , we study another such heuristic—data augmentation ( Bengio et al. , 2011 ; Dao et al. , 2018 ; Rajput et al. , 2019 ; Chen et al. , 2019 ) —and show that it has a regularizing effect reducing what we call the rugosity , or “ roughness ” , of the function learned by the deep network . Our rugosity measure quantifies how far the mapping f is from a locally linear mapping on the data . Building on the concept of Hessian eigenmaps introduced by Donoho & Grimes ( 2003 ) , we evaluate this deviation from a locally linear mapping using the tangent Hessian , assuming the data points x ∈ RD lie on a low-dimensional manifoldM with dimension d D. Intuitively , we would expect functions that generalize well to be smooth—that is , to not have a high amount of rugosity . For example , in classification problems , if we were to partition the feature space by class , then we might expect the set corrresponding to any given class to consist of the union of a few contiguous regions with fairly smooth boundaries . A classification function with high rugosity , particularly near the true class boundaries , will likely have an unstable decision boundary , especially for noisy observations off of the data manifold . See Section 5 for a further discussion of the caveats of using rugosity alone as a rule for learning predictors . A key part of our analysis leverages a new perspective on deep networks provided by Balestriero & Baraniuk ( 2018a ; b ) . Let f ( x ) represent the mapping from the input to output of a deep network constructed using continuous , piecewise affine activations ( e.g. , ReLU , leaky ReLU , absolute value ) . Such a network partitions the input space RD based on the activation pattern of the network ’ s units ( neurons ) . We call these partitions Qj ⊂ RD , the vector quantization ( VQ ) regions of the network , and for x ∈ RD we let Q ( x ) denote the VQ region to which x belongs . Inside one VQ region , f is simply an affine mapping . Consequently , f can be written as a continuous , piecewise affine operator f ( x ) = AQ ( x ) x+ bQ ( x ) = A [ x ] x+ b [ x ] . ( 1 ) For the sake of brevity , we will use the simplified notation f ( x ) = A [ x ] x+ b [ x ] in the sequel . This framework enables us to cleanly reason about our extension of the rugosity measure to piecewise affine functions using finite differences , enabling us to make the connection between data augmentation the measure of rugosity via the A matrices . Contributions . We summarize our contributions as follows : [ C1 ] We develop a new rugosity measure of function complexity that measures how much a function deviates from locally linear , with a nontrivial extension of this measure to piecewise affine functions based on finite differences . [ C2 ] We formally connect the popular heuristic of data augmentation to an explicit rugosity regularization . [ C3 ] We verify experimentally that data augmentation reduces rugosity . [ C4 ] We experimentally assess the effect of explicit rugosity regularization in training deep networks . Related work . Recently , Savarese et al . ( 2019 ) have shown that , for one-dimensional , one-hiddenlayer ReLU networks , an ` 2 penalty on the weights ( Tikhonov regularization ) is equivalent to a penalty on the integral of the absolute value of the second derivative of the network output . Moreover , several recent works ( Belkin et al. , 2018 ; 2019 ; Hastie et al. , 2019 ) have shown that in some linear and nonlinear inference problems , properly regularized overparameterized models can generalize well . This again indicates the importance of complexity regularization for understanding generalization in deep networks . Rifai et al . ( 2011 ) proposed an auto-encoder whose regularization penalizes the change in the encoder Jacobian . Also , Dao et al . ( 2018 ) studied the effects of data augmentation by modeling data augmentation as a kernel . 2 DATA AUGMENTATION . Data augmentation ( Wong et al. , 2016 ; Perez & Wang , 2017 ) is an oft-used yet poorly understood heuristic applied in learning the parameters of deep networks . The data augmentation procedure augments the set of n training data points { ( xi , yi ) } ni=1 to ( m + 1 ) n training data points { { ( xi + uij , yi ) } mj=0 } n i=1 by applying transformations to the training data xi such that they continue to lie on the data manifoldM . Example transformations applied to images include translation , rotation , color changes , etc.1 In such cases , uij is the vector difference xj − xi , where xj is the translated/rotated image , and ui0 = 0 . Consider training a deep network with continuous , piecewise affine activations given the original training dataset { ( xi , yi ) } ni=1 by minimizing the loss L , n∑ i=1 ` i = n∑ i=1 ` ( f ( xi ) , yi ) , ( 2 ) where ` ( · , · ) is any convex loss function . After data augmentation , the loss can be written as Laug = 1 m+ 1 n∑ i=1 ` augi , ` aug i , ` ( f ( xi ) , yi ) + m∑ j=1 ` ( f ( xi + uij ) , yi ) . ( 3 ) In the following result , we obtain an upper bound on the augmented loss , which consists of the sum of the non-augmented loss plus a few terms pertaining to the local similarity of the output function around the training data . 1Common transformations also include flipping images , which is not a continuous operation ; however , our analysis here pertains primarily to continuous transformations , for which the vector differences uij are small . Theorem 1 . Consider a deep network with continuous , piecewise affine activations and thus prediction function f as in ( 1 ) . Assume that ‖xi‖2 ≤ R , and that f has Lipschitz constant K1 ; that is , ‖A [ x ] ‖2 ≤ K1 , for all x . Further , assume that the loss function ` has Lipschitz constant K2 . Then , for ‖uij‖2 ≤ ε , Laug in ( 3 ) can be upper bounded by Laug ≤ L+ RK2 m+ 1 n∑ i=1 m∑ j=1 ‖A [ xi + uij ] −A [ xi ] ‖2 + K2 m+ 1 n∑ i=1 m∑ j=1 |b [ xi + uij ] − b [ xi ] |+ K1K2mnε m+ 1 + o ( ε ) . ( 4 ) We focus our attention on the second term of the right-hand side of the above inequality . Denote this term by Ĉ ( f ) , n∑ i=1 m∑ j=1 ‖A [ xi + uij ] −A [ xi ] ‖2 . ( 5 ) In the sequel , we show that this term is effectively a Monte Carlo approximation to a Hessian-based measure of rugosity evaluated at the training data points . 3 A HESSIAN-BASED RUGOSITY MEASURE . To arrive at Ĉ ( f ) and link data augmentation to an explicit regularization penalty , we first define our new Hessian-based rugosity measure for deep networks with continuous activations . Then , we extend this definition to networks with piecewise-linear activations , and finally we describe its Monte Carlo approximation . We note that while deep networks are our focus , this measure of rugosity is more broadly applicable . 3.1 NETWORK WITH SMOOTH ACTIVATIONS . Let f : RD → R be the prediction function of a deep network whose nonlinear activation functions are smooth functions ( e.g. , sigmoid ) . For regression , we can take f as the mapping from the input to the output of the network . For classification , we can take f as the mapping from the input to one of the inputs of the final softmax operation . We assume that the training data points { xi } ni=1 lie close to a d-dimensional smooth manifoldM ⊆ RD . This assumption has been studied in an extensive literature on unsupervised learning ( Tenenbaum et al. , 2000 ; Belkin & Niyogi , 2003 ; Donoho & Grimes , 2003 ) , and holds at least approximately for many practical datasets , including real world images . For p ≥ 1 , we define the following measure of rugosity : Cp ( f ) , ( ∫ M ∥∥∥∇2 ( tan ) f ( x ) ∥∥∥p F dx ) 1/p ( 6 ) where ∇2 ( tan ) f ( x ) is the Hessian of f at x in the coordinates of d-dimensional affine space tangent to manifold at x ∈ M. This is an extension of the quadratic form from the Hessian eigenmaps approach of Donoho & Grimes ( 2003 ) , which is equal to C22 ( f ) . From Donoho & Grimes ( 2003 ) , we know that Cp ( f ) measures the average “ curviness ” of f over the manifoldM and that Cp ( f ) is zero if and only if f is an affine function onM . In the simplistic case of one-dimensional data and one-hidden-layer networks , Savarese et al . ( 2019 ) have related C1 ( f ) to the sum of the squared Frobenius norms of the weight matrices .

Summary: This paper shows (theorem 1) that data augmentation (DA) induces a reduction of rugsity on the loss function associated to the model. Here rugosity is defined as a measure of the curvature (2nd order) of the function. However, the two concepts seems to be different because the authors empirically show that directly reducing the rugosity of a network does not improve generalization (in contrast to DA).