On Regularization and Robustness of Deep Neural Networks

1 Introduction . Learning predictive models for complex tasks often requires large amounts of annotated data . Interestingly , successful models such as convolutional neural networks are also hugedimensional and typically involve more parameters than training samples . Such a setting is challenging : besides the fact that training with small datasets is difficult , these models also lack robustness to small adversarial perturbations ( Szegedy et al. , 2013 ; Biggio & Roli , 2018 ) . In the context of perceptual tasks such as vision ( Szegedy et al. , 2013 ) or speech recognition ( Carlini & Wagner , 2018 ) , these perturbed examples are often perceived identically to the original ones by a human , but can lead to arbitrarily different model predictions . In this paper , we present a unified perspective on regularization and robustness , by viewing convolutional neural networks as elements of a particular reproducing kernel Hilbert space ( RKHS ) following the work of Bietti & Mairal ( 2018 ) on deep convolutional kernels . For such kernels , the RKHS contains indeed deep convolutional networks similar to generic ones—up to smooth approximations of rectified linear units . Such a point of view is interesting to adopt for deep networks since it provides a natural regularization function , the RKHS norm , which allows us to control the variations of the predictive model according to the geometry induced by the kernel . Besides , the norm also acts as a Lipschitz constant , which provides a direct control on the stability to adversarial perturbations . In contrast to traditional kernel methods , the RKHS norm can not be explicitly computed in our setup . Yet , these norms admit numerous approximations—lower bounds and upper bounds—which lead to new strategies for regularization based on penalties , constraints , or combinations thereof . Additionally , our framework provides an interpretation of some existing regularization approaches as controlling either upper or lower bounds on the RKHS norm , such as using penalties or constraints on the spectral norm of the filters ( Yoshida & Miyato , 2017 ) , various forms of robust optimization ( Madry et al. , 2018 ) , double backpropagation ( Drucker & Le Cun , 1991 ) , and tangent propagation ( Simard et al. , 1998 ) . Moreover , regularization and robustness are tightly linked in our kernel framework . In particular , we study the connection between adversarial training under ` 2 perturbations and penalizing with the RKHS norm , in a similar vein to Xu et al . ( 2009b ) ; from a statistical point of view , we extend margin-based generalization bounds in the spirit of Bartlett et al . ( 2017 ) ; Boucheron et al . ( 2005 ) to the setting of adversarially robust generalization ( see Schmidt et al. , 2018 ) . Following these observations , we provide an empirical evaluation of the different regularization strategies obtained , by considering learning tasks with small datasets or with adversarial perturbations . Importantly , we find that robust optimization approaches often lead to a poor control of the RKHS norm , in favor of local stability and data fit , and that approaches based on spectral norms can be locally unstable . In contrast , our new lower bound penalties , as well as combined approaches based on both upper and lower bounds , enable a tighter control of the RKHS norm , yielding empirical benefits in various regimes and better guarantees . We also provide a new perspective on recent successful approaches to training generative adversarial networks as optimizing a kernel two-sample test ( see , e.g. , Gretton et al. , 2012 ) . Related work . The construction of hierarchical kernels and the study of neural networks in the corresponding RKHS was studied by Mairal ( 2016 ) ; Zhang et al . ( 2016 ; 2017 ) ; Bietti & Mairal ( 2018 ) . Our study of the relationship between robustness and regularization follows from Xu et al . ( 2009a ; b ) , where the main focus is on linear models with quadratic or hinge losses . Some of the regularization strategies we obtain from our kernel perspective are closely related to previous approaches to adversarial robustness ( Cisse et al. , 2017 ; Madry et al. , 2018 ; Simon-Gabriel et al. , 2018 ; Roth et al. , 2018 ) , to improving generalization ( Drucker & Le Cun , 1991 ; Miyato et al. , 2018b ; Sedghi et al. , 2018 ; Simard et al. , 1998 ; Yoshida & Miyato , 2017 ) , and stable training of generative adversarial networks ( Roth et al. , 2017 ; Gulrajani et al. , 2017 ; Arbel et al. , 2018 ; Miyato et al. , 2018a ) . The notion of adversarial generalization was considered by Schmidt et al . ( 2018 ) , who provide lower bounds on a specific example of data distribution . Sinha et al . ( 2018 ) provide generalization guarantees in the different setting of distributional robustness ; compared to our bound , they consider expected loss instead of classification error , and their bounds do not highlight the dependence on the model complexity . 2 Regularization strategies for deep neural networks . In this section , we recall the kernel perspective on deep networks introduced by Bietti & Mairal ( 2018 ) , and present upper and lower bounds on the RKHS norm of a given model , leading to various regularization strategies . For simplicity , we first consider real-valued networks and binary classification , before discussing multi-class extensions . 2.1 Relation between deep neural networks and RKHSs . Kernel methods consist of mapping data living in a set X to a RKHS H associated to a positive definite kernel K through a mapping function Φ : X → H , and then learning simple machine learning models in H. Specifically , when considering a real-valued regression or binary classification problem , classical kernel methods find a prediction function f : X → R living in the RKHS which can be written in linear form , i.e. , such that f ( x ) = 〈f , Φ ( x ) 〉H for all x in X . While explicit mapping to a possibly infinite-dimensional space is of course only an abstract mathematical operation , learning f can be done implicitly by computing kernel evaluations and typically by using convex programming ( Schölkopf & Smola , 2001 ) . Moreover , the RKHS norm ‖f‖H acts as a natural regularization function , which controls the variations of model predictions according to the geometry induced by Φ : |f ( x ) − f ( x′ ) | ≤ ‖f‖H · ‖Φ ( x ) − Φ ( x′ ) ‖H . ( 1 ) Unfortunately , traditional kernel methods become difficult to use when the datasets are large or when evaluating the kernel is intractable . Here , we propose a different approach that considers explicit parameterized representations of functions contained in the RKHS , given by generic convolutional neural networks , and leverage properties of the RKHS and the kernel mapping in order to regularize when learning the network parameters . Consider indeed a real-valued deep convolutional network f : X → R , where X is simply Rd , with rectified linear unit ( ReLU ) activations and no bias units . By constructing an appropriate multi-layer hierarchical kernel , Bietti & Mairal ( 2018 ) show that the corresponding RKHS H contains a convolutional network with the same architecture and parameters as f , but with activations that are smooth approximations of ReLU . Although the model predictions might not be strictly equal , we will abuse notation and denote this approximation with smooth ReLU by f as well , with the hope that the regularization procedures derived from the RKHS model will be effective in practice on the original CNN f . Besides , Bietti & Mairal ( 2018 ) show that the mapping Φ ( · ) is non-expansive : ‖Φ ( x ) − Φ ( x′ ) ‖H ≤ ‖x− x′‖2 , ( 2 ) so that controlling ‖f‖H provides some robustness to additive ` 2-perturbations , by ( 1 ) . Additionally , with appropriate pooling operations , Bietti & Mairal ( 2018 ) show that the kernel mapping is also stable to deformations , meaning that the RKHS norm also controls robustness to translations and other transformations including scaling and rotations ( Engstrom et al. , 2017 ) , which can be seen as deformations when they are small . In contrast to standard kernel methods , where the RKHS norm is typically available in closed form , this norm is difficult to compute in our setup , and requires approximations . The following sections present upper and lower bounds on ‖f‖H , with linear convolutional operations denoted by Wk for k = 1 , . . . , L , where L is the number of layers . Defining θ : = { Wk : k = 1 , . . . , L } , we then leverage these bounds to approximately solve the following penalized or constrained optimization problems on a training set ( xi , yi ) , i = 1 , . . . , n : min θ 1 n n∑ i=1 ` ( yi , fθ ( xi ) ) + λ‖fθ‖2H or min θ : ‖fθ‖H≤C 1 n n∑ i=1 ` ( yi , fθ ( xi ) ) . ( 3 ) We also note that while the construction of Bietti & Mairal ( 2018 ) considers VGG-like networks ( Simonyan & Zisserman , 2014 ) , the regularization algorithms we obtain in practice can be easily adapted to different architectures such as residual networks ( He et al. , 2016 ) . 2.2 Exploiting Lower bounds of the RKHS norm . In this section , we devise regularization algorithms by leveraging lower bounds on ‖f‖H , which are obtained by relying on the following variational characterization of Hilbert norms : ‖f‖H = sup ‖u‖H≤1 〈f , u〉H . ( 4 ) At first sight , this definition is not useful since the set U = { u ∈ H : ‖u‖H ≤ 1 } may be infinite-dimensional and the inner products 〈f , u〉H can not be computed in general . Thus , we devise tractable lower bound approximations by considering smaller sets Ū ⊂ U . Adversarial perturbation penalty . The non-expansiveness of Φ allows us to consider the subset Ū ⊂ U defined as Ū = { Φ ( x+ δ ) −Φ ( x ) : x ∈ X , ‖δ‖2 ≤ 1 } , leading to the bound ‖f‖H ≥ sup x∈X , ‖δ‖2≤1 f ( x+ δ ) − f ( x ) , ( 5 ) which is reminiscent of adversarial perturbations . Adding a regularization parameter > 0 in front of the norm then corresponds to different sizes of perturbations : ‖f‖H = sup ‖u‖H≤ 〈f , u〉H ≥ sup x∈X , ‖δ‖2≤ f ( x+ δ ) − f ( x ) . ( 6 ) Using this lower bound or its square as a penalty in the objective ( 3 ) when training a neural network can then provide a way to regularize . Optimizing over adversarial perturbations has been useful to obtain robust models ( e.g. , the PGD method of Madry et al. , 2018 ) , yet our approach differs in two important ways : ( i ) it involves a global maximization problem on the input space X , as opposed to only maximizing on perturbations near training data ; ( ii ) it involves a separate penalty term in the training objective , in contrast to PGD which encourages a perfect fit of the training data by considering perturbations on the loss term . We further discuss the links with the robust optimization problem solved by PGD below . In practice , optimizing over all x in X is difficult , and we can replace X with random subsets of examples ( such as mini-batches ) , which may be labeled or not , yielding further lower bounds on the penalty . In the context of a mini-batch stochastic gradient algorithm , one can obtain a subgradient of this penalty by first finding maximizers x̂ , δ̂ ( where x̂ is one of the examples in the mini-batch ) , and then simply computing gradients of fθ ( x̂+ δ̂ ) −fθ ( x̂ ) ( or its square ) w.r.t . θ using back-propagation . We compute the perturbations δ̂ for each example x by using a few steps of projected subgradient ascent with constant step-lengths , in a similar fashion to Madry et al . ( 2018 ) in the context of robust optimization . Relationship with robust optimization . In some contexts , our penalized approach is related to solving the robust optimization problem min θ 1 n n∑ i=1 sup ‖δ‖2≤ ` ( yi , fθ ( xi + δ ) ) , ( 7 ) which is commonly considered for training adversarially robust classifiers ( Kolter & Wong , 2017 ; Madry et al. , 2018 ; Raghunathan et al. , 2018 ) . In particular , Xu et al . ( 2009b ) show that the penalized and robust objectives are equivalent in the case of the hinge loss with linear predictors , when the data is non-separable . They also show the equivalence for kernel methods when considering the ( intractable ) full perturbation set U around each point in the RKHS Φ ( xi ) , that is , predictions 〈f , Φ ( xi ) + u〉H with u in U . Intuitively , when a training example ( xi , yi ) is misclassified , we are in the “ linear ” part of the hinge loss , so that sup ‖u‖H≤ ` ( yi , 〈f , Φ ( xi ) + u〉H ) = ` ( yi , 〈f , Φ ( xi ) 〉H ) + sup ‖u‖H≤ 〈f , u〉H = ` ( yi , f ( xi ) ) + ‖f‖H . For other losses such as the the logistic loss , a regularization effect is still present even for correctly classified examples , though it may be smaller since the loss has a reduced slope for such points , leading to a more adaptive regularization mechanism which may automatically reduce the amount of regularization when the data is easily separable . However , this approach might only encourage local stability , while the global quantity ‖f‖H may grow uncontrolled in order to better fit the data . Nevertheless , it is easy to show that the robust objective ( 7 ) lower bounds the penalized objective with penalty ‖f‖H . Gradient penalties . Taking Ū = { Φ ( x ) −Φ ( y ) ‖x−y‖2 : x , y ∈ X } , which is a subset of U by Eq . ( 2 ) —it turns out that this is the same set as above , since Φ is homogeneous ( Bietti & Mairal , 2018 ) and X = Rd—we obtain a lower bound based on the Lipschitz constant of f : ‖f‖H ≥ ‖f‖L : = sup x , y∈X f ( x ) − f ( y ) ‖x− y‖2 ≥ sup x∈X ‖∇f ( x ) ‖2 where the last inequality becomes an equality when X is convex , and the supremum is taken over points where f is differentiable . Although we are unaware of previous work using such a lower bound for a generic regularization penalty , we note that variants replacing the supremum over x by an expectation over data have been recently used to stabilize the training of generative adversarial networks ( Gulrajani et al. , 2017 ; Roth et al. , 2017 ) , and we provide insights in Section 3.2 on the benefits of RKHS regularization in such a setting . Related penalties have been considered in the context of robust optimization , for regularization or robustness , noting that a penalty based on the gradient of the loss can give a good approximation of ( 7 ) when is small ( Drucker & Le Cun , 1991 ; Lyu et al. , 2015 ; Roth et al. , 2018 ; Simon-Gabriel et al. , 2018 ) . This has the advantage of overcoming the difficulties of the maximization problem over δ by leveraging the closed-form expression of the gradient norm , but it may only provide a poor approximation when is large , and exhibits the same concerns as other robust optimization approaches in terms of poorly controlling ‖f‖H . Penalties based on deformation stability . We may also obtain new penalties by considering sets Ū = { Φ ( x̃ ) −Φ ( x ) : x ∈ X , x̃ is a small deformation of x } , where the amount of deformation is dictated by the stability bounds of Bietti & Mairal ( 2018 ) in order to ensure that Ū ⊂ U . In particular , such bounds depend on the maximum displacement and maximum Jacobian norm of the diffeomorphisms considered . These can be easily computed for various parameterized families of transformations , such as translations , scaling or rotations , leading to simple ways to control the regularization strength through the parameters of these transformations . One can also consider infinitesimal deformations from such parameterized transformations , and by replacing the supremum over X by an expectation over training data , we then obtain a gradient penalty that resembles the tangent propagation regularization strategy of Simard et al . ( 1998 ) . If instead we consider robust optimization formulation ( 7 ) , we obtain a form of data augmentation where transformations are optimized instead of sampled , as in Engstrom et al . ( 2017 ) . One advantage of these approaches based on lower bounds is that the obtained penalties are independent of the model parameterization , making them flexible enough to use with more complex architectures in practice . In addition , the connection with robust optimization can provide a useful mechanism for adaptive regularization . However , these lower bounds do not guarantee a control on the RKHS norm , and this is particularly true for robust optimization approaches , which may favor small training loss and local stability over global stability through ‖f‖H . Nevertheless , our experiments suggest that our new approaches based on separate penalties often do help in controlling upper bounds as well ( see Section 4 ) .

Summary: In this paper, the authors consider CNN models from the lens of kernel methods. They build upon past work that showed that such models can be seen to lie in appropriate RKHS, and derive upper and lower bounds for the kernel norm. These bounds can be used as regularizers that help train more robust neural networks, especially in the context of euclidean perturbations of the inputs, and training GANs. They show that the bounds can also be used to recover existing special cases such as spectral norm penalizations and gradient regularization. They derive generalization bounds from the point of view of adversarial learning, and report experiments to buttress their claims.