Provable Robustness by Geometric Regularization of ReLU Networks

1 INTRODUCTION . Neural networks have been very successful in tasks such as image classification and speech recognition . However , recent work ( Szegedy et al. , 2014 ; Goodfellow et al. , 2015 ) has demonstrated that neural networks classifiers can be arbitrarily fooled by small , adversarially-chosen perturbations of their inputs . Notably , Su et al . ( 2017 ) demonstrated that neural network classifiers which can correctly classify “ clean ” images may be vulnerable to targeted attacks , e.g. , misclassify those same images when only a single pixel is changed . Previous work demonstrating this fragility of neural network classifiers to adversarial noise has motivated the development of many heuristic defenses including adversarial training ( Madry et al. , 2018 ) as well as certifiably robust classifiers such as randomized smoothing ( Cohen et al. , 2019 ; Salman et al. , 2019 ) which characterize the robustness of a classifier according to its smoothness . The intrinsic relationship between smoothness , or Lipschitz continuity—and their corresponding local variants—and robustness has motivated a variety of techniques to encourage uniform and local smoothness through the explicit regularization of approximations of the global and local Lipschitz constants ( Zhang et al. , 2018 ; Weng et al. , 2018a ; b ) . Recently , Lecuyer et al . ( 2019 ) ; Li et al . ( 2018 ) ; Cohen et al . ( 2019 ) ; Salman et al . ( 2019 ) proposed and extended a simple , scalable technique—randomized smoothing—to transform arbitrary functions ( e.g . neural network classifiers ) into certifiably and robust classifiers on ` 2 perturbations . Alternatively , previous work has also addressed adversarial robustness in the context of piecewiselinear classifiers ( e.g. , feedforward neural networks with ReLU activations ) . Wong & Kolter ( 2018 ) ; Jordan et al . ( 2019 ) propose to certify the robustness of a network f at an example x by considering a bound on the radius of the maximum ` p-norm ball contained within a union of polytopes over which f predicts the same class . Related to our work , Croce et al . ; Liu et al . ( 2020 ) propose maximum margin regularizers ( MMR ) which quantifies robustness of a network at a point according to the local region in which it lies and the distance to the classification boundary . Recent work also includes recovery and analysis of the piecewise linear function learned by an ReLU neural network during a training process ( Arora et al. , 2018 ; Montúfar et al. , 2014 ; Croce & Hein , 2019 ) . Typically , work in this area centers around studying the complexity , interpretation , and improvement of stability and robustness of neural networks . For example , Montúfar et al . ( 2014 ) ; Serra et al . ( 2017 ) studied piecewise linear representations of neural networks and proposed the “ activation profile ” to characterize the linear regions . In this work , we propose a novel regularizer for feedforward piecewise-linear neural networks , including convolutional neural networks , to increase their robustness to adversarial perturbations . Our Geometric Regularization ( GR ) method is based on the fact that ReLU networks define continuous piecewise affine functions and is inspired by classical techniques from convex geometry and linear programming . We provide a novel robustness certificate based on the local polytope geometry of a point and show that our regularizer provably maximizes this certificate . We evaluate the efficacy of our method on three datasets . Notably , our method works regardless of the perturbation model and relies on fewer hyperparameters compared with related approaches . We demonstrate that our regularization term leads to classifiers that are empirically robust and are comparable to the state of the art algorithms with respect to clean and robust test accuracy under ` 1 and ` ∞-norm adversarial perturbations . 2 PRELIMINARIES . In this section , we briefly present background terminology pertaining to polytopes and their characterizations , adversarially robust classification , and the polytope decomposition of the domain induced by an ReLU network and its linearization over a given polytope . 2.1 PIECEWISE-LINEAR NETWORKS . An ReLU network is a neural network such that all nonlinear activations are ReLU functions , where we denote the ReLU activation by σ : R → R , σ ( x ) = max { 0 , x } . Informally , we define σ : Rd → Rd by σ ( x ) = [ σ ( x1 ) , . . . , σ ( xd ) ] . Let f : Rd → [ 0 , 1 ] k be a feedforward ReLU network with L hidden layers ; for example , f may map from a d-dimensional image to a k-dimensional vector corresponding to likelihoods for k classes . Let nl be the number of hidden units at layer l , the input layer is of size n0 = d , and let W ( l ) ∈ Rnl−1×nl and b ( l ) ∈ Rnl denote the weight matrix and bias vector at layer l , respectively . Since f may be represented as the composition of L+ 1 linear transformations and L continuous piecewise-affine functions , f must necessarily be continuous and piecewise-affine ( for brevity , we will say piecewise-linear ) . The half-space representation , orH-representation , of convex polytopes is defined as follows : Definition 2.1 ( Convex polytope ) . A convex polytope K is the convex hull of finitely many points . Alternatively , a convex polytope may be expressed as an intersection of m half-spaces . The Hrepresentation of a polytope is defined as the solution set to a system of linear inequalities Ax ≤ b : K = { x : ∀j ∈ [ m ] , aj · x ≤ bj } Following definition 3.1 from Croce et al. , a function is piecewise-linear if there exists a finite set of convex polytopes { Qr } mr=1 ( referred to as linear regions of f ) such that ∪mr=1Qr = Rd and f is affine when restricted to each Qr , i.e. , can be expressed on Qr as f ( x ) = V x+ a . Given a feedforward ReLU network f and an input x ∈ Rd , we intend to recover the polytope Q conditioned on x and the linear restriction of f on Q . Therefore , we need to find A and b such that Ax ≤ b , where A and b define the intersection of k half-spaces , i.e . a polytope , and V and a corresponding to the linearization of f within this polytope such that f ( x ) = V x+ a when restricted to Q . We follow the formulation and notations of Croce et al .. If x ∈ Rd and g ( 0 ) ( x ) = x we recursively define the pre- and post-activation output of every layer : f ( l ) ( x ) = W ( l ) g ( l−1 ) ( x ) + b ( l ) g ( l−1 ) ( x ) = σ ( f ( l ) ( x ) ) The resulting classifier is then : f ( L+1 ) = W ( L+1 ) g ( L ) ( x ) + b ( L+1 ) . By rewriting σ as an affine function , we can write f ( l ) formally as a composition of affine functions : f ( l ) ( x ) = W ( l ) Σ ( l−1 ) ( x ) ( . . . ( W ( 1 ) x+ b ( 1 ) ) . . . ) + b ( h ) , where we define ∆ ( l ) , Σ ( l ) ∈ Rnl×nl conditioned on x and defined elementwise as : ∆ ( l ) i , j = { sign ( f ( l ) i ( x ) ) if i = j 0 otherwise , Σ ( l ) i , j = { 1 if i = j and f ( l ) i ( x ) > 0 0 otherwise We now derive the polytope of f at x and the linear restriction of f on polytope Q . By expanding the composition , we can concisely write f ( l ) ( x ) = V ( l ) x+ a ( l ) , where V ( l ) = W ( l ) ( l−l∏ h=1 Σ ( l−h ) ( x ) W ( l−h ) ) , a ( l ) = b ( l ) + l−1∑ h=1 ( l−1∏ m=1 W ( l+1−m ) Σ ( l−m ) ( x ) ) b ( h ) , ( 1 ) and characterize the polytope Q where x lies as the intersection of N = ∑L l=1 nl half-spaces : Γl , i = { z ∈ Rd|∆ ( l ) ( x ) ( V ( l ) z + a ( l ) ) ≥ 0 } . 2.2 POLYTOPE CENTERS . There are various definitions of the “ center ” of a polytope . Notably , the Chebyshev center of a polytope K is the center of the largest inscribed ball of K , or equivalently , the interior point of K that maximizes the minimum distance between itself and the boundary of K. More formally , Definition 2.2 ( Chebyshev center ) . Let K be a convex polytope . The Chebyshev center of K with respect to an ` p distance is the point x ∈ Rd which satisfies the following min-max problem : arg min x̂ max x∈K ||x− x̂||2p Previous work explored the Chebyshev center in the context of adversarial machine learning ( Croce et al . ; Jordan et al. , 2019 ) . For example , Croce & Hein ( 2019 ) propose to include the minimum distance to the boundary in a non-smooth regularization term in their Maximum Margin Regularizer ( MMR ) to encourage samples to lie close to the Chebyshev centers of their respective polytopes . In contrast , we explore the application of an alternative polytope center : the analytic center . The analytic center of a convex polytope K expressed via theH-representation Ax ≤ b is canonically defined as an element of K that maximizes the product of the distances to the hyperplanes characterized by the rows of A and the elements of b . Definition 2.3 ( Analytic center ) . Let K be a convex polytope expressed via the H-representation : Ax ≤ b . The analytic center of K is the point x ∈ Rd which satisfies the following objectives : xac = arg max x m∏ i=1 ( bi − d∑ j=1 aijxj ) = arg min x − m∑ i=1 log ( bi − d∑ j=1 aijxj ) ( 2 ) s.t . x ∈ K s.t . x ∈ K , where the second objective is canonically known as the logarithmic barrier potential ( Nesterov & Nemirovskii , 1994 ) . It naturally follows that when boundary-planes of the polytope are symmetric about the analytic center ( e.g . for polytopes that satisfy a central symmetry property ) , it exactly corresponds to the Chebyshev center . A polytope K ⊂ Rd is centrally symmetric if K = −K ; that is , x ∈ K if and only if there is a unique point y such that the reflection of x about y is in K : 2y − x ∈ K. The analytic center , namely a weighted analytic center , has been extensively used in interior point ( IP ) methods for linear programming ( Boyd & Vandenberghe , 2004 ) . In general , the measure of closeness to the analytic center is an important factor for the convergence of center-following algorithms . The properties of the log barrier potential that make it a popular choice for interior point methods ( including being twice continuously differentiable and convex ) suggest we might use it as a regularizer to encourage robustness during learning . Note , however , that the analytic center depends on how the set of particular inequalities is defined . The addition of redundant inequalities could feasibly push the analytic center arbitrarily close to the boundary of the polytope . We discuss this further in Sec . 3 .

Summary: The authors exploit the piecewise linear nature of ReLU neural networks to design a new regularizer that improves the robustness of the neural network. It can be viewed as a alternative to the regularizer proposed in Croce et al. (2018) -- the current regularizer uses the analytic center, whereas the previous work (named MMR) guarantees that are straightforward consequences of the geometric interpretation. The experimental setup is similar to the one in Croce et al. (2018), which compares the result of different adversarial defense methods on MNIST, F-MNIST, and CIFAR10 on small shallow networks. These computational results generally slightly outperform MMR on MNIST.