Training Certifiably Robust Neural Networks with Efficient Local Lipschitz Bounds

1 Introduction . With the ever-growing deployment of deep neural networks , formal robustness guarantees are needed in many safety-critical applications . Strategies to improve robustness such as adversarial training only provide empirical robustness , without formal guarantees , and many existing adversarial defenses have been successfully broken using stronger attacks [ 1 ] . In contrast , certified defenses give formal robustness guarantees that any norm-bounded adversary can not alter the prediction of a given network . Bounding the global Lipschitz constant of a neural network is a computationally efficient and scalable approach to provide certifiable robustness guarantees [ 2–4 ] . The global Lipschitz bound is typically computed as the product of the spectral norm of each layer . However , this bound can be quite loose because it needs to hold for all points from the input domain , including those inputs that are far away from each other . Training a network while constraining this loose bound often imposes to high a degree of regularization and reduces network capacity . It leads to considerably lower clean accuracy in certified training compared to standard and adversarial training [ 5 , 6 ] . A local Lipschitz constant , on the other hand , bounds the norm of output perturbation only for inputs from a small region , usually selected as a neighborhood around each data point . It produces a 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . tighter bound by considering the geometry in a local region and often yields much better robustness certification [ 7 , 8 ] . Unfortunately , computing the exact local Lipschitz constant is NP-complete [ 9 ] . Obtaining reasonably tight local Lipschitz bounds via semidefinite programming [ 10 ] or mixed integer programming [ 11 ] is typically only applicable to small , previously trained networks since it is difficult to parallelize the optimization solver and make it differentiable for training . On the other hand , many existing certified defense methods have achieved success by using a training-based approach with a relatively weak but efficient bound [ 12–14 ] . Therefore , to incorporate local Lipschitz bound in training , a computationally efficient and training-friendly method must be developed . Our contributions : We propose an efficient method to incorporate a local Lipschitz bound in training deep networks , by considering the interactions between an activation layer such as a Rectified Linear Unit ( ReLU ) layer and a linear ( or convolution ) layer . Our bound is computed for each data point and this translates to different types of outputs from the activation function : constant or varying under input perturbations . If the outputs of some activation neurons are constant under local perturbation , we eliminate the corresponding rows in the previous weight layer and the corresponding columns in the next weight layer , and then compute the spectral norm of the reduced matrix . Our main insight is to use training to make the proposed local Lipschitz bound tight . This is different from existing works that find local Lipschitz bound for a fixed network [ 8 , 10 , 11 ] . Instead , we aim to enable a network to learn to tighten our proposed local Lipschitz bound during training . To achieve this , we propose to clip the activation function with an individually learnable threshold ✓ . Take ReLU for example , the output of a “ clipped ” ReLU becomes a constant when input is greater than this threshold ( see Figure 1 ) . Once the input of the ReLU is greater than the threshold or less than 0 , then this ReLU neuron does not contribute to the local Lipschitz constant , and thus the corresponding row or column of weight matrices can be removed . We also apply this method to non-ReLU activation functions such as MaxMin [ 15 ] to create constant output regions . Additionally , we also use a hinge loss function to encourage more neurons to have constant outputs . Our method can be used as a plug-in module in existing certifiable training algorithms that involve computing Lipschitz bound . Our contributions can be summarized as : • To the best of our knowledge , we are the first to incorporate a local Lipschitz bound during training for certified robustness . Our bound is provably tighter than the global Lipschitz bound and is also computationally efficient for training . • We propose to use activation functions with learnable threshold to encourage more fixed neurons during training , which assists the network to learn to tighten our bound . We show that more than 45 % rows and columns can be removed from weight matrices . • We consistently outperform state-of-the-art Lipschitz based certified defense methods for ` 2 norm robustness . On CIFAR-10 with perturbation ✏ = 36255 , we obtain 54.3 % verified accuracy with ReLU activation function and 60.7 % accuracy with MaxMin [ 15 ] , outperforming the SOTA baselines , and also achieve better clean accuracy . Our code is available at https : //github . com/yjhuangcd/local-lipschitz . 2 Related work . Bounds on Local Lipschitz Constant A sound upper bound of local Lipschitz constant ( simply referred to as “ local Lispschitz bound ” in our paper ) is a crucial property to determine the robustness of a classifier . Finding an exact local Lipschitz constant for a neural network is generally NP hard [ 16 ] , so most works focus on finding a sound upper bound . Hein and Andriushchenko [ 7 ] derived an analytical bound for 2 layer neural networks and found that local Lipschitz bounds could be much tighter than the global one and give better robustness certificates . RecurJac [ 8 ] is a recursive algorithm that analyzes the local Lipschitz constant in a neural network using a bound propagation [ 14 ] based approach . FastLip [ 17 ] is a special and weaker form of RecurJac . Fazlyab et al . [ 10 ] used a stronger semidefinite relaxation to compute a tighter bound of local Lipschitz constant . Jordan and Dimakis [ 11 ] formulated the computation of Lipschitz as an mixed integer linear programming ( MILP ) problem and they were able to solve the exact local Lipschitz constant . Although these approaches can obtain reasonably tight and sound local Lipschitz constants , none of them have been demonstrated effective for training a certifiably robust network , where high efficiency and scalability are required . Note that although an empirical estimate of local Lipschitz constant can be easily found via gradient ascent ( e.g. , the local lower bounds reported in [ 4 ] ) , it is not a sound bound and does not provide certifiable robustness guarantees . Certifiably Robust Training using Lipschitz constants The Lipschitz constant plays a central role in many works on training a certifiably robust neural network , especially for ` 2 norm robustness . Since naturally trained networks usually have very large global Lipschitz constant bounds [ 18 ] , most existing works train the network to encourage small a Lipschitz bound . Cisse et al . [ 2 ] designed networks with orthogonal weights , whose Lipschitz constants are exactly 1 . As this can be too restrictive , later works mostly use power iteration to obtain per-layer induced norms , whose product is a Lipschitz constant . Lipschitz Margin Training ( LMT ) [ 19 ] and Globally-Robust Neural Networks ( Gloro ) [ 4 ] both upper bound the worst margin via global Lipschitz constant with different loss functions . LMT constructs a new logit by adding the worst margin to all its entries except the ground truth class . Gloro construct a new logit with one more class than the original logit vector , determines whether the input sample can be certified . However , these approaches did not exploit the available local information to tighten Lipschitz bound and improve certified robustness . Box constrained propagation ( BCP ) [ 20 ] achieves a tighter outer bound than global Lipschitz based outer bound , by taking local information into consideration via interval bound ( box ) propagation . They compute the worst case logit based on the intersection of a ( global ) ball and a ( local ) box . Although box propagation considers local information , the ball propagation still uses global Lipschitz constant , and its improvement is still limited with low clean accuracy . Other Certified Defenses Besides using Lipschitz constants , one of the most popular certifiable defense against ` 1 norm bounded inputs is via the convex outer adversarial polytope [ 21 , 22 ] . [ 13 ] takes a similar approach via abstract interpretation . These methods uses linear relaxations of neural networks to compute an outer bound at the final layer . However , because the convex relaxations employed are relatively expensive , these methods are typically slow to train . A simple and fast certifiable defense for ` 1 norm bounded inputs is interval bound propagation ( IBP ) [ 12 , 13 ] . Since the IBP bound can be quite loose for general networks , its good performance relies on appropriate hyper-parameters . CROWN-IBP [ 14 ] outperforms previous methods by combining IBP bound in a forward bounding pass and a tighter linear relaxation bound in a backward bound pass . [ 23 ] improved IBP with better initialization to accelerate training . Additionally , randomized smoothing [ 24–26 ] is a probabilistic method to certify ` 2 norm robustness with arbitrarily high confidence . The prediction of a randomized smooth classifier is the most likely prediction returned by the base classifier that is fed by samples from a Gaussian distribution . Salman et al . [ 27 ] further improves the performance of randomized smoothing via adversarial training . 3 Method . We begin with notations and background for Lipschitz bound . We introduce our method for local Lipschitz bound computation in Section 3.2 . Then we introduce how to incorporate our efficient local Lipschitz bound in robust training ( Section 3.3 ) . 3.1 Notation and Background . Notations . We denote the Euclidean norm of a vector x as kxk and kAk is the spectral norm of matrix A. Subscript of vector x denotes element , i.e. , xi is the i-th element of x . We use LBl and UBl to denote lower bounds and upper bounds of pre-ReLU activation values for layer l. Definition 1 . The Lipschitz constant of a function f : Rd ! Rm over an open set X is defined as , L ( f , X ) : = sup x , y2X , x 6=y ||f ( y ) f ( x ) || ||x y|| , If L ( f , X ) exists and is finite , we say that f is Lipschitz continuous over X . Suppose X = dom ( f ) , L ( f , X ) is the global Lipschitz constant of f ; if X is defined as the ✏-ball at point x , i.e. , X : = { x0|||x x0|| ✏ } , then L ( f , X ) is the local Lipschitz constant of f at x . Global Lipschitz bound in existing works Consider a L-layer ReLU neural network which maps input x to output zL+1 = F ( x ; W ) using the following architecture , for l = 1 , ... , L 1 z1 = x ; zl+1 = ( W lzl ) ; zL+1 = WLzL ( 1 ) where W = { W 1 : L } are the parameters , and ( · ) = max ( · , 0 ) is the element-wise ReLU activation functions . Here we consider the bias parameters to be zero because they do not contribute to the Lipschitz bound . Since the Lipschitz constant of ReLU activation ( · ) is equal to 1 , a global Lipschitz bound of F is , Lglob ||WL|| · ||WL 1|| · · · ||W 1|| ( 2 ) where ||W l|| equals the spectral norm ( maximum singular value ) of the weight matrix W l. However , the global Lipchitz bound ignores the highly nonlinear property of deep neural networks . In what follows , we introduce our method that considers the interaction between ReLU and linear layer to obtain a tighter local Lipchitz bound in a computationally efficient way , that allows us to train a certifiably robust network using local Lipschitz bounds .

Summary: Per data-example, a local Lipschitz bound on the neural network output is computed and then used to certify robustness. This bound is made tighter by exploiting invariances of the ReLU activations throughout the network under small perturbations around the given data point. The paper further increases invariances by clipping ReLU and training the clipping threshold.