Training Certifiably Robust Neural Networks with Efficient Local Lipschitz Bounds

1 Introduction . With the ever-growing deployment of deep neural networks , formal robustness guarantees are needed in many safety-critical applications . Strategies to improve robustness such as adversarial training only provide empirical robustness , without formal guarantees , and many existing adversarial defenses have been successfully broken using stronger attacks [ 1 ] . In contrast , certified defenses give formal robustness guarantees that any norm-bounded adversary can not alter the prediction of a given network . Bounding the global Lipschitz constant of a neural network is a computationally efficient and scalable approach to provide certifiable robustness guarantees [ 2–4 ] . The global Lipschitz bound is typically computed as the product of the spectral norm of each layer . However , this bound can be quite loose because it needs to hold for all points from the input domain , including those inputs that are far away from each other . Training a network while constraining this loose bound often imposes to high a degree of regularization and reduces network capacity . It leads to considerably lower clean accuracy in certified training compared to standard and adversarial training [ 5 , 6 ] . A local Lipschitz constant , on the other hand , bounds the norm of output perturbation only for inputs from a small region , usually selected as a neighborhood around each data point . It produces a 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . tighter bound by considering the geometry in a local region and often yields much better robustness certification [ 7 , 8 ] . Unfortunately , computing the exact local Lipschitz constant is NP-complete [ 9 ] . Obtaining reasonably tight local Lipschitz bounds via semidefinite programming [ 10 ] or mixed integer programming [ 11 ] is typically only applicable to small , previously trained networks since it is difficult to parallelize the optimization solver and make it differentiable for training . On the other hand , many existing certified defense methods have achieved success by using a training-based approach with a relatively weak but efficient bound [ 12–14 ] . Therefore , to incorporate local Lipschitz bound in training , a computationally efficient and training-friendly method must be developed . Our contributions : We propose an efficient method to incorporate a local Lipschitz bound in training deep networks , by considering the interactions between an activation layer such as a Rectified Linear Unit ( ReLU ) layer and a linear ( or convolution ) layer . Our bound is computed for each data point and this translates to different types of outputs from the activation function : constant or varying under input perturbations . If the outputs of some activation neurons are constant under local perturbation , we eliminate the corresponding rows in the previous weight layer and the corresponding columns in the next weight layer , and then compute the spectral norm of the reduced matrix . Our main insight is to use training to make the proposed local Lipschitz bound tight . This is different from existing works that find local Lipschitz bound for a fixed network [ 8 , 10 , 11 ] . Instead , we aim to enable a network to learn to tighten our proposed local Lipschitz bound during training . To achieve this , we propose to clip the activation function with an individually learnable threshold ✓ . Take ReLU for example , the output of a “ clipped ” ReLU becomes a constant when input is greater than this threshold ( see Figure 1 ) . Once the input of the ReLU is greater than the threshold or less than 0 , then this ReLU neuron does not contribute to the local Lipschitz constant , and thus the corresponding row or column of weight matrices can be removed . We also apply this method to non-ReLU activation functions such as MaxMin [ 15 ] to create constant output regions . Additionally , we also use a hinge loss function to encourage more neurons to have constant outputs . Our method can be used as a plug-in module in existing certifiable training algorithms that involve computing Lipschitz bound . Our contributions can be summarized as : • To the best of our knowledge , we are the first to incorporate a local Lipschitz bound during training for certified robustness . Our bound is provably tighter than the global Lipschitz bound and is also computationally efficient for training . • We propose to use activation functions with learnable threshold to encourage more fixed neurons during training , which assists the network to learn to tighten our bound . We show that more than 45 % rows and columns can be removed from weight matrices . • We consistently outperform state-of-the-art Lipschitz based certified defense methods for ` 2 norm robustness . On CIFAR-10 with perturbation ✏ = 36255 , we obtain 54.3 % verified accuracy with ReLU activation function and 60.7 % accuracy with MaxMin [ 15 ] , outperforming the SOTA baselines , and also achieve better clean accuracy . Our code is available at https : //github . com/yjhuangcd/local-lipschitz . 2 Related work . Bounds on Local Lipschitz Constant A sound upper bound of local Lipschitz constant ( simply referred to as “ local Lispschitz bound ” in our paper ) is a crucial property to determine the robustness of a classifier . Finding an exact local Lipschitz constant for a neural network is generally NP hard [ 16 ] , so most works focus on finding a sound upper bound . Hein and Andriushchenko [ 7 ] derived an analytical bound for 2 layer neural networks and found that local Lipschitz bounds could be much tighter than the global one and give better robustness certificates . RecurJac [ 8 ] is a recursive algorithm that analyzes the local Lipschitz constant in a neural network using a bound propagation [ 14 ] based approach . FastLip [ 17 ] is a special and weaker form of RecurJac . Fazlyab et al . [ 10 ] used a stronger semidefinite relaxation to compute a tighter bound of local Lipschitz constant . Jordan and Dimakis [ 11 ] formulated the computation of Lipschitz as an mixed integer linear programming ( MILP ) problem and they were able to solve the exact local Lipschitz constant . Although these approaches can obtain reasonably tight and sound local Lipschitz constants , none of them have been demonstrated effective for training a certifiably robust network , where high efficiency and scalability are required . Note that although an empirical estimate of local Lipschitz constant can be easily found via gradient ascent ( e.g. , the local lower bounds reported in [ 4 ] ) , it is not a sound bound and does not provide certifiable robustness guarantees . Certifiably Robust Training using Lipschitz constants The Lipschitz constant plays a central role in many works on training a certifiably robust neural network , especially for ` 2 norm robustness . Since naturally trained networks usually have very large global Lipschitz constant bounds [ 18 ] , most existing works train the network to encourage small a Lipschitz bound . Cisse et al . [ 2 ] designed networks with orthogonal weights , whose Lipschitz constants are exactly 1 . As this can be too restrictive , later works mostly use power iteration to obtain per-layer induced norms , whose product is a Lipschitz constant . Lipschitz Margin Training ( LMT ) [ 19 ] and Globally-Robust Neural Networks ( Gloro ) [ 4 ] both upper bound the worst margin via global Lipschitz constant with different loss functions . LMT constructs a new logit by adding the worst margin to all its entries except the ground truth class . Gloro construct a new logit with one more class than the original logit vector , determines whether the input sample can be certified . However , these approaches did not exploit the available local information to tighten Lipschitz bound and improve certified robustness . Box constrained propagation ( BCP ) [ 20 ] achieves a tighter outer bound than global Lipschitz based outer bound , by taking local information into consideration via interval bound ( box ) propagation . They compute the worst case logit based on the intersection of a ( global ) ball and a ( local ) box . Although box propagation considers local information , the ball propagation still uses global Lipschitz constant , and its improvement is still limited with low clean accuracy . Other Certified Defenses Besides using Lipschitz constants , one of the most popular certifiable defense against ` 1 norm bounded inputs is via the convex outer adversarial polytope [ 21 , 22 ] . [ 13 ] takes a similar approach via abstract interpretation . These methods uses linear relaxations of neural networks to compute an outer bound at the final layer . However , because the convex relaxations employed are relatively expensive , these methods are typically slow to train . A simple and fast certifiable defense for ` 1 norm bounded inputs is interval bound propagation ( IBP ) [ 12 , 13 ] . Since the IBP bound can be quite loose for general networks , its good performance relies on appropriate hyper-parameters . CROWN-IBP [ 14 ] outperforms previous methods by combining IBP bound in a forward bounding pass and a tighter linear relaxation bound in a backward bound pass . [ 23 ] improved IBP with better initialization to accelerate training . Additionally , randomized smoothing [ 24–26 ] is a probabilistic method to certify ` 2 norm robustness with arbitrarily high confidence . The prediction of a randomized smooth classifier is the most likely prediction returned by the base classifier that is fed by samples from a Gaussian distribution . Salman et al . [ 27 ] further improves the performance of randomized smoothing via adversarial training . 3 Method . We begin with notations and background for Lipschitz bound . We introduce our method for local Lipschitz bound computation in Section 3.2 . Then we introduce how to incorporate our efficient local Lipschitz bound in robust training ( Section 3.3 ) . 3.1 Notation and Background . Notations . We denote the Euclidean norm of a vector x as kxk and kAk is the spectral norm of matrix A. Subscript of vector x denotes element , i.e. , xi is the i-th element of x . We use LBl and UBl to denote lower bounds and upper bounds of pre-ReLU activation values for layer l. Definition 1 . The Lipschitz constant of a function f : Rd ! Rm over an open set X is defined as , L ( f , X ) : = sup x , y2X , x 6=y ||f ( y ) f ( x ) || ||x y|| , If L ( f , X ) exists and is finite , we say that f is Lipschitz continuous over X . Suppose X = dom ( f ) , L ( f , X ) is the global Lipschitz constant of f ; if X is defined as the ✏-ball at point x , i.e. , X : = { x0|||x x0|| ✏ } , then L ( f , X ) is the local Lipschitz constant of f at x . Global Lipschitz bound in existing works Consider a L-layer ReLU neural network which maps input x to output zL+1 = F ( x ; W ) using the following architecture , for l = 1 , ... , L 1 z1 = x ; zl+1 = ( W lzl ) ; zL+1 = WLzL ( 1 ) where W = { W 1 : L } are the parameters , and ( · ) = max ( · , 0 ) is the element-wise ReLU activation functions . Here we consider the bias parameters to be zero because they do not contribute to the Lipschitz bound . Since the Lipschitz constant of ReLU activation ( · ) is equal to 1 , a global Lipschitz bound of F is , Lglob ||WL|| · ||WL 1|| · · · ||W 1|| ( 2 ) where ||W l|| equals the spectral norm ( maximum singular value ) of the weight matrix W l. However , the global Lipchitz bound ignores the highly nonlinear property of deep neural networks . In what follows , we introduce our method that considers the interaction between ReLU and linear layer to obtain a tighter local Lipchitz bound in a computationally efficient way , that allows us to train a certifiably robust network using local Lipschitz bounds .

Summary: This paper provides a trainable local Lipschitz upper-bound for a neural network that is tighter than the global Lipschitz upper-bound computed via multiplying the Lipschitz constant of each individual layer. By making use of the interaction between the weight matrices and the piece-wise linear activation functions, the authors demonstrate that the global Lipschitz bound could be improved when the activation functions are constant locally. In addition, the authors proposed a variant of ReLU that is clipped at a certain upper threshold to increase the constant region to help achieve a tighter Lipschitz upper-bound. Empirically, the authors showed that optimizing the local Lipschitz bound, along with using the modified ReLU, improved the certified robust performance of existing algorithms on several standard image classification tasks.