Improved deterministic l2 robustness on CIFAR-10 and CIFAR-100

1 INTRODUCTION . Given a neural network f : Rd → Rk , the Lipschitz constant1 Lip ( f ) enforces an upper bound on how much the output is allowed to change in proportion to a change in the input . Previous work has demonstrated that a small Lipschitz constant is useful for improved adversarial robustness ( Szegedy et al. , 2014 ; Cissé et al. , 2017 ) , generalization bounds ( Bartlett et al. , 2017 ; Long & Sedghi , 2020 ) , interpretable gradients ( Tsipras et al. , 2018 ) and Wasserstein distance estimation ( Villani , 2008 ) . Lip ( f ) also upper bounds the increase in the norm of gradient during backpropagation and can thus prevent gradient explosion during training , enabling us to train very deep networks ( Xiao et al. , 2018 ) . While heuristic methods to enforce Lipschitz constraints ( Miyato et al. , 2018 ; Gulrajani et al. , 2017 ) have achieved much practical success , they do not provably enforce a bound on Lip ( f ) globally and it remains challenging to achieve similar results when Lip ( f ) is provably bounded . Using the property : Lip ( g ◦ h ) ≤ Lip ( g ) Lip ( h ) , the Lipschitz constant of the neural network can be bounded by the product of the Lipschitz constant of all layers . While this allows us to construct 1-Lipschitz neural networks by constraining each layer to be 1-Lipschitz , Anil et al . ( 2018 ) identified a key difficulty with this approach . Because a 1-Lipschitz layer can only reduce the norm of gradient during backpropagation , backprop through each layer reduces the gradient norm , resulting in small gradient values for layers closer to the input , making training slow and difficult . To address this problem , they introduce Gradient Norm Preserving ( GNP ) architectures where each layer preserves the gradient norm during backpropagation . This involves constraining the Jacobian of each linear layer to be an orthogonal matrix and using a GNP activation function called GroupSort . GroupSort activation function ( Anil et al. , 2018 ) first separates the vector of preactivations z ∈ Rm into groups of pre-specified sizes , sorts each group in the descending order and then concatenates these sorted groups . When the group size is 2 , the resulting activation function is called MaxMin . For 1-Lipschitz CNNs , the robustness certificate for a sample x from class l is computed as Mf ( x ) / √ 2 whereMf ( x ) = fl ( x ) −maxi6=l fi ( x ) . Naturally , larger values of fl ( x ) and smaller values of maxj 6=l fj ( x ) will lead to larger certificates . This requires the last weight matrix in the 1Unless specified , we assume the Lipschitz constant under the l2 norm in this work . network , denoted by W ∈ Rk×m ( k is the number of classes , m is the dimension of the penultimate layer , m > k ) , to enforce the following constraints throughout training : ∀j , ‖Wj , :‖2 = 1 , i 6= j , Wj , : ⊥Wi , : where Wi , : denotes the ith row of W. Now suppose that for some input image with label l , we want to update W to increase the logit for the lth class . Since ‖Wl , :‖2 is constrained to be 1 , the logit can only be increased by changing the direction of the vector Wl , : . Because the other rows { Wi , : , i 6= l } are constrained to be orthogonal to Wl , : , this further requires an update for all the rows of W. Thus during training , any update made to learn some class must necessitate the forgetting of information relevant for the other classes . This can be particularly problematic when the number of classes k and thus the number of orthogonality constraints per row ( i.e. , k − 1 ) is large ( such as in CIFAR-100 ) . To address this limitation , we propose to keep the last weight layer of the network unchanged . But then the resulting function is no longer 1-Lipschitz and the certificateMf ( x ) / √ 2 is not valid . Thus , we introduce a new certification procedure that does not require the last weight layer of the network W to be orthogonal . Our certificate is then given by the following equation : min i 6=l fl ( x ) − fi ( x ) ‖Wl , : −Wi , :‖2 However , a limitation of using the above certificate is that because the weight layers are completely unconstrained , larger norms of rows ( i.e. , ‖Wi , :‖ ) can result in larger values of ‖Wl , : −Wi , :‖2 and thus smaller certificate values . To address this limitation , we normalize all rows to be of unit norm before computing the logits . While this still requires all the rows to be of unit norm , their directions are now allowed to change freely thus preventing the need to update other rows and forgetting of learned information . We show that this provides significant improvements when the number of classes is large . We call this procedure Last Layer Normalization ( abbreviated as LLN ) . On CIFAR-100 , this significantly improves both the standard ( > 3 % ) and provable robust accuracy ( > 4 % at ρ = 36/255 ) across multiple 1-Lipschitz CNN architectures ( Table 1 ) . Here , ρ is the l2 attack radius . Another limitation of existing 1-Lipschitz CNNs ( Li et al. , 2019b ; Trockman & Kolter , 2021 ; Singla & Feizi , 2021 ) is that their robustness guarantees do not scale properly with the l2 radius ρ . For example , the provable robust accuracy of ( Singla & Feizi , 2021 ) drops ∼ 30 % at ρ = 108/255 compared to 36/255 on CIFAR-10 ( Table 2 ) . To address this limitation , we introduce a certificate regularization denoted by CR ( Section 5 ) that when used along with the Householder activation results in significantly improved provable robust accuracy at larger radius values with minimal loss in standard accuracy . On the CIFAR-10 dataset , we achieve significant improvements in the provable robust accuracy for large ρ = 108/255 ( min gain of +4.96 % ) across different architectures with minimal loss in the standard accuracy ( max drop of −0.56 % ) . Results are in Table 2 . Additionally , we characterize the MaxMin activation function as a special case of the more general Householder ( HH ) activations . Recall that given z ∈ Rm , the HH transformation is a linear function reflecting z about the hyperplane vTx = 0 ( ‖v‖2 = 1 ) , given by ( I− 2vvT ) z where I− 2vvT is orthogonal because ‖v‖2 = 1 . The Householder activation function σv is defined below : σv ( z ) = { z , vT z > 0 , ( I− 2vvT ) z , vT z ≤ 0 . ( 1 ) First , note that since z = ( I − 2vvT ) z along vT z = 0 , σv is continuous . Moreover , the Jacobian ∇z σv is either I or I− 2vvT ( both orthogonal ) implying σv is GNP . Since these properties hold ∀ v : ‖v‖2 = 1 , v can be learned during the training . In fact , we prove that any GNP piecewise linear function that changes from Q1z to Q2z ( Q1 , Q2 are square orthogonal matrices ) along vT z = 0 must satisfy Q2 = Q1 ( I− 2vvT ) to be continuous ( Theorem 1 ) . Thus , this characterization proves that every GroupSort activation is a special case of the more general Householder activation function ( example in Figure 1 , discussion in Section 6 ) . In summary , in this paper , we make the following contributions : • We introduce a certification procedure without orthogonalizing the last linear layer called Last Layer Normalization . This procedure significantly enhances the standard and provable robust accuracy when the number of classes is large . Using the LipConvnet-15 network on CIFAR-100 , our modification achieves a gain of +4.71 % in provable robust accuracy ( at ρ = 36/255 ) with a gain of +4.80 % in standard accuracy ( Table 1 ) . • We introduce a Certificate Regularizer that significantly advances the provable robust accuracy with a small reduction in standard accuracy . Using LipConvnet-15 network on CIFAR-10 , we achieve +5.81 % improvement in provable robust accuracy ( at ρ = 108/255 ) with only a −0.29 % drop in standard accuracy over the existing methods ( Table 2 ) . • We introduce a class of piecewise linear GNP activation functions called Householder or HH activations . We show that the MaxMin activation is a special case of the HH activation for certain settings . We prove that Householder transformations are necessary for any GNP piecewise linear function to be continuous ( Theorem 1 ) . 2 RELATED WORK . Provably Lipschitz convolutional neural networks : The class of fully connected neural networks ( FCNs ) which are Gradient Norm Preserving ( GNP ) and 1-Lipschitz were first introduced by Anil et al . ( 2018 ) . They orthogonalize weight matrices and use GroupSort as the activation function to design each layer to be GNP . While there have been numerous works on enforcing Lipschitz constraints on convolution layers ( Cissé et al. , 2017 ; Tsuzuku et al. , 2018 ; Qian & Wegman , 2019 ; Gouk et al. , 2020 ; Sedghi et al. , 2019 ) , they either enforce loose Lipschitz bounds or are not scalable to large networks . To ensure that the Lipschitz constraint on convolutional layers is tight , multiple recent works try to construct convolution layers with an orthogonal Jacobian matrix ( Li et al. , 2019b ; Trockman & Kolter , 2021 ; Singla & Feizi , 2021 ) . These approaches avoid the aforementioned issues and allow the training of large , provably 1-Lipschitz CNNs while achieving impressive results . Provable defenses against adversarial examples : A provably robust classifier is one for which we can guarantee that the classifier ’ s prediction remains constant within some region around the input . Most of the existing methods for provable robustness either bound the Lipschitz constant of the neural network or the individual layers ( Weng et al. , 2018 ; Zhang et al. , 2019 ; 2018 ; Wong et al. , 2018 ; Wong & Kolter , 2018 ; Raghunathan et al. , 2018 ; Croce et al. , 2019 ; Singh et al. , 2018 ; Singla & Feizi , 2020 ; Zhang et al. , 2021 ) . However , these methods do not scale to large and practical networks on the ImageNet dataset ( Deng et al. , 2009 ) . To scale to such large networks , randomized smoothing ( Liu et al. , 2018 ; Cao & Gong , 2017 ; Lécuyer et al. , 2018 ; Li et al. , 2019a ; Cohen et al. , 2019 ; Salman et al. , 2019 ; Levine et al. , 2019 ; Kumar et al. , 2020a ; b ) has been proposed as a probabilistically certified defense . However , certifying robustness with high probability requires generating a large number of noisy samples leading to high inference-time computation . In contrast , the defense we propose is deterministic and hence not comparable to randomized smoothing . While Levine & Feizi ( 2021 ) provide deterministic robustness certificates using randomized smoothing , their certificates are in the l1 norm and not directly applicable for the l2 threat model studied in this work . We discuss the differences between l1 and l2 certificates in Appendix Section C .

Summary: This paper addresses the problem of provable L2 robustness in image classification. Three techniques are proposed as additions on top of the SOC work (Singla & Feizi, 2021): 1) let the last linear layer be non-orthogonal and adjust provable L2 robustness calculation accordingly; 2) add a loss term that encourages larger margins in logits; 3) a new nonlinearity.  Comparisons against the original SOC work are reported on CIFAR-10 and CIFAR-100.