Improved deterministic l2 robustness on CIFAR-10 and CIFAR-100

1 INTRODUCTION . Given a neural network f : Rd → Rk , the Lipschitz constant1 Lip ( f ) enforces an upper bound on how much the output is allowed to change in proportion to a change in the input . Previous work has demonstrated that a small Lipschitz constant is useful for improved adversarial robustness ( Szegedy et al. , 2014 ; Cissé et al. , 2017 ) , generalization bounds ( Bartlett et al. , 2017 ; Long & Sedghi , 2020 ) , interpretable gradients ( Tsipras et al. , 2018 ) and Wasserstein distance estimation ( Villani , 2008 ) . Lip ( f ) also upper bounds the increase in the norm of gradient during backpropagation and can thus prevent gradient explosion during training , enabling us to train very deep networks ( Xiao et al. , 2018 ) . While heuristic methods to enforce Lipschitz constraints ( Miyato et al. , 2018 ; Gulrajani et al. , 2017 ) have achieved much practical success , they do not provably enforce a bound on Lip ( f ) globally and it remains challenging to achieve similar results when Lip ( f ) is provably bounded . Using the property : Lip ( g ◦ h ) ≤ Lip ( g ) Lip ( h ) , the Lipschitz constant of the neural network can be bounded by the product of the Lipschitz constant of all layers . While this allows us to construct 1-Lipschitz neural networks by constraining each layer to be 1-Lipschitz , Anil et al . ( 2018 ) identified a key difficulty with this approach . Because a 1-Lipschitz layer can only reduce the norm of gradient during backpropagation , backprop through each layer reduces the gradient norm , resulting in small gradient values for layers closer to the input , making training slow and difficult . To address this problem , they introduce Gradient Norm Preserving ( GNP ) architectures where each layer preserves the gradient norm during backpropagation . This involves constraining the Jacobian of each linear layer to be an orthogonal matrix and using a GNP activation function called GroupSort . GroupSort activation function ( Anil et al. , 2018 ) first separates the vector of preactivations z ∈ Rm into groups of pre-specified sizes , sorts each group in the descending order and then concatenates these sorted groups . When the group size is 2 , the resulting activation function is called MaxMin . For 1-Lipschitz CNNs , the robustness certificate for a sample x from class l is computed as Mf ( x ) / √ 2 whereMf ( x ) = fl ( x ) −maxi6=l fi ( x ) . Naturally , larger values of fl ( x ) and smaller values of maxj 6=l fj ( x ) will lead to larger certificates . This requires the last weight matrix in the 1Unless specified , we assume the Lipschitz constant under the l2 norm in this work . network , denoted by W ∈ Rk×m ( k is the number of classes , m is the dimension of the penultimate layer , m > k ) , to enforce the following constraints throughout training : ∀j , ‖Wj , :‖2 = 1 , i 6= j , Wj , : ⊥Wi , : where Wi , : denotes the ith row of W. Now suppose that for some input image with label l , we want to update W to increase the logit for the lth class . Since ‖Wl , :‖2 is constrained to be 1 , the logit can only be increased by changing the direction of the vector Wl , : . Because the other rows { Wi , : , i 6= l } are constrained to be orthogonal to Wl , : , this further requires an update for all the rows of W. Thus during training , any update made to learn some class must necessitate the forgetting of information relevant for the other classes . This can be particularly problematic when the number of classes k and thus the number of orthogonality constraints per row ( i.e. , k − 1 ) is large ( such as in CIFAR-100 ) . To address this limitation , we propose to keep the last weight layer of the network unchanged . But then the resulting function is no longer 1-Lipschitz and the certificateMf ( x ) / √ 2 is not valid . Thus , we introduce a new certification procedure that does not require the last weight layer of the network W to be orthogonal . Our certificate is then given by the following equation : min i 6=l fl ( x ) − fi ( x ) ‖Wl , : −Wi , :‖2 However , a limitation of using the above certificate is that because the weight layers are completely unconstrained , larger norms of rows ( i.e. , ‖Wi , :‖ ) can result in larger values of ‖Wl , : −Wi , :‖2 and thus smaller certificate values . To address this limitation , we normalize all rows to be of unit norm before computing the logits . While this still requires all the rows to be of unit norm , their directions are now allowed to change freely thus preventing the need to update other rows and forgetting of learned information . We show that this provides significant improvements when the number of classes is large . We call this procedure Last Layer Normalization ( abbreviated as LLN ) . On CIFAR-100 , this significantly improves both the standard ( > 3 % ) and provable robust accuracy ( > 4 % at ρ = 36/255 ) across multiple 1-Lipschitz CNN architectures ( Table 1 ) . Here , ρ is the l2 attack radius . Another limitation of existing 1-Lipschitz CNNs ( Li et al. , 2019b ; Trockman & Kolter , 2021 ; Singla & Feizi , 2021 ) is that their robustness guarantees do not scale properly with the l2 radius ρ . For example , the provable robust accuracy of ( Singla & Feizi , 2021 ) drops ∼ 30 % at ρ = 108/255 compared to 36/255 on CIFAR-10 ( Table 2 ) . To address this limitation , we introduce a certificate regularization denoted by CR ( Section 5 ) that when used along with the Householder activation results in significantly improved provable robust accuracy at larger radius values with minimal loss in standard accuracy . On the CIFAR-10 dataset , we achieve significant improvements in the provable robust accuracy for large ρ = 108/255 ( min gain of +4.96 % ) across different architectures with minimal loss in the standard accuracy ( max drop of −0.56 % ) . Results are in Table 2 . Additionally , we characterize the MaxMin activation function as a special case of the more general Householder ( HH ) activations . Recall that given z ∈ Rm , the HH transformation is a linear function reflecting z about the hyperplane vTx = 0 ( ‖v‖2 = 1 ) , given by ( I− 2vvT ) z where I− 2vvT is orthogonal because ‖v‖2 = 1 . The Householder activation function σv is defined below : σv ( z ) = { z , vT z > 0 , ( I− 2vvT ) z , vT z ≤ 0 . ( 1 ) First , note that since z = ( I − 2vvT ) z along vT z = 0 , σv is continuous . Moreover , the Jacobian ∇z σv is either I or I− 2vvT ( both orthogonal ) implying σv is GNP . Since these properties hold ∀ v : ‖v‖2 = 1 , v can be learned during the training . In fact , we prove that any GNP piecewise linear function that changes from Q1z to Q2z ( Q1 , Q2 are square orthogonal matrices ) along vT z = 0 must satisfy Q2 = Q1 ( I− 2vvT ) to be continuous ( Theorem 1 ) . Thus , this characterization proves that every GroupSort activation is a special case of the more general Householder activation function ( example in Figure 1 , discussion in Section 6 ) . In summary , in this paper , we make the following contributions : • We introduce a certification procedure without orthogonalizing the last linear layer called Last Layer Normalization . This procedure significantly enhances the standard and provable robust accuracy when the number of classes is large . Using the LipConvnet-15 network on CIFAR-100 , our modification achieves a gain of +4.71 % in provable robust accuracy ( at ρ = 36/255 ) with a gain of +4.80 % in standard accuracy ( Table 1 ) . • We introduce a Certificate Regularizer that significantly advances the provable robust accuracy with a small reduction in standard accuracy . Using LipConvnet-15 network on CIFAR-10 , we achieve +5.81 % improvement in provable robust accuracy ( at ρ = 108/255 ) with only a −0.29 % drop in standard accuracy over the existing methods ( Table 2 ) . • We introduce a class of piecewise linear GNP activation functions called Householder or HH activations . We show that the MaxMin activation is a special case of the HH activation for certain settings . We prove that Householder transformations are necessary for any GNP piecewise linear function to be continuous ( Theorem 1 ) . 2 RELATED WORK . Provably Lipschitz convolutional neural networks : The class of fully connected neural networks ( FCNs ) which are Gradient Norm Preserving ( GNP ) and 1-Lipschitz were first introduced by Anil et al . ( 2018 ) . They orthogonalize weight matrices and use GroupSort as the activation function to design each layer to be GNP . While there have been numerous works on enforcing Lipschitz constraints on convolution layers ( Cissé et al. , 2017 ; Tsuzuku et al. , 2018 ; Qian & Wegman , 2019 ; Gouk et al. , 2020 ; Sedghi et al. , 2019 ) , they either enforce loose Lipschitz bounds or are not scalable to large networks . To ensure that the Lipschitz constraint on convolutional layers is tight , multiple recent works try to construct convolution layers with an orthogonal Jacobian matrix ( Li et al. , 2019b ; Trockman & Kolter , 2021 ; Singla & Feizi , 2021 ) . These approaches avoid the aforementioned issues and allow the training of large , provably 1-Lipschitz CNNs while achieving impressive results . Provable defenses against adversarial examples : A provably robust classifier is one for which we can guarantee that the classifier ’ s prediction remains constant within some region around the input . Most of the existing methods for provable robustness either bound the Lipschitz constant of the neural network or the individual layers ( Weng et al. , 2018 ; Zhang et al. , 2019 ; 2018 ; Wong et al. , 2018 ; Wong & Kolter , 2018 ; Raghunathan et al. , 2018 ; Croce et al. , 2019 ; Singh et al. , 2018 ; Singla & Feizi , 2020 ; Zhang et al. , 2021 ) . However , these methods do not scale to large and practical networks on the ImageNet dataset ( Deng et al. , 2009 ) . To scale to such large networks , randomized smoothing ( Liu et al. , 2018 ; Cao & Gong , 2017 ; Lécuyer et al. , 2018 ; Li et al. , 2019a ; Cohen et al. , 2019 ; Salman et al. , 2019 ; Levine et al. , 2019 ; Kumar et al. , 2020a ; b ) has been proposed as a probabilistically certified defense . However , certifying robustness with high probability requires generating a large number of noisy samples leading to high inference-time computation . In contrast , the defense we propose is deterministic and hence not comparable to randomized smoothing . While Levine & Feizi ( 2021 ) provide deterministic robustness certificates using randomized smoothing , their certificates are in the l1 norm and not directly applicable for the l2 threat model studied in this work . We discuss the differences between l1 and l2 certificates in Appendix Section C .

Summary: This work is following the line of (Anil et al 2018), its goal is to propose criteria that allows to train models wirth orthogonal weights and to certify they are. The authors generalize (Anil et al 2018) by considering a more expressive non-linearity which leads to unitary jacobians, with a softer criterium during supervision which allows to obtain better accuracies. Overall, I feel the paper is well motivated, and the results are convincing.