L2-Nonexpansive Neural Networks

This paper proposes a class of well-conditioned neural networks in which a unit amount of change in the inputs causes at most a unit amount of change in the outputs or any of the internal layers . We develop the known methodology of controlling Lipschitz constants to realize its full potential in maximizing robustness , with a new regularization scheme for linear layers , new ways to adapt nonlinearities and a new loss function . With MNIST and CIFAR-10 classifiers , we demonstrate a number of advantages . Without needing any adversarial traning , the proposed classifiers exceed the state of the art in robustness againstwhite-boxL2-bounded adversarial attacks . They generalize better than ordinaryetworks from noisy data with partially random labels . Their outputs are quantit tively meaningful and indicate levels of confidence and generalization , among other desirable properties . 1 INTRODUCTION . Artificial neural networks are often ill-conditioned systems in that a small change in the inputs can cause significant changes in the outputs ( Szegedy et al.,2014 ) . This results in poor robustness and vulnerability under adversarial attacks which has beenr ported on a variety of networks including image classification ( Carlini & Wagner , 2017a ; Goodfellow et al. , 2014 ) , speech recognition ( Kreuk et al. , 2018 ; Alzantot et al. , 2018 ; Carlini & Wagner,2018 ) , image captioning ( Chen et al. , 2017 ) and natural language processing ( Gao et al. , 2018 ; Ebrahimi et al. , 2017 ) . These issues bring up both theoretical questions of how neural networks generalize ( Kawaguchi et al. , 2017 ; Xu & Mannor , 2012 ) and practical concerns of security in applications ( Akhtar & Mian , 2018 ) . A number of remedies have been proposed for these issues and will be discussed in Section 4 . Whitebox defense is particularly difficult and many proposals have failed . For example , Athalye et al . ( 2018 ) reported that out of eight recent defense works , onlyMadry et al . ( 2017 ) survived strong attacks . So far the mainstream and most successful remedy is that of adversarial training ( Madry et al. , 2017 ) . However , as will be shown in Tables 1 and 2 , the robustnes by adversarial training diminishes when a white-box attacker ( Carlini & Wagner , 2017a ) isllowed to use more iterations . This paper explores a different approach and demonstrates that a combination of the following three conditions results in enhanced robustness : 1 ) the Lipschitz onstant of a network from inputs to logits is no greater than 1 with respect to theL2-norm ; 2 ) the loss function explicitly maximizes confidence gap , which is the difference between the largest and second largest logits of a classifier ; 3 ) the network architecture restricts confidence gaps as little as possible . We will elaborate . There are previous works that achieve the first condition ( Cisse et al. , 2017 ; Hein & Andriushchenko , 2017 ) or bound responses to input perturbations by other means ( Kolter & Wong , 2017 ; Raghunathan et al. , 2018 ; Haber & Ruthotto , 2017 ) . For example , Parseval networks ( Cisse et al. , 2017 ) bound the Lipschitz constant by requiring each linear or convolution layer be composed of orthonormal filters . However , the report d obustness and guarantees are often under weak attacks or with low noise magnitude , and none of these works has demonstrated results that are comparable to adversarial training . In contrast , we are able to build MNIST and CIFAR-10 classifiers , without needing any adversarial training , that exceed the state of the art ( Madry et al. , 2017 ) in robustness against white-boxL2bounded adversarial attacks . The defense is even stronger if adversarial training is added . We will refer to these networks asL2-nonexpansive neural networks ( L2NNNs ) . Our advantage comes from a set of new techniques : our weight regularization , which is key in enforcing the first condition , allows greater degrees of freedom in parameter training than the scheme in Cisse et al . ( 2017 ) ; a new loss function is specially designed for the second condition ; we adapt various layers in new ways for the third condition , for example norm-pooling and two-sided ReLU , which will be presented later . Let us begin with intuitions behind the second and third conditions . Consider a multi-class classifier . Let g ( x ) denote its confidence gap for an input data pointx . If the classifier is a single L2NNN,1 we have a guarantee2 that the classifier will not change its answer as long as the inputx is modified by no more than anL2-norm ofg ( x ) / √ 2 . Therefore maximizing the average confidence gap directly boosts robustness and this motivates the second condition.T explain the third condition , let us introduce the notion of preserving distance : the distance between any pair of input vectors with two different labels ought to be preserved as much as possible atth outputs , while we do not care about the distance between a pair with the same label . Letd ( x1 , x2 ) denote theL2-distance between the output logit-vectors for two input pointsx1 andx2 that have different labels and that are classified correctly . It is straightforward to verify the condition3 of g ( x1 ) + g ( x2 ) ≤ √ 2 · d ( x1 , x2 ) . Therefore a network that maximizes confidence gaps well mustbe one that preserves distance well . Ultimately some distances are preserved while others are lost , and ideally the decision of which distance to lose is made by parameter training rather than by artifacts of network architecture . Hence the third condition involves distance-preserving architecture choices that leave the decision to parameter training as much as possible , and this motivates many of our design decisions such as Sections 2.2 and 2.3 . In practice we employ the strategy of divide and conquer and buil each layer as a nonexpansive map with respect to theL2-norm . It is straightforward to see that a feedforward network composed of nonexpansive layers must implement a nonexpansive map overall . How to adapt subtleties like recursion and splitting-reconvergence is included in the appendix . Besides being robust against adversarial noises , L2NNNs have other desirable properties . They generalize better from noisy training labels than ordinarynetworks : for example , when 75 % of MNIST training labels are randomized , an L2NNN still achieves 93.1 % accuracy on the test set , in contrast to 75.2 % from the best ordinary network . The problem of exploding gradients , which is common in training ordinary networks , is avoided because the gradient of any output with respect to any internal signal is bounded between -1 and 1 . Unlike ordina y networks , the confidence gap of an L2NNN classifier is a quantitatively meaningful indicat on of confidence on individual data points , and the average gap is an indication of generalization . 2 L2-NONEXPANSIVE NEURAL NETWORKS This section describes how to adapt some individual operators in neural networks for L2NNNs . Discussions on splitting-reconvergence , recursion and normalization are in the appendix . 2.1 WEIGHTS . This section covers both the matrix-vector multiplicationn a fully connected layer and the convolution calculation between input tensor and weight tensor in a convolution layer . The convolution calculation can be viewed as a set of vector-matrix multiplications : we make shifted copies of the input tensor and shuffle the copies into a set of small vectorssuch that each vector contains input entries in one tile ; we reshape the weight tensor into a matrix by flattening all but the dimension of the output filters ; then convolution is equivalent to multiplying each of the said small vectors with the flattened weight matrix . Therefore , in both cases , abasic operator isy = Wx . To be a nonexpansive map with respect to theL2-norm , a necessary and sufficient condition is yTy ≤ xTx =⇒ xTWTWx ≤ xTx , ∀x ∈ RN ρ ( WTW ) ≤ 1 ( 1 ) whereρ denotes the spectral radius of a matrix . 1This is only an example and we recommend building a classifier as multiple L2NNNs , see Section 2.4 . 2See Lemma 1 in Appendix D for the proof of the guarantee . 3See Lemma 2 in Appendix D for the proof of the condition . The exact condition of ( 1 ) is difficult to incorporate into training . Instead we use an upper bound:4 ρ ( WTW ) ≤ b ( W ) , min ( r ( WTW ) , r ( WWT ) ) , wherer ( M ) = max i ∑ j |Mi , j | ( 2 ) The above is where our linear and convolution layers differ from those in Cisse et al . ( 2017 ) : they requireWWT to be an identity matrix , and it is straightforward to see that eir scheme is only one special case that makesb ( W ) equal to 1 . Instead of forcing filters to be orthogonal to eachother , our bound ofb ( W ) provides parameter training with greater degrees of freedom . One simple way to use ( 2 ) is replacingW with W ′ = W/ √ b ( W ) in weight multiplications , and this would enforce that the layer is strictly nonexpansive . Another method is described in the appendix . As mentioned , convolution can be viewed as a first layer of making copies and a second layer of vector-matrix multiplications . With the above regularizat on , the multiplication layer is nonexpansive . Hence we only need to ensure that the copying layer is nonexpansive . For filter size ofK1 by K2 and strides ofS1 byS2 , we simply divide the input tensor by a factor of √ ⌈K1/S1⌉ · ⌈K2/S2⌉ . 2.2 RELU AND OTHERS . Let us turn our attention to the third condition from Section1 . ReLU , tanh and sigmoid are nonexpansive but do not preserve distance well . This section presents a method that improves ReLU and is generalizable to other nonlinearities . A different approach to improve sigmoid is in the appendix . To understand the weakness of ReLU , let us consider two inputdata points A and B , and suppose that a ReLU in the network receives two different negative values for A and B and outputs zero for both . Comparing the A-B distance before and after this ReLU layer , there is a distance loss and this particular ReLU contributes to it . We usetwo-sided ReLUwhich is a function fromR to R2 and simply computes ReLU ( x ) and ReLU ( −x ) . Two-sided ReLU has been studied in Shang et al . ( 2016 ) in convolution layers for accuracy improvement . It is straightforward to verify that two-sided ReLU is nonexpansive with respect to anyLp-norm and that it preserves distance in the above scenario . We will empirically verify its effectiveness in increasingconfidence gaps in Section 3 . Two-sided ReLU is a special case of the following general technique . Letf ( x ) be a nonexpansive and monotonically increasing scalar function , and note that ReLU , tanh and sigmoid all fit these conditions . We can define a function fromR to R2 that computesf ( x ) andf ( x ) − x . Such a new function is nonexpansive with respect to anyLp-norm5 and preserves distance better thanf ( x ) alone . 2.3 POOLING . The popular max-pooling is nonexpansive , but does not preserv distance as much as possible . Consider a scenario where the inputs to pooling are activations that represent edge detection , and consider two images A and B such that A contains an edge that passes a particular pooling window while B does not . Inside this window , A has positive values while B has all zeroes . For this window , the A-B distance before pooling is theL2-norm of A ’ s values , yet if max-pooling is used , the A-B distance after pooling becomes the largest of A ’ s values , which can be substantially smaller than the former . Thus we suffer a loss of distance between A and B whilepassing this pooling layer . We replace max-pooling with norm-pooling , which was reported in Boureau et al . ( 2010 ) to occasionally increase accuracy . Instead of taking the max of values inside a pooling window , we take the L2-norm of them . It is straightforward to verify that norm-pooling is nonexpansive6 and would entirely preserve theL2-distance between A and B in the hypothetical scenario above . OtherLp-norms can also be used . We will verify its effectiveness in increasing confidence gaps in Section 3 . If pooling windows overlap , we divide the input tensor by √ K whereK is the maximum number of pooling windows in which an entry can appear , similar to convolution layers discussed earlier . 4The spectral radius of a matrix is no greater than its naturalL∞-norm.WTW andWWT have the same non-zero eigenvalues and hence the same spectral radius . 5See Lemma 4 in Appendix D for the proof of nonexpansiveness . 6See Lemma 5 in Appendix D for the proof of nonexpansiveness .

Summary: The paper presents techniques for training a non expansive network, which keeps the Lipchitz constant of all layers lower than 1. While being non-expansive, means are taken to preserve distance information better than standard networks. The architectural changes required w.r.t standard networks are minor, and the most interesting changes are made to the loss minimized. The main claim of the paper is that the method is robust against adversarial attacks of a certain kind. However, the results presented show that a) such robustness comes at a high cost of accuracy for standard examples, and b) even though the network is preferable to a previous alternative in combating adversarial examples, the accuracy obtained in the face of adversarial attacks is too low to be of practical value. Other properties of the networks, explored empirically, are that the confidence of the prediction is indicative of robustness (to adversarial attacks) and that the networks learn better in the presence of high label noise.