L2-Nonexpansive Neural Networks

This paper proposes a class of well-conditioned neural networks in which a unit amount of change in the inputs causes at most a unit amount of change in the outputs or any of the internal layers . We develop the known methodology of controlling Lipschitz constants to realize its full potential in maximizing robustness , with a new regularization scheme for linear layers , new ways to adapt nonlinearities and a new loss function . With MNIST and CIFAR-10 classifiers , we demonstrate a number of advantages . Without needing any adversarial traning , the proposed classifiers exceed the state of the art in robustness againstwhite-boxL2-bounded adversarial attacks . They generalize better than ordinaryetworks from noisy data with partially random labels . Their outputs are quantit tively meaningful and indicate levels of confidence and generalization , among other desirable properties . 1 INTRODUCTION . Artificial neural networks are often ill-conditioned systems in that a small change in the inputs can cause significant changes in the outputs ( Szegedy et al.,2014 ) . This results in poor robustness and vulnerability under adversarial attacks which has beenr ported on a variety of networks including image classification ( Carlini & Wagner , 2017a ; Goodfellow et al. , 2014 ) , speech recognition ( Kreuk et al. , 2018 ; Alzantot et al. , 2018 ; Carlini & Wagner,2018 ) , image captioning ( Chen et al. , 2017 ) and natural language processing ( Gao et al. , 2018 ; Ebrahimi et al. , 2017 ) . These issues bring up both theoretical questions of how neural networks generalize ( Kawaguchi et al. , 2017 ; Xu & Mannor , 2012 ) and practical concerns of security in applications ( Akhtar & Mian , 2018 ) . A number of remedies have been proposed for these issues and will be discussed in Section 4 . Whitebox defense is particularly difficult and many proposals have failed . For example , Athalye et al . ( 2018 ) reported that out of eight recent defense works , onlyMadry et al . ( 2017 ) survived strong attacks . So far the mainstream and most successful remedy is that of adversarial training ( Madry et al. , 2017 ) . However , as will be shown in Tables 1 and 2 , the robustnes by adversarial training diminishes when a white-box attacker ( Carlini & Wagner , 2017a ) isllowed to use more iterations . This paper explores a different approach and demonstrates that a combination of the following three conditions results in enhanced robustness : 1 ) the Lipschitz onstant of a network from inputs to logits is no greater than 1 with respect to theL2-norm ; 2 ) the loss function explicitly maximizes confidence gap , which is the difference between the largest and second largest logits of a classifier ; 3 ) the network architecture restricts confidence gaps as little as possible . We will elaborate . There are previous works that achieve the first condition ( Cisse et al. , 2017 ; Hein & Andriushchenko , 2017 ) or bound responses to input perturbations by other means ( Kolter & Wong , 2017 ; Raghunathan et al. , 2018 ; Haber & Ruthotto , 2017 ) . For example , Parseval networks ( Cisse et al. , 2017 ) bound the Lipschitz constant by requiring each linear or convolution layer be composed of orthonormal filters . However , the report d obustness and guarantees are often under weak attacks or with low noise magnitude , and none of these works has demonstrated results that are comparable to adversarial training . In contrast , we are able to build MNIST and CIFAR-10 classifiers , without needing any adversarial training , that exceed the state of the art ( Madry et al. , 2017 ) in robustness against white-boxL2bounded adversarial attacks . The defense is even stronger if adversarial training is added . We will refer to these networks asL2-nonexpansive neural networks ( L2NNNs ) . Our advantage comes from a set of new techniques : our weight regularization , which is key in enforcing the first condition , allows greater degrees of freedom in parameter training than the scheme in Cisse et al . ( 2017 ) ; a new loss function is specially designed for the second condition ; we adapt various layers in new ways for the third condition , for example norm-pooling and two-sided ReLU , which will be presented later . Let us begin with intuitions behind the second and third conditions . Consider a multi-class classifier . Let g ( x ) denote its confidence gap for an input data pointx . If the classifier is a single L2NNN,1 we have a guarantee2 that the classifier will not change its answer as long as the inputx is modified by no more than anL2-norm ofg ( x ) / √ 2 . Therefore maximizing the average confidence gap directly boosts robustness and this motivates the second condition.T explain the third condition , let us introduce the notion of preserving distance : the distance between any pair of input vectors with two different labels ought to be preserved as much as possible atth outputs , while we do not care about the distance between a pair with the same label . Letd ( x1 , x2 ) denote theL2-distance between the output logit-vectors for two input pointsx1 andx2 that have different labels and that are classified correctly . It is straightforward to verify the condition3 of g ( x1 ) + g ( x2 ) ≤ √ 2 · d ( x1 , x2 ) . Therefore a network that maximizes confidence gaps well mustbe one that preserves distance well . Ultimately some distances are preserved while others are lost , and ideally the decision of which distance to lose is made by parameter training rather than by artifacts of network architecture . Hence the third condition involves distance-preserving architecture choices that leave the decision to parameter training as much as possible , and this motivates many of our design decisions such as Sections 2.2 and 2.3 . In practice we employ the strategy of divide and conquer and buil each layer as a nonexpansive map with respect to theL2-norm . It is straightforward to see that a feedforward network composed of nonexpansive layers must implement a nonexpansive map overall . How to adapt subtleties like recursion and splitting-reconvergence is included in the appendix . Besides being robust against adversarial noises , L2NNNs have other desirable properties . They generalize better from noisy training labels than ordinarynetworks : for example , when 75 % of MNIST training labels are randomized , an L2NNN still achieves 93.1 % accuracy on the test set , in contrast to 75.2 % from the best ordinary network . The problem of exploding gradients , which is common in training ordinary networks , is avoided because the gradient of any output with respect to any internal signal is bounded between -1 and 1 . Unlike ordina y networks , the confidence gap of an L2NNN classifier is a quantitatively meaningful indicat on of confidence on individual data points , and the average gap is an indication of generalization . 2 L2-NONEXPANSIVE NEURAL NETWORKS This section describes how to adapt some individual operators in neural networks for L2NNNs . Discussions on splitting-reconvergence , recursion and normalization are in the appendix . 2.1 WEIGHTS . This section covers both the matrix-vector multiplicationn a fully connected layer and the convolution calculation between input tensor and weight tensor in a convolution layer . The convolution calculation can be viewed as a set of vector-matrix multiplications : we make shifted copies of the input tensor and shuffle the copies into a set of small vectorssuch that each vector contains input entries in one tile ; we reshape the weight tensor into a matrix by flattening all but the dimension of the output filters ; then convolution is equivalent to multiplying each of the said small vectors with the flattened weight matrix . Therefore , in both cases , abasic operator isy = Wx . To be a nonexpansive map with respect to theL2-norm , a necessary and sufficient condition is yTy ≤ xTx =⇒ xTWTWx ≤ xTx , ∀x ∈ RN ρ ( WTW ) ≤ 1 ( 1 ) whereρ denotes the spectral radius of a matrix . 1This is only an example and we recommend building a classifier as multiple L2NNNs , see Section 2.4 . 2See Lemma 1 in Appendix D for the proof of the guarantee . 3See Lemma 2 in Appendix D for the proof of the condition . The exact condition of ( 1 ) is difficult to incorporate into training . Instead we use an upper bound:4 ρ ( WTW ) ≤ b ( W ) , min ( r ( WTW ) , r ( WWT ) ) , wherer ( M ) = max i ∑ j |Mi , j | ( 2 ) The above is where our linear and convolution layers differ from those in Cisse et al . ( 2017 ) : they requireWWT to be an identity matrix , and it is straightforward to see that eir scheme is only one special case that makesb ( W ) equal to 1 . Instead of forcing filters to be orthogonal to eachother , our bound ofb ( W ) provides parameter training with greater degrees of freedom . One simple way to use ( 2 ) is replacingW with W ′ = W/ √ b ( W ) in weight multiplications , and this would enforce that the layer is strictly nonexpansive . Another method is described in the appendix . As mentioned , convolution can be viewed as a first layer of making copies and a second layer of vector-matrix multiplications . With the above regularizat on , the multiplication layer is nonexpansive . Hence we only need to ensure that the copying layer is nonexpansive . For filter size ofK1 by K2 and strides ofS1 byS2 , we simply divide the input tensor by a factor of √ ⌈K1/S1⌉ · ⌈K2/S2⌉ . 2.2 RELU AND OTHERS . Let us turn our attention to the third condition from Section1 . ReLU , tanh and sigmoid are nonexpansive but do not preserve distance well . This section presents a method that improves ReLU and is generalizable to other nonlinearities . A different approach to improve sigmoid is in the appendix . To understand the weakness of ReLU , let us consider two inputdata points A and B , and suppose that a ReLU in the network receives two different negative values for A and B and outputs zero for both . Comparing the A-B distance before and after this ReLU layer , there is a distance loss and this particular ReLU contributes to it . We usetwo-sided ReLUwhich is a function fromR to R2 and simply computes ReLU ( x ) and ReLU ( −x ) . Two-sided ReLU has been studied in Shang et al . ( 2016 ) in convolution layers for accuracy improvement . It is straightforward to verify that two-sided ReLU is nonexpansive with respect to anyLp-norm and that it preserves distance in the above scenario . We will empirically verify its effectiveness in increasingconfidence gaps in Section 3 . Two-sided ReLU is a special case of the following general technique . Letf ( x ) be a nonexpansive and monotonically increasing scalar function , and note that ReLU , tanh and sigmoid all fit these conditions . We can define a function fromR to R2 that computesf ( x ) andf ( x ) − x . Such a new function is nonexpansive with respect to anyLp-norm5 and preserves distance better thanf ( x ) alone . 2.3 POOLING . The popular max-pooling is nonexpansive , but does not preserv distance as much as possible . Consider a scenario where the inputs to pooling are activations that represent edge detection , and consider two images A and B such that A contains an edge that passes a particular pooling window while B does not . Inside this window , A has positive values while B has all zeroes . For this window , the A-B distance before pooling is theL2-norm of A ’ s values , yet if max-pooling is used , the A-B distance after pooling becomes the largest of A ’ s values , which can be substantially smaller than the former . Thus we suffer a loss of distance between A and B whilepassing this pooling layer . We replace max-pooling with norm-pooling , which was reported in Boureau et al . ( 2010 ) to occasionally increase accuracy . Instead of taking the max of values inside a pooling window , we take the L2-norm of them . It is straightforward to verify that norm-pooling is nonexpansive6 and would entirely preserve theL2-distance between A and B in the hypothetical scenario above . OtherLp-norms can also be used . We will verify its effectiveness in increasing confidence gaps in Section 3 . If pooling windows overlap , we divide the input tensor by √ K whereK is the maximum number of pooling windows in which an entry can appear , similar to convolution layers discussed earlier . 4The spectral radius of a matrix is no greater than its naturalL∞-norm.WTW andWWT have the same non-zero eigenvalues and hence the same spectral radius . 5See Lemma 4 in Appendix D for the proof of nonexpansiveness . 6See Lemma 5 in Appendix D for the proof of nonexpansiveness .

Summary: This paper presents a combination of methods that, together, yield neural networks that are robust to small changes in L2 distance. The main idea is to ensure that changing the input by a bounded L2 distance never changes the output by more than the same L2 distance. Then, the difference between the highest-scoring class and the second-highest scoring class provides a bound on how much the input must change. The trivial way to do this is to rescale the final output layer so that all of the magnitudes are very small; however, this would give no additional robustness at all. To counteract this, the paper introduces several additional heuristics for increasing the gap between the highest-scoring class and the second-highest scoring one. Adversarial training can be used to make the models even more robust.