Towards Understanding the Regularization of Adversarial Robustness on Neural Networks

1 INTRODUCTION . Despite the remarkable performance ( Krizhevsky et al. , 2012 ) of Deep Neural Networks ( NNs ) , they are found to be rather fragile and easily fooled by adversarial examples ( Szegedy et al. , 2014 ) . More intriguingly , these adversarial examples are generated by adding imperceptible noise to normal examples , and thus are indistinguishable for humans . NNs that are more robust to adversarial examples tend to have lower standard accuracy ( Su et al. , 2018 ) , i.e. , the accuracy measured on natural examples . The trade-off between robustness and accuracy has been observed ( Kurakin et al. , 2017 ; Madry et al. , 2018 ; Tsipras et al. , 2019 ) . To understand such a phenomenon , Tsipras et al . ( 2019 ) show that for linear models , if examples are closed to decision boundaries , robustness provably conflicts with accuracy , though the proof seems unlikely to generalize to NNs . Zhang et al . ( 2019 ) show that a gap exists between surrogate risk gap and 0-1 risk gap if many examples are close to decision boundaries , and better robustness can be achieved by pushing examples away from decision boundaries . But pushing examples away again degrades NN performance in their experiments . A more established remedy is developed to require NNs to be -adversarially robust ( AR ) , e.g. , via Adversarial Training ( Madry et al. , 2018 ) , Lipschitz-Margin Training ( Tsuzuku et al. , 2018 ) ; that is , they require the model not to change predicted labels when any given input examples are perturbed within a certain range . Note that such hard requirement is different from penalties on the risk function employed by Lyu et al . ( 2015 ) and Miyato et al . ( 2018 ) , which is not our subject of investigation ( more discussion in appendix A ) . In practice , hard-requirement methods are found to lead to worse performance measured in standard classification accuracy . We aim to study this branch of methods . We investigate how adversarial robustness influence the behaviors of NNs to make them more robust but have lower performance . In an earlier time ( Szegedy et al. , 2014 ) , adversarial training has been suggested as a form of regularization : it augments the training of NNs with adversarial examples , thus might improve the generalization of the end models . How does a possible improvement in generalization end up degrading performance ? It prompts us to analyze the regularization effects of AR on NNs . A successful regularization technique is expected to improve test performance , but an improved performance is only one of the possible outcomes of improved generalization . Technically , improved generalization implies the reduction in gap between training errors and test errors . Regularization achieves the gap reduction by reducing the size of the hypothesis space , which reduces the variance , but meanwhile increases the bias of prediction made — a constant classifier can have zero generalization errors , but also have low test performance . Thus , when a hypothesis space is improperly reduced , another possible outcome is biased poorly performing models with reduced generalization gaps . Key results . Through a series of theoretically motivated experiments , we find that AR is achieved by regularizing/biasing NNs towards less confident solutions by making the changes in the feature space of most layers ( which are induced by changes in the instance space ) smoother uniformly in all directions ; so to a certain extent , it prevents sudden change in prediction w.r.t . perturbations . However , the end result of such smoothing concentrates examples around decision boundaries and leads to worse standard performance . We elaborate the above statement in details shortly in section 1.1 . Overall , the investigation of generalization behaviors of NNs points out possible directions where we might go if we are to resolve the issue of the test performance degradation done by AR . The main result shows that the hypothesis space of NNs is improperly reduced , thus we might investigate how to avoid it when enforcing AR . Though beyond the scope of this work , we conjecture that the improper reduction comes from the indistinguishability of the change induced in the intermediate layers of NNs by adversarial noise and that by inter-class difference . To guarantee AR , NNs are asked to smoothe out difference uniformly in all directions in a high dimensional space , and thus are biased towards diffident solutions that make similar/concentrated predictions . We leave the investigation of the conjecture as future works . 1.1 AR LEADS TO DIFFIDENT NNS WITH MORE INDECISIVE MISCLASSIFICATIONS . This section elaborates the key results we briefly present previously . 1 . AR reduces the variance of ( norms of ) the activation/outputs ( compared with NNs with different . AR strength ) at most layers that are emitted/induced by feeding perturbations ( of any directions ) to that layer from the previous layer . Through a series of theoretically motivated experiments , the results prompt us to look at the singular value distributions of the weight matrix of each layer of the NNs . Shown in fig . 1a , we find that overall the standard deviation ( STD ) of singular values associated with a layer of the NN trained with lower AR strength 4 is larger than that of the NN with higher AR strength 161 — the green dots are mostly below the red dots . Note that given a matrix W and an example x , singular values of W determine how the norm ||Wx|| is changed 1The AR strength is characterized by the maximally allowed l∞ norm of adversarial examples that are used to train the NNs — we use adversarial training ( Madry et al. , 2018 ) to build adversarial robustness into NNs . Details can be found in appendix B.1 when compared with ||x|| . More specifically , let σmin , σmax be the maximal and minimal singular values , if x is not in the null space of W , then we have ||Wx|| ∈ [ σmin||x|| , σmax||x|| ] , where || · || denotes 2-norm . This applies to norm ||δx|| of a perturbation as well ; that is , given possible changes δx of x of the same norm ||δx|| = c , where c is a constant , the variance of σ ( W ) roughly determines the variance of ||W δx|| , where σ ( W ) denotes all singular values { σi } of W . In more details , note that by SVD decomposition , W δx = ∑ i σiuiv T i δx , thus σi determines how the component vTi δx in the direction of vi is amplified . To see an example , suppose that σmin = σmax = σ0 , then the variance of σ ( W ) is zero , and ||W δx|| = σ0||δx|| . In this case , the variance of ||W δx|| ( given an ensemble of perturbations δx of the same norm c ) is zero as well . The conclusion holds as well for ReLU ( W δx ) , where W here is a weight matrix of a layer of a NN , and ReLU denotes Rectifier Linear Unit activation function ( proved by applying Cauchy interlacing law by row deletion ( Chafai ) to lemma 4.1 ) . Consequently , by reducing the variance of singular values of weight matrix of a layer of the NN , AR reduces the norm variance of layer activations induced by input perturbations . 2 . The reduced norm variance induced by example perturbations concentrates examples , and it empirically concentrates them around decision boundaries ; that is , predictions are more diffident . The reduced variance implies that the outputs of each layer of the NN are more concentrated , but it does not tell where they are concentrated . Note that in the previous paragraph , the variance relationship discussed between ||W δx|| and ||δx|| equally applies to ||Wx|| and ||x|| , where x is an actual example instead of perturbations . Thus , to find out the concentration of perturbations , we can look at the concentration of samples . Technically , we look at margins of examples . In a multi-class setting , suppose a NN computes a score function f : Rd → RL , where L is the number of classes ; a way to convert this to a classifier is to select the output coordinate with the largest magnitude , meaning x 7→ argmaxi fi ( x ) . The confidence of such a classifier could be quantified by margins . It measures the gap between the output for the correct label and other labels , meaning fy ( x ) −maxi 6=y fi ( x ) . Margin piece-wise linearly depends on the scores , thus the variance of margins is also in a piece-wise linear relationship with the variance of the scores , which are computed linearly from the activation of a NN layer . Thus , the consequence of concentration of activation discussed in the previous paragraph can be observed in the distribution of margins . More details of the connection between singular values and margins are discussed in section 5.2.2 , after we present lemma 4.1 . A zero margin implies that a classifier has equal propensity to classify an example to two classes , and the example is on the decision boundary . We plot the margin distribution of the test set of CIFAR10 in fig . 1b , and find that margins are increasingly concentrated around zero — that is , the decision boundaries — as AR strength grows . 3 . The sample concentration around decision boundaries smoothes sudden changes induced perturbations , but also increases indecisive misclassification . The concentration of test set margins implies that the induced change in margins by the perturbation in the instance space is reduced by AR . The statement may not be immediately obvious , so we explain in details as follows . Given two examples x , x′ from the test set , δx = x− x′ can be taken as a significant perturbation that changes the example x to x′ . The concentration of overall margins implies the change induced by δx is smaller statistically in NNs with higher AR strength . Thus , for an adversarial perturbation applied on x , statistically the change of margins is smaller as well — experimentally it is reflected in the increased adversarial robustness of the network , as shown in the increasing curve in fig . 1c . That is , the sudden changes of margins originally induced by adversarial perturbations are smoothed ( to change slowly ) . However , the cost of such smoothness is lower confidence in prediction , and more test examples are slightly/indecisively moved to the wrong sides of the decision boundaries — incurring lower accuracy , as shown in the decreasing curve in fig . 1c . Lastly , we note that experiments in this section are used to illustrate our main arguments in this section . Further consistent quality results are reported in section 5 by conducting experiments on CIFAR10/100 and Tiny-ImageNet with networks of varied capacity . 1.2 OUTLINE AND CONTRIBUTIONS . As briefly discussed at the beginning , this work carries out generalization analysis on NNs with AR . The quantities we investigate in the previous section are identified by the generalization errors ( GE ) upper bound we establish at theorem 4.1 , which characterizes the regularization of AR on NNs . The key result is actually obtained at the end of a series of analysis , thus we present the outline of the analysis here . Outline . After presenting some preliminaries in section 3 , we proceed to analyze the regularization of AR on NNs , and establish a GE upper bound in section 4 . The bound prompts us to look at the GE gaps in experiments . In section 5.1 , we find that for NNs trained with higher AR strength , the surrogate risk gaps ( GE gaps ) decrease for a range of datasets , i.e. , CIFAR10/100 and Tiny-ImageNet ( ImageNet , 2018 ) . It implies AR effectively regularizes NNs . We go further to study the finer behavior change of NNs that might lead to such a gap reduction . Again , we follow the guidance of theorem 4.1 . We look at the margins in section 5.2.1 , then at the singular value distribution in section 5.2.2 , and discover the main results described in section 1.1 . More corroborative experiments are run in appendix B.4 to show that such phenomenon exists in a broad range of NNs with varied capacity , and more complementary results are present in appendix B.3 to explain some seemingly abnormal observations . More related works are present in section 2 . Contributions . Overall , the core contribution in this work is to show that adversarial robustness ( AR ) regularizes NNs in a way that hurts its capacity to learn to perform in test . More specifically : • We establish a generalization error ( GE ) bound that characterizes the regularization of AR on NNs . The bound connects margin with adversarial robustness radius via singular values of weight matrices of NNs , thus suggesting the two quantities that guide us to investigate the regularization effects of AR empirically . • Our empirical analysis tells that AR effectively regularizes NNs to reduce the GE gaps . To understand how reduced GE gaps turns out to degrade test performance , we study variance of singular values of layer-wise weight matrices of NNs and distributions of margins of samples , when different strength of AR are applied on NNs . • The study shows that AR is achieved by regularizing/biasing NNs towards less confident solutions by making the changes in the feature space of most layers ( which are induced by changes in the instance space ) smoother uniformly in all directions ; so to a certain extent , it prevents sudden change in prediction w.r.t . perturbations . However , the end result of such smoothing concentrates samples around decision boundaries and leads to worse standard performance .

Summary: The paper presents new theory to develop understanding about why adversarially robust neural networks show lower test performance compared to their standard counterparts despite being more robust to perturbations in the data. The main hypothesis is that the degradation in performance in adversarially robust networks is due to many samples being concentrated around the decision boundary, which makes the network less confident about its decisions. The paper studies this hypothesis by deriving a bound on the generalization error based on the margin between the samples in the training set and the decision boundary. The paper then presents empirical demonstrations that aim to illustrate the theoretical findings.