Clustering Effect of Adversarial Robust Models

1 Introduction . Nowadays , deep neural networks ( DNNs ) have shown a strong learning capacity through a huge number of parameters and diverse structures [ 16 , 31 , 11 ] . Meanwhile , adversary has raised increasing security concerns on DNNs due to the observation of adversarial examples ( i.e. , the examples mislead a classifier when crafted with human imperceptible but carefully designed perturbations ) [ 13 ] . Adversarial robustness and defense techniques have thus become crucial for deep learning , yet many adversarial defense techniques are found with deficiencies such as obfuscated gradient [ 2 ] . Up to now , the widely accepted techniques to improve adversarial robustness are adversarial training variants [ 23 , 32 , 38 , 33 , 35 , 36 ] , by training a DNN after data augmentation with the worst-case adversarial examples . Recent study has found that not only do robust models show moderate robustness under vast attacks , but also they can surprisingly boost the downstream tasks ( e.g. , domain adaption task with subpopulation shift ) [ 29 ] . However , the underlying mechanism of adversarial robustness is still not clear . Existing studies have explored adversary on some specific components of DNNs , such as batch normalization [ 12 ] , skip connection [ 34 ] , or activation layers [ 14 , 4 ] , which yield some enlightening understanding . On the other hand , despite variation in DNNs ’ structure , the components can be roughly divided into linear and non-linear ones . The non-linear components tend to be instance-wise . For example , adversarial * Equal contribution . †Correspondence to : Yisen Wang ( yisen.wang @ pku.edu.cn ) and Shu-Tao Xia ( xiast @ sz.tsinghua.edu.cn ) . 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . and benign examples tend to present different patterns on activation [ 4 ] . In contrast , for those linear components , once the optimization is done and model evaluation mode is activated , they are shared and fixed independently from inputs . Nowadays , the intrinsic relationship of adversary with linearity has been studied . Some ascribes the adversary to linear accumulation of perturbations from inputs to final outputs , dubbed ‘ accidental steganography ’ [ 13 ] . Some show that linear backward propagation enhances adversarial transferability [ 34 , 14 ] . Nevertheless , such mentioned linearity analyses are all on the original non-linear models , which hinders a deeper understanding on linearity directly . Further essential exploration is encouraged . In this paper , we thus study adversarial robustness on the statistical regularity of linear components . We provide a novel insight on adversarial robustness by showing the clustering effect well aligned with class hierarchy . As shown in Fig . 1 ( d ) , given a data set , superclasses are defined by grouping semantically similar subclasses together to form a hierarchy , e.g. , in CIFAR-10 , ‘ Cat ’ and ‘ Dog ’ represent different subclasses/fine labels , despite both being the same superclasses/coarse labels ‘ Animal ’ . Based on this , we conduct backward propagation on linearized sub-networks of robust and non-robust DNNs ( through adversarial/standard training respectively ) to extract the corresponding implicit linear expression . In this fashion , we could obtain a Dinput⇥Doutput linear matrix W , which is extracted from inputs to predictive outputs . Dinput represents for the dimension of one input data , specifically defined as width ⇥ height ⇥ channel . Doutput represents for the dimension of one output vector , which is also the number of classes . We further study the correlation of vectors in W by matrix C , whose shape is Doutput ⇥Doutput . The details of W and C are introduced in Sec . 3.2 . Then we find that robust models tend to show hierarchical clustering effect in correlation matrix C , which is surprisingly consistent with class hierarchy . For example , in Fig . 1 on CIFAR-10 , the fine labels ( 0,1,8,9 ) belong to ‘ Non-animal ’ , others belong to ‘ Animal ’ . The matrix C of robust model shows block clustering : values are close to ‘ +1 ’ in the two superclasses ( 0,1,8,9 ) and ( 2,3,4,5,6,7 ) , and values are close to ‘ -1 ’ across these two superclasses . Such clustering effect is enhanced by our clustering regularization penalty in Fig . 1 ( d ) . This phenomenon could be semantically explained as subclasses from the same superclass share more feature similarities . Such hierarchical clustering effect can give a novel and insightful explanation on the superiority of robust models in some observed downstream tasks , as they can extract more semantic and representative features . To further confirm our findings , we enhance the hierarchical clustering effect in both adversarial robustness and downstream tasks , e.g. , domain adaption proposed in [ 29 , 28 ] . The improvements of experimental results demonstrate that the superiority of robust models is closely related to their hierarchical classification capability . Our contributions are summarized as follows : • To the best of our knowledge , we are the first to systematically analyze the statistical regularity of adversarially robust models ( through adversarial training ) compared to nonrobust models ( through standard training ) on their linearized sub-networks . • We present an intriguing phenomenon of hierarchical clustering effect in robust models , and provide a novel yet insightful understanding of adversarial robustness . The clustering effect aligned with class hierarchy demonstrates more semantic and representative feature extraction capacity of robust models , which benefits a lot in various tasks . • Based on the observations , we propose a plugged-in hierarchical clustering training strategy to generally enhance adversarial robustness and investigate some intriguing adversarial attack findings . Besides adversarial-related study , we further explore some downstream tasks with the understanding of hierarchical clustering , e.g. , domain adaption with subpopulation shift . Experimental results show that the clustering effect and hierarchical classification learned by robust model benefits the task as well . 2 Related Work . 2.1 Adversarial Attack and Defense . Attack Given a clean example x , its true label y , a DNN model F with parameters ✓ and its loss function L , the goal of adversarial attack is to find an adversarial example x0 that is close to clean example on pixel level but can fool F to give an incorrect prediction . Fast Gradient Sign Method ( FGSM ) [ 13 ] adds perturbations on clean example x by one step with step size ✏ : x0 = x+ ✏sign ( rxL ( F ( x , ✓ ) , y ) ) . ( 1 ) Projected Gradient Descent ( PGD ) [ 23 ] adds perturbations on clean example x by K steps with smaller step size ↵ . After every step ( k-th ) attack , adversarial example is projected into the ✏-ball B✏ ( x ) around clean example ( function ⇧ ) : xk = ⇧ ( xk 1 + ↵sign ( rxL ( F ( xk 1 , ✓ ) , y ) ) ) . ( 2 ) There are other attack techniques , such as white-box CW attack [ 6 ] , black-box attacks [ 21 , 3 ] and adaptive AutoAttack [ 9 ] . Defense Many adversarial defense techniques have been proposed since then , such as input preprocessing [ 19 ] , defensive distillation [ 24 ] , model compression [ 10 ] , and adversarial training [ 23 ] . Among them , adversarial training variants are assumed to be most effective in comprehensive attack evaluations . They solve a min-max problem as : min ✓ max x02B✏ ( x ) L ( F ( x0 , ✓ ) , y ) . ( 3 ) The inner maximum problem often generates adversarial examples by FGSM or PGD , while the outer minimum problem optimizes the worst-case loss . Some defense technique improves robustness through class-wise feature clustering [ 1 , 30 ] , which is different from our instance-wise feature clustering . 2.2 Linearity Exploration in Adversary . Beside attack or defense techniques , the cause and understanding of adversary are studied . The linearity is one essential perspective with no doubt . Previous studies attribute the existing of adversarial examples to the linearity of DNNs [ 13 ] and observe the adversarial transferability enhancement on more linear models [ 34 , 14 ] . Some works study linearity from another perspective by focusing on the properties of weight layers , e.g. , their norms , variances and orthogonality . Lp norm regularization works by pulling examples far away from the decision boundary , which comes as the side-effect of adversarial robustness [ 37 , 18 ] . Spectral norm [ 7 , 26 ] is useful in adversarial defense , which is defined as i = kwixk2 kxk2 and is used to constrain the Lipschiz constant of a DNN F on x ( assuming Lipschiz continuous ) kF ( x+ ) F ( x ) k2 kxk2 Q ikxk2 . The weight scale shifting issue is also discussed in adversary [ 22 ] , that is , scale of weights can be shifted between layers without changing the input-output function specified by the network , which could affect the capacity to regularize models . Then one weight scale shift invariant regularization is proposed and improves adversarial robustness . Moreover , the orthogonality could help improve generalization and adversarial robustness [ 5 , 7 ] by inducing uncorrelated features . However , these experiments are all conducted on the original non-linear models instead of the linear components directly , which hinders a deeper understanding on linearity . 3 Observations on Linear Components . The robust models perform completely differently from non-robust ones especially under comprehensive adversarial attacks . However , the statistical regularity on linear components of these robust models is yet under little exploration . Different from related works , we explore linear components directly . To be specific , we extract a weight matrix expression of linearized sub-networks , estimating the linear propagation from inputs to outputs . 3.1 Extracting a Linear Weight Matrix . Given a DNN F composed of L linear layers and non-linear components , the output y = F ( x ) could be expressed as y = gL ( wL ⇥ gL 1 ( wL 1 ⇥ . . . g1 ( w1 ⇥ x+ b1 ) · · ·+ bL 1 ) + bL ) , ( 4 ) where gi is a series of non-linear components ( e.g. , activation function ReLUs ) , wi and bi are the expressions of linear ones . Then a corresponding linear sub-network of this original one could be extracted with the same weights and architectures , yet it removes activation function ( e.g. , ReLU layers ) and batch normalization layers , and further replaces maximum pooling layers with average pooling layers . This output of linear network ylinear = Flinear ( x ) is expressed as ylinear = wL ⇥ ( wL 1 ⇥ ... ( w1 ⇥ x+ b1 ) ... + bL 1 ) + bL . ( 5 ) Thus in Flinear , each input x is multiplied by an instance-agnostic total weight W = wL ⇥ wL 1 ⇥ ... w1 + X i , j bi ... ⇥ wj .... ( 6 ) However , such total weight expression W in DNNs is difficult to compute directly , because the multiplication of convolution layers is infeasible . Instead , we propose to compute W by applying backward propagation on the linear sub-network . To be specific , given a random input x and a pre-trained DNN F , W is computed as the gradient propagated on linear sub-network Flinear following Algorithm 1 . For a non-linear model , as the activation part is example-wise , it is difficult to analyse the original network with non-linear components . In contrast , the linear component is generally applied on all examples and example-agnostic . So we extract a linear expression from the original non-linear model to approximate and analyse the model performance . Intuitively , although the linearized network has limited capacity , we think that the class-wise directions ( the extracted linearized weight vectors ) should still represent some class-wise directions , which holds a connection with the feature space of the non-linear models . Algorithm 1 Extracting the Linear Weight Matrix W Input : A random input x , a pre-trained DNN model F with Doutput classes Output : Weight matrix W Get the corresponding linear sub-network Flinear from F Conduct forward propagation as : ylinear = Flinear ( x ) for i in range ( Doutput ) do ylinear [ i ] .backward ( ) W [ : , i ] = x.grad end for return W

Summary: This paper introduces the "clustering effect" of linear components of adversarial robust models. To be specific, the authors propose to remove the non-linear functions in networks and further integrate all linear components into a single weight matrix W in order to directly study the effect of linearity in adversarial models. By conducting experiments with W, the authors find that adversarially trained models tend to show clustering effects in both weight and feature dimensions. This effect further benefits some downstream tasks such as adversarial attack and transfer learning and can be enhanced with additional supervision. In all, the phenomenon showed in this paper is interesting and deserves more and deeper research.