Towards a Unified Game-Theoretic View of Adversarial Perturbations and Robustness

This paper provides a unified view to explain different adversarial attacks and defense methods , i.e . the view of multi-order interactions between input variables of DNNs . Based on the multi-order interaction , we discover that adversarial attacks mainly affect high-order interactions to fool the DNN . Furthermore , we find that the robustness of adversarially trained DNNs comes from category-specific low-order interactions . Our findings provide a potential method to unify adversarial perturbations and robustness , which can explain the existing defense methods in a principle way . Besides , our findings also make a revision of previous inaccurate understanding of the shape bias of adversarially learned features . Our code is available online at https : //github.com/Jie-Ren/ A-Unified-Game-Theoretic-Interpretation-of-Adversarial-Robustness . 1 Introduction . Adversarial robustness of deep neural networks ( DNNs ) has received increasing attention in recent years . Related studies include adversarial defense and attacks [ 48 , 21 ] . In terms of defense , adversarial training is an effective and the most widely-used method [ 34 , 72 , 54 , 55 , 60 ] . In spite of their fast development , the essential mechanism of the adversarial robustness is still unclear . Thus , the understanding of adversarial attacks and defense is an emerging direction in recent years . Ilyas et al . [ 29 ] demonstrated adversarial examples could be attributed to the presence of non-robust yet discriminative features . Some methods [ 20 , 58 ] explored the mathematical bound for the model robustness . Zhang and Zhu [ 73 ] , Tsipras et al . [ 50 ] found adversarial training helped DNNs learn a more interpretable ( more shape-biased ) representation . Besides the feature interpretability , Tsipras et al . [ 50 ] further showed an inherent tension between the adversarial robustness and the generalization power . Unlike above perspectives for explanations , we aim to propose a unified view to explain the essential reason why and how adversarial examples emerge , as well as essential mechanisms of various adversarial defense methods . We rethink the adversarial robustness from the novel perspective of ∗Equal contribution †This work was done when Meng Zhou was an undergraduate at Shanghai Jiao Tong University . ‡Quanshi Zhang is the corresponding author . zqs1022 @ sjtu.edu.cn . This work is supervised by Dr. Quanshi Zhang . He is with the John Hopcroft Center and the MoE Key Lab of Artificial Intelligence , AI Institute , at the Shanghai Jiao Tong University , China . 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . interactions between input variables of a DNN . It is because in an adversarial example , adversarial perturbations on different pixels do not attack the DNN independently . Instead , perturbation pixels usually interact with each other to form a specific pattern for attacking . Surprisingly , we find that such interactions can explain various aspects of adversarial robustness . Specifically , this study aims to answer the following three questions . • How to disentangle feature representations that are sensitive to adversarial perturbations . Based on the game-theoretic interactions , we aim to summarize the distinct property of feature representations , which are sensitive to adversarial perturbations , among overall feature representations . • How to explain the effectiveness of the adversarial training . The above summarized property of sensitive feature representations also provides a new perspective to explain the utility of the adversarial training , i.e . why and how the adversarial training penalizes such sensitive feature representations . • How to unify various adversarial defense methods in a single theoretic system . Our research provides a unified understanding for the success of adversarial defense methods [ 65 , 15 , 30 ] . As a prerequisite of analyzing the adversarial robustness , let us first revisit the interaction between input variables in DNNs . Let a set of input variables collaborate with each other to form an inference pattern , the Shapley interaction index [ 22 ] is a standard metric to measure the numerical benefits to the inference from their collaborations . This metric can be further extended to the multi-order interaction [ 70 ] . For the interaction between two input variables ( i , j ) , the interaction order measures the number of contextual variables that influence the significance of the interaction between i and j . In other words , low-order interactions represent simple collaborations between input variables with small contexts , while high-order interactions indicate complex collaborations over large contexts ( see Figure 1 ( right ) ) . We further prove that the network output can be decomposed into the sum of multi-order interactions between different pairs of input variables . Thus , the overall effects of adversarial perturbations on the network output can be decomposed into elementary effects on different interaction components . Therefore , we can explain adversarial robustness using such elementary interaction components . ( 1 ) We discover and partially prove that adversarial perturbations mainly affect high-order interactions , rather than low-order interactions ( see Figure 1 ( left ) ) . In comparison , low-order interactions are naturally robust to attacks . In other words , adversarial attacks mainly affect the complex and large-scale collaborations among most pixels in the image . Based on this , we can successfully disentangle sensitive feature representations , i.e . high-order interactions . Interaction vs. frequency & rank . Some studies explained adversarial examples as high-frequency features [ 66 , 51 , 23 ] and high-rank features [ 30 ] . We have conducted experiments to show that high-order interactions can better explain the essential property of attacking-sensitive representations , i.e . the complex and large-scale visual concepts . ( 2 ) A clear difference between standard DNNs and adversarially trained DNNs is as follows . Adversarial training significantly increases the robustness of high-order interactions . In other words , attacks mainly affect complex collaborations in standard DNNs , while for adversarially trained DNNs , complex collaborations are not so vulnerable w.r.t . simple collaborations . Then , we further explain the reason for the high robustness of adversarially trained DNNs . For interactions of each order , we define the disentanglement metric to identify whether interactions of this order are discriminative for the classification of a specific category , or represent common knowledge shared by different categories . For example , in Figure 1 ( left ) , interactions representing the blue water may be shared by different categories , while interactions corresponding to the head of the red-breasted merganser are discriminative for this category . Based on the disentanglement metric , we discover that compared with standard DNNs , adversarially trained DNNs usually encode more discriminative low-order interactions . Discriminative low-order interactions make high-order interactions of adversarially trained DNNs robust to attacks , because contexts of high-order interactions are composed of many small contexts of low-order interactions . For example , if the simple ( low-order ) interactions for the red-breasted merganser are learned to be discriminative , instead of being shared by the bicycle category , then it is difficult to attack this image towards the bicycle category . It is because it is difficult to use the low-order interactions of the red-breasted merganser ’ s head to construct high-order interactions of bicycles . ( 3 ) Our research provides a unified understanding for the success of several existing adversarial defense methods , including the attribution-based detection of adversarial examples [ 65 ] , the recoverability of adversarial examples to normal samples , the cutout method [ 15 ] , and the rank-based method [ 30 ] ( which is proved by [ 30 ] to be related to frequency-based methods [ 66 , 51 , 61 ] ) . Above findings also slightly revise the previous explanation of adversarially learned features [ 21 , 50 , 16 , 73 ] . They claimed that adversarial training learned more information about foreground shapes . We discover that these adversarially learned features are actually low-order interactions ( usually local shapes ) , instead of modeling the global shape of the foreground . Explainable AI system based on game-theoretic interactions . In fact , our research group led by Dr. Quanshi Zhang have proposed game-theoretic interactions , including interactions of different orders [ 69 ] and multivariate interactions [ 71 ] . The interaction can be used as an typical metric to explain signal processing in DNNs from different perspectives . For example , the game-theoretic metric can be used to guide the learning of baseline values of Shapley values [ 41 ] . Furthermore , we have built up a tree structure to explain the hierarchical interactions between words encoded by NLP models [ 68 ] . We have also used interactions to explain the generalization power of DNNs [ 70 ] . The interaction can also explain how adversarial perturbations contribute to the attacking task [ 53 ] , and explain the transferability of adversarial perturbations [ 52 ] . Furthermore , we have also used the interaction to formulate the visual aesthetics [ 12 ] and signal-processing properties of different types of visual concepts [ 11 ] in DNNs . As an extension of the system of game-theoretic interactions , in this study , we explain the adversarial robustness based on interactions . 2 Related work . Attacking methods can be roughly summarized into white-box attacks [ 48 , 21 , 31 , 38 , 8 ] and blackbox attacks [ 32 , 39 , 10 , 6 , 28 , 59 , 4 ] . For defense , adversarial training is one of the most effective and widely-used defense methods [ 34 , 72 , 54 , 55 , 60 ] . Other defense methods include masking gradients [ 39 , 36 ] , modifying networks [ 13 , 19 ] , and applying pre-processing on input images for testing [ 14 , 35 , 62 , 3 ] . Explanations for adversarial examples . Some previous studies focused on the reason for the existence of adversarial examples . Goodfellow et al . [ 21 ] explained adversarial examples as a result of the high linearity of feature representations . Gilmer et al . [ 20 ] , Ma et al . [ 33 ] proved that the existence of adversarial examples was due to the geometry of the high-dimensional manifold . Xie et al . [ 62 ] , Xu et al . [ 63 , 64 ] , Bai et al . [ 5 ] discovered that adversarial perturbations usually activated substantial “ noise ” and semantically irrelevant features . Engstrom et al . [ 17 ] investigated the vulnerability of DNNs to rotations and translations . Tsipras et al . [ 50 ] , Ilyas et al . [ 29 ] demonstrated that adversarial examples were attributed to non-robust yet discriminative features . Understandings of adversarial training . Athalye et al . [ 2 ] suggested that adversarial training did not cause the obfuscated gradients phenomenon , which boosted the robustness . Goodfellow et al . [ 21 ] , Dong et al . [ 16 ] , Tsipras et al . [ 50 ] , Zhang and Zhu [ 73 ] found that adversarially trained DNNs learned more shape-biased features than standard DNNs . Chalasani et al . [ 9 ] proved that adversarially trained DNNs w.r.t . the ℓ∞ attack exhibited more sparse attributions . Tian et al . [ 49 ] showed that there is robustness imbalance among classes in adversarial training . Song et al . [ 46 ] considered the adversarial training as the enumeration of all potential adversarial perturbations . Wang et al . [ 54 ] explained adversarial training from the perspective of min-max optimization . Yin et al . [ 66 ] , Wang et al . [ 56 ] , Harder et al . [ 23 ] discovered adversarial training pushed DNNs to utilize low-frequency components in inputs . Understanding of the robustness . Szegedy et al . [ 48 ] , Hein and Andriushchenko [ 25 ] computed Lipschitz constant to explain the robustness . Ignatiev et al . [ 27 ] , Boopathy et al . [ 7 ] investigated the connection between network interpretability and adversarial robustness . Tsipras et al . [ 50 ] proved the inherent tension between adversarial robustness and standard generalization power . Fawzi et al . [ 18 ] , Gilmer et al . [ 20 ] , Weng et al . [ 58 ] proved lower/upper bounds on the robustness . Pal and Vidal [ 37 ] proposed a game-theoretic framework to understand the adversarial robustness , by formulating the game between attackers and defenders w.r.t . a trained DNN . Under the game-theoretic framework , Pal and Vidal [ 37 ] theoretically proved that the FGM attack and the random smoothing defense formed a Nash Equilibrium under some assumptions . Unlike previous explanations of the existence of adversarial examples and adversarial robustness , we explain adversarial examples and adversarial training from a new perspective , i.e . the complexity of visual concepts that are learned by a DNN .

Summary: The authors discover and partially prove that adversarial perturbations mainly affect high-order interactions, rather than low-order interactions.   The authors find that there is a clear difference between standard DNNs and adversarially trained DNNs. Adversarial training significantly increases the robustness of high-order interactions.  The research provides a unified understanding for the success of several existing adversarial defense methods,