Inequality phenomenon in $l_{\infty}$-adversarial training, and its unrealized threats
Authors: Ranjie Duan
YueFeng Chen
Yao Zhu
Xiaojun Jia
Rong Zhang
Hui Xue'
Venue: ICLR 2023 notable top 25% (2023)
Keywords: Adversarial training
Adversarial robustness
Adversarial feature represenation

Abstract: The appearance of adversarial examples raises attention from both academia and industry. Along with the attack-defense arms race, adversarial training is the most effective against adversarial examples.
However, we find inequality phenomena occur during the $l_{\infty}$-adversarial training, that few features dominate the prediction made by the adversarially trained model. We systematically evaluate such inequality phenomena by extensive experiments and find such phenomena become more obvious when performing adversarial training with increasing adversarial strength (evaluated by $\epsilon$). We hypothesize such inequality phenomena make $l_{\infty}$-adversarially trained model less reliable than the standard trained model when few ``important features" are influenced. To validate our hypothesis, we proposed two simple attacks that either perturb or replace important features with noise or occlusion. Experiments show that $l_{\infty}$-adversarially trained model can be easily attacked when the few important features are influenced. 
Our work shed light on the limitation of the practicality of $l_{\infty}$-adversarial training.

Published as a conference paper at ICLR 2023
INEQUALITY PHENOMENON IN Lo-ADVERSARIAL TRAINING, AND ITS UNREALIZED THREATS
Ranjie Duan', Yuefeng Chen', Yao Zhu 1,2 Xiaojun Jia Rong Zhang' & Hue Xue
Allibba Group, *Zhejiang University, 3 University of Chinese Academy of Sciences {ranjie. drj,
stone. zhangr,
hui.
com com
ee ee ju.zhu@@zj..cc jauuyyzju.edu.cco edu. en, jiaxiaojuneiiiee ac.
ABSTRACT
The appearance of adversarial examples raises attention from both academia and industry. Along with the attack-defense arms race, adversarial training is the most effective against adversarial examples. However, we find inequality phe- nomena occur during the -adeessraal training, that few features dominate the prediction made by the udversarially trained model. We systematically evaluate such inequality phenomena by extensive experiments and find such phenomena become more obvious when performing adversarial training with increasing ad- versarial strength (evaluated by e). We hypothesize such inequality phenomena make 4o adveerrrarillly trained model less reliable than the standard trained model when the few important features are influenced. To validate our hypothesis, we proposed two simple attacks that either perturb important features with noise or occlusion. Experiments show that 4x- -dverrrrrryally trained model can be easily attacked when a few important features are influenced, Our work sheds light on the limitation of the practicality of too adversarial training.
INTRODUCTION
Szegedy et al. 2013) discovered adversarial examples of deep neural networks (DNNs), which pose significant threats to deep learning-based applications such as autonomous driving and face recog- nition. Prior to deploying DNN-based applications in real-world scenarios safely and securely, we must defend against adversarial examples. After the emergence of adversarial examples, several defensive strategies have been proposed (Guo et al. 2018 Prakash et al.) 2018 Mummadi et al.) 2019 Akhtar et al. 2018 By retraining adversarial samples generated in each training loop, ad- versarial training (Goodfellow et al., 20155 Zhang et al. [2019] Madry et al. 2018b) is regarded as the most effective detense against adversarial attacks. The most prevalent adversarial training is loc adversarial training, which applies adversarial samples with 1.2 bounded perturbation by e.
Numerous works have been devoted theoretical and empirical comprehension of adversarial train- ing (Andriushchenko & Flammarion] 2020 Allen-Zhu & Li] 2022 Kim et al. 2021). For example, Ilyas et al. (2019) proposed that an adversarially trained model (robust model for short) learns bust features from adversarial examples and discards non-robust ones. Engstrom et al. (2019) also proposed that adversarial training forces the model lcarning be invariant to features to which humans are also invariant. Therefore, adversarial training results in robust models' feature represen- tations that are more comparable to humans. Theoretically validated by Chalasani et al. (2020), the lxx adversaraaill training suppresses the significance of the redundant features, and the robust model, therefore, has sparser and better-behaved feature representations than the standard trained model. In general, previous research indicates that robust models have sparse representation of features and view such sparse representation as advantageous because it is more human-align auman--liignd. Several works investigate this property of robust models and attempt to transfer such feature representation to standard trained model using various methods (Ross & Doshi- Velez 2018 Salman et al. 2020 Deng al. 2021).

---

Published as a conference paper at ICLR 2023
However, contrary to the claim of previous work regarding such sparse feature representation as an advantage, we find that such sparseness also indicates inequality phenomena (see Section 3.1
for detailed explanation) that may pose unanticipated threats to -obbstt models. During 6o0"
adversarial training, the model not only suppresses the redundant features (Chalasani et al. 2020) but
also suppresses the importance of other features including robust ones. The degree of suppression is proportional to the adversarial attack budget (evaluated by E). Hence, given the input images for an -robust model, only handful of features dominate the prediction. Intuitively, standard- trained models make decisions based on various features, and some redundant features serve as "bulwark" when a few crucial features are accidentally distorted. However, in the case of lo robust model, the decision is primarily determined by a small number of characteristics, so the prediction susceptible to change when these significant characteristics are modified (see Figure [ As shown in Figure [ an & -robust model recognizes a street sign using very few regions of the sign. Even with very small occlusions, the robust model cannot recogmize a street sign we obscure the region that the model considers to be the most important (but well recognized by humans and the standard- trained model). Even if an autonomous vehicle is deployed with a robust model that achieves high adversarial robustness against worst-case adversarial examples, it will still be susceptible to small occlusions. Thus, the applicability of such robust model is debatable.
Clean img.
Feature attr and
Occludded img. std
Feature ads.
Occluded img. adv.
STOP
ST
Street sign
Giui: 0.59
Pred.: Street sign, Conf:42.84
Gini: 0.93
Pred.:: Parking meter, Cond:27.27
Figure I: -robutt model fails to recognize street sign with small occlusions. With given feature attribution maps that attribute the importance of each pixel, we occlude the image's pixels of high importance with small patches. The resultant image fools the robust model successfully. We notice prior works (Tsipras et al.) showed that feature attribution maps of robust models are perceptually aligned. For clarity we strongly suggest the readers check Appendix A.2
In this work, we name such a phenomenon that only a few features are extremely crucial for mod- els' recognition as "inequality phenomenon". we study the inequality from two aspects: 1) global inequality: characterized by the dominance of small number of pixels. 2) regional inequality: char- acterized by the tendency of pixels deemed significant by the model to cluster in particular regions. We analyze such phenomena on ImageNet- and CIFAR (O-trained models with various architcc- tures. We further devise attacks to expose the vulnerabilities resulting from such inequality based on our findings. Experiments demonstrate that under the premise that human observers can recognize the resulting images, lae -rbbust models are significantly more susceptible than the standard-trained models. Specifically, they are susceptible to occlusion and noise with error rates of 100% and 94% respectively, whereas standard-trained models are only affected by 30.15 % and 34.5%. In summary, our contribution can be summed up in the following manner:
We identify the occurrence of the inequality phenomenon during -dvverssali training. We design correlative indices and assess such inequality phenomena from various perspec- tives (global and regional). We systematically evaluate such phenomena by conducting extensive experiments on broad datasets and models.
Then, we identify unrealized threats posed by such inequality phenomena that loo -rbust models are much more vulnerable than standard trained ones under inductive noise or occlusion. this case, during the las -avversaaia training, the adversarial robustness is achieved at the expense of another more practical robustness.
Our work provides intuitive understanding of the weakness of loc robust model's 's feature representation from a novel perspective. Moreover, our work sheds light on the limitation and the hardness of adveesaaria training.
2

---

Published as a conference paper at ICLR 2023
2 BACKGROUND AND RELATED WORK
2.1 ADVERSARIAL, ATTACK
DNNs are known to have various risks. These risks include adversarial attacks Li et al. 2022
; Zhu
ct al.
p: Mao et al] (202T): [i et al; Gu ct al.] (2022), backdoor attacks (Guo et a 2023 Qi
al. 2023) privacy concerns (Li et al. 2020) and etc. Given model denoted f(;; ; 0)
R
and training dataset denoted as D, empirical risk minimization (ERM) a standard way (denoted
as standard training) to train the model f through:
ming wiweeeasseelloss(z
(1)
where y E IR* the one-hot label for the image and loss (xr,)) is usually cross-entropy loss. With such training scheme, the model typically performs well clean test samples. Adversarial examples (Szegedy et al. 2013) aim to generate perturbation superimposed on clean sample æ to
fool a well-trained model f. Adversarial example x' can be crafted by either following the direction
of adversarial gradients (Goodfellow et al.] 2013]] Kurakin et al. [0166] Madry et al. 20188 Duan
et al.. 2021) or optimizing perturbation with given loss (Carlini & Wagner 2017
Chen et al.
2018
2.2
ADVERSARIAL TRAINING
Several defensive strategies proposed improve the models' adversarial robustness (Wong & Kolter 2018 Akhtar et al. 2018 Meng & Chen 2017 Raghunathan et al. 2018 Wu et al., 2022). However, analysis by Athalye et (2018) shows that among various defensive strategies against adversarial examples, only the adversarial training does not rely on the obfuscated gradient and truly increases the model's robustness. A model is considered robust against adversarial examples if:
argmar (z; f)) = argmaz (a + a; s.t./ S c
where represents the magnitude of the perturbation. Therefore, the core idea of adversarial training to train models with adversarial examples, formally:
loss(z, y) = E(g.)) [max
(tte:))ee_mmxx(llll.... 5e + +0, o,,)],
The objective max os. y) introduces the model to minimize empirical risk on the training data points while also being locally stable in the (radius-c) neighborhood around each of data points J. The objective is approximated via gradient-based optimization methods such as PGD (Madry let al. (2118b). Several following works attempt to improve adversarial training from various aspects (Shafahi et al. 2019 Sriramanan et al., [022]] Jia et al. 2022b Cui et al. 2021 Jia et al., 2022a
Interestingly. Ilyas et al. (2019) proposes that by suppressing the importance of non-robust features, adversarial training makes the trained model more focused on robust and perceptually-aligee ture representations. In this process, the feature representation becomes jet al. (2020); [Salman et al. (2020): [trera et al. (2020) suggests that the feature representation gen- erated by robust model is concise as sparse and human-friendly, It only assigns the feature that is truly predictive of the output with significant contributions.
fea- sparse. Chalasani
2.3 HOW INEQUALITY FORMS DURING Lo0 ADVERSARIAL TRAINING
In (Chalasani et al. 2020 they theoretically prove the connection between adversarial robustness and sparseness: During Too adversarial training, supposed the adversarial perturbation a satisfying ||a|| < the model attempts find robust features serving as strong signals against perturba- tion. Meanwhile, the non-robust ones which serve as relatively weak signals, and their significance (acquired by feature attribution methods) more aggressively shrunk toward zero. The shrinkage rate is proportional to adversaries" strength (evaluated by e). In other words, standard training can result in models where many non-robust features have significant importance for models, whereas loc adverraaaaa training tends to selectively reduce the magnitude of the significance of non-robust features with weakly relevant or irrelevant signals and push their significance close to zero. In the end, the feature attribution maps generated by gradients-base
feature attribution methods (Smilkov jtt al. 2017; Lundberg & Lee 2017 Sundararajan an et al., (017)) look more sparse. They regard such sparseness merit of adversarial training it produces more concise and numan-aligne feature attributions. However, we further study such sparseness and find it introduces a phenomenon of extreme inequality, which results in unanticipated threats to -robstt models.
3

---

Published as a conference paper at ICLR 2023
3 METHODOLOGY
In this section, we first introduced the index used to measure inequality from two aspects. Then we propose two types of attacks validate our hypothesis: extreme inequality brings in unexpected threats to lo---rr -robust model.
3.1 MEASURING THE INEQUALITY OF A TEST DATA POINT
Firstly, feature attribution maps are required characterize the inequality degree by given test data point x and model f. Several feature attribution methods have been proposed recent years (Smilkov al. 2017: Lundberg & Lee 2017 Sundararajan al. 2017). In general, feature attri- bution methods rank the input features according to their purported importance in model prediction. To be specific, treat the input image x as set of pixels x { I...M) and denote the generated feature attribution map of x of model f as A (z). where A (x) is composed of a, Fea- ture attribution methods attribute an effect a, to each 24, and summing the effects of all feature attributions approximates the output fzz). I3 achieves the top-most score (a;) is regarded as the most important pixel for prediction, whereas those with the bottom-most score are considered least important.
With given sorted (x))) = {u;, i 1...Maa < dx+1) generated by a typical feature attribution method, if the prediction f(x) can be approximated with the sum of N most important features and N is much less than M, we name such distribution of A' (x) is unequal. Namely, the prediction on made by model f is dominated by a few pixels. Formally, measure the inequality of the distribution of given feature attribution map. Given a population set indexed in non-decreasing order o- { = 1...m/6: Φ1+1). Gini coefficient can be calculated as:
use Gini index (Dorfman 1979) to
Gini( 1 11 x (n + 1 - i) Di
(4)
n1
> di
An advantage of the Gini(-) index is that inequality of the entire distribution can be summarized by using a single statistic that is relatively easy to interpret (see Appendix |A.3 a more detailed comparison between Gini and other sparsity measures). The Gini index ranges from O, when the value of every di is equal, to I, when a single é, takes all the sum. This allows us to compare the inequality degree among feature attributions with different sizes. We define two types of inequality as follows:
Global inequality: Given a feature attribution map A² i ac+1 on test data point x, we only consider the inequality degree of the global distribution of A (x)) and take no into account for other factors, the inequality degree is calculated with Ginig (A'))) directly. The higher of Gini, (A (z)). the more unequal the distribution A (æ). the fewer pixels take the most prediction power. When Ginig (x)) is equal to 1, indicates one pixel dominates the prediction while all the other pixels have no contribution to the current prediction.
Regional inequality: We also consider inequality degree together with spatial fac- tor, whether important feature tends to cluster at specific regions. A region is de- fined as block with size of n of input space. We first divide pixels into differ- ent regions and calculate the sum of pixels' importance by regions, formally, Af(r) = {ar., i 1...m|ar, < "r where a, is the sum of a, in the region. Therefore, the Gini value on (x())) reflects the inequality degree of different regions of input space. The higher the value of (A,,
the more important pixels tend to cluster in the specific regions. When Gini,(Af (z)) is equal to I, it represents all pixels that contribute to the prediction cluster in one region (block).
(A,(x)),
In what follows, we propose potential threats caused by such inequality (global and regional inequal- ity). We devise attacks utilizing common corruptions to reveal the unreliability of such decision pattern by robust model.
4

---

Published as a conference paper at ICLR 2023
3.2 ATTACK ALGORITHMS
We propose two simple attacks to validate potential threats caused by such inequality phenomena: 1) Inductive noise attack. 2) Inductive occlusion attack.
3.2.1 INDUCTIVE NOISE ATTACK
We evaluate the models' performance under attacks designed by two types of noise.
Noise (Type I): Given an image we perturb the most influential pixels of images with Gaussian noise a E (0, via masking M. Formally:
[0,a; dire
x +
where
M; 11,0; 2 atre
(5)
where atre represents the threshold. I; with value that is below to the aore will be kept, and T; whose a; 2 Otre is perturbed by Gaussian noise.
Noise (Type II): About the second type of noise attack, directly replace important pixels with Gaussian noise, formally x' M æ + M where represents reverse mask of M. Compared to Noise-I, Noise replaces important pixels totally and disturbs images more severely.
If the model's decision pattern is extremely unequal, the performance will be highly influenced when important features are corrupted by inductive noise attacks.
3.2.2 INDUCTIVE OCCLUSION ATTACK
With respect to inductive occlusion attack, obseure regions of important pixels with occlusions gradually. During the attack, the max count of occlusions is N with radius at max R. The order of regions to perturb is decided by the value A (z). that region higher a,, perturbed priority by occlusions with size r E {I.../t}. The number of occlusions is constrained by n E {I...N.. We also consider occlusion with different colors reflect potential real-world occlusion. The inductive occlusion attack algorithm is listed as follows:
Algorithm Inductive Occlusion Attack
Require: Test data point (z, v). Model f. Regional Attribution map Af Max count and radius N, R, Perturb color
Ensure: f(x) = < 1,r 1, =
Ensure the test data æ is correctly classified by model f.
for n = to N do
for r 1 to R do
M + get_perturb_ mask(A/(2))
function to acquire the perturbation mask. D Perturb by mask M with color c.
= M x + M *
fx²) # y :break
end for end for return z'
Note the intention of this work is not to propose strong adversarial attacks. Although either noise or occlusion is beyond the threat model considered in loo adversarial training, we intend to reveal threats caused by such inequality phenomena that previous work ignored. In summary, the extreme inequality decision pattern of loo -tained adversarial models to result in themselves being more fragile under some corruptions.
EXPERIMENTS
In this section, first outline the experimental setup. We then evaluate the inequality degree (by Gini) different models. Then, we evaluate the performance of the proposed attacks. Finally, we perform ablation study about the selection of feature attribution methods.
5

---

Published as a conference paper at ICLR 2023
4.1 EXPERIMENTAL SETTINGS
Dataset and models. We perform a series of experiments on ImageNet Deng el al. 2009) and CI-
FARIO Krizhevsky al. 2009). With respect to experiments on ImageNet, we use ResNet18 (He
et al. 2016). ResNet50. WideResNet50 (Zagoruyko & Komodakis, 2016) provided by Microsoft
For CIFAR1O, we use ResNet18, DenseNet (Huang et al.] (017) (see A. for detailed configura- tions). Regarding feature attribution methods (implementation by Captun we consider methods including Input X Gradients (Shrikumar et al., 2016), Integrated Gradients (Sundararajan et al.)
2017), Shapley Value (Lundberg & Leel 2017) and SmoothGrad (Smilkov et 2017). Considering space and time efficiency, we primarily present experimental results based on Integrated Gradients and perform ablation study the other feature attribution methods.
Metrics. For all the tests about the models' performance, we use error rate (%) as the met- ric evaluate the model's performance under corruptions (c.g., noise and occlusions), which is the proportion of misclassified test images among the total number of test images defined as N [x)(f(r) * f(r')), where x represents clean test images, and x' represents test images cor- rupted by noise and occlusions. For a fair comparison, we first select 1000 random images from ImageNet that are correctly classified by all the models before performing the attack.
1
4.2 INEQUALITY TEST
In this section, we first evaluate the inequality degree (both global and regional inequality) of -robusttt models and standard trained models with different architectures (ResNet18, ResNet50, WideResNet, DenseNet) trained on ImageNet and CIFARIO. We also evaluate the inequality degree of different models adversarially trained with increasing adversarial strength (c = 1, 2. 4, 8). In the case of the evaluation on Gini, We applied the Gini index to the sorted absolute value of the flattened feature attribution maps. On evaluating regional inequality, we set the region's size as 16 * 16 for experiments ImageNet and 4 4 for CIFAR10. The results presented in Table As shown
Table 1:
Gini index across different models. We evaluate the Gini coefficient of different models trained with different on ImageNet and CIFAR10. 10..
Dataset
Model
Std. trained
<= 1.0
6 2.0 e 4.0
6 8.0
Global Incquality
ResNet18
0.58 0.05
0.65 + 0.05
0.67 + 0.06
0.69 - 0.06
0.73 + 0.06
CIFAR1O
DenseNet
0.57 0.04
0.66 + 0.06
0.67 + 0.06
0.69 = 0.06
0.72 - 0.07
Regional Inequality
ResNet18
0.79 0.02
0.87 + 0.04
0.87 0.04
0.88 0.04
0.88 0.04
DenseNet
0.79 = 0.02
0.85 + 0.04
0.86 - 0.04
0.87 = 0.04
0.88 = 0.03
Global Inequality
ResNet18
0.60 = 0.04
0.69 + 0.06
0.79 + 0.04
0.92 = 0.01
0.95 - 0.01
ResNet50
0.62 0.04
0.75 + 0.05
0.86 + 0.03
0.92 + 0.02
0.94 + 0.01
ImageNet
WideResNet
0.62 0.05
0.74 + 0.05
0.79 + 0.04
0.88 + 0.03
0.94 + 0.01
Regional Inequality
ResNet18
0.80 0.02
0.83 + 0.04
0.88 0.03
0.95 0.01
0.97 0.01
ResNet50
0.84 = 0.02
0.91 4 0.05
0.95 + 0.02
0.96 : 0.01
0.97 - 0.01
WideKesNet
0.81 = 0.03
0.86 + 0.03
0.88 + 0.03
0.93 = 0.03
0.97 = 0.02
in Table [ on CIFARIO, the global inequality degree of the standard trained model with different architectures is around 0.58. The Gini (global inequality) for -robust model is around 0.73 when = 8. Notably, the inequality phenomena is much more obvious on ImageNet. Especially for an adversarially trained Resnet50 ( 8), the Gini achieves 0.94, which indicates that only a handful of pixels dominate the prediction. Experiments on CIFARIO and ImageNet show that L robustt models rely on fewer pixels support the prediction with the increasing of the adversarial strength

---

Published as a conference paper at ICLR 2023
(c). We also test the inequality degree different classes belonging to ImageNet; classes related to animal tends to have a higher value on Gimi index. For example, class 'Bustard' has the highest value Gini of 0.950. Classes related to scenes stuff tend to have lower Gini. For example, Ap.7 A.7).
class 'Web site' has the lowest inequality of 0.890 (See Appendix
We visualize the features' attribution of given images for the standard and aadeervrsari trained ResNet50 respectively in Figure 2
When the model adversarially trained with weak adversarial
img.
Std. trainedd
Adv. tratied (e 1
Ado trained 2.0)
Adar trsined
Adv tratied
Gimi:
Gini: 6464
Gini: 0.75
Gimi:
Ginl: 0.93
Gini: 0.95
Figure 2: Feature attributions of different models. We visualize feature attributions generated by -robustt models (adversarially trained by adversaries of different c), the larger of c, the fewer features that model relies on for prediction.
strength 1), the model has better feature attribution aligned human observers. However, when the adversarial strength increases, the model gradually assigns higher importance to fewer pixels and resulting in extreme inequality regarding feature attribution. Moreover, these most important pixels tend to gather in few specific regions Additional visualizations for ImageNet and CIFAR 10 are in Appendix A. and A.10 10 respectively).
4.3 EVALUATION UNDER INDUCTIVE NOISE ATTACK
In this part, we compare the performance of standard- and adversarially- trained ResNer50 under random and inductive noise. We set noise with different scales, including subpixels of 500, 1000, 5000, 10000, and 20000, We present the results in Figure g
Models' Performance (error under Nolse.
Parturbed Inge
Detaile
(16)
enitted Restier0 (renddi))
Ady. trancd ReaNet5 munndrii
Sed trained Rechie50 (lerse it
Veltel
AdV trained Reckintoo (hoise
Rondee Welio
2500
7500
10000
12500
15000
37500
20000
parturbed subpixel
Evaluation under noise. We plot the enror rate of standard- and adversarially-trinn trained models on images perturbed by the increasing number of noise.
Figure 3:
Under random noise, the success rate of attack on the robust model achieves 73.4%, but only 18.8% for standard-trained model. Under Noise of Type I, the robust model is fooled by 94.0%, while the standard trained model is only fooled by 34.5%. Under Noise Type II, even when we control the amount of noise with small threshold (e.g.. 1000 pixels), more than 50% of predictions made by the robust model is affected. When we enlarge the threshold to 20000, the robust model (c=8) is almost fooled with 100% success rate. In summary, compared the standard trained model,

---

Published as a conference paper at ICLR 2023
lce robust model relies much fewer pixels to make decision; such decision pattern results in unstable prediction under noise.
4.4 EVALUATION UNDER INDUCTIVE OCCLUSION ATTACK
In this part, we perform an inductive occlusion attack and evaluate the standard- and -robust ResNet50s' performance. We set two group experiments with different thresholds.
In the first group of experi- ments, we generate occlusions with max count of 5 and a max radius of 10, In this case, the adversarially-ttained model fooled at a 71.7% error rate, but the standard trained model' predictions are only affected by enlarge the
Table 2: Models' performance (Error rate %) under occlu- sions. We evaluate the models' performance by gradually oc- cluding important areas with patches of different sizes and col- ors.
Model
Std.
c= 1.0
<= 2.0.
c=
c= 8.0
31.6%. When threshold and set max count as 10 and radius as 20 for occlu- sions, both = 4 and e = 8 adversarially trained model can be fooled with 100% success rate while only 41.2% attack suc- cess rate for the standard-trainee model. We visualize the re- sults in Figure shows, loc -d-verrsaarlllyy trained model with larger e could be easily attacked by smaller occlusions under the same thresh- old. For example, in Figure the standard trained model can recognize 'Bulbul' well with the head part occluded, but the adversarially trained model fails recognize the 'Bulbul' if only the beak of the bulbul is occluded. Moreover, compared with adversarial perturbation, occlusion is more practi-
Max ent N = 5,
10
Occlusion-G Occlusion-W W
23.5%
31.6% 48.4%
38.4%
32.4%
54.0%
28.3%
57.5%
61.3%
71.7%
Occlusion-B 31.6%
51.5%
53.3%
48.9%
64.6%
Max ent N 10, R = 20
Occlusion-G Occlusion-W Occlusion-B 41.2%
30.1 %
48 18.2%
56.3%
100.0%
100.0%
40.1%
59.1%
73.3%
100.0%
100.0%
70.2%
72.2%
100.0%
100.0%
As the figure
Clean
Std. trained
1.0)
Add.trateed (c- 2.0)
Ado.trained (c 4.0)
Adv. weh.trriieedeee (e 0.U)
Butbal
Bulbal
Chiskudee
Chickadee
Chirkadw
Border terrier
Barder tervier
Border terrier
Border terrier
Muzale
Cuirass
Correct
WWrang
Figure 4: Visualization occluded images. We visualize images occluded with different patches of different sizes and the corresponding predictions made by standard and -aveessrrrlll trained models. Compared to a mondard-trainedd model, the adversarially trained model fragile when occlusion covers the area of important features.
cal occlusion frequently appears in the real world. We also evaluate the transferability of attacked results between the robust model and the standard trained model, the results are consistent with our observation (see Appendix A.4
8

---

Published as a conference paper at ICLR 2023
4.5 ABLATION STUDY
We consider four attribution methods for the ablation study: Input X Gradient (Shrikumar et al., 2016), SmoothGrad (Smilkov et al, alla 2017), Gradient Shapley Value (GradShap for short) (Lundberg & Lee 2017) and Integrated Gradients (Sundararajan et al. 2017) (see Appendix [A.8 for detailed configuration). We perform an ablation study to evaluate the effect of selection on the feature at- tribution methods (see Table 3 Among various attribution methods, SmoothGrad produces more
Table 3: Ablation study on selection. We evaluate our hypothesis with different feature attribution methods.
Attribution Method
Model
Gini
Noise
Noise Occlusion-B
Occlusion-G
Occlusion-
Std. trained
0.63
16.1%
45.0%
24.7%
16.8%
23.5%
Input X Gradient
Adv. trained
0.93
60.9%
90.4%
63.3%
51.2%
63.2%
trained
0.62
19.8%
54.7%
31.5%
24.1%
29.9%
GradShap
Adv. trained
0.93
62.3%
93.7 %
64.8%
53.3%
71.6%
Std. trained
0.75
63.5%
45.0%
32.3%
25.6%
30.3%
SmoothGrad
Adv. trained
0.98
82.5%
98.3%
61.6%
49.7%
60.8%
Std. trained
0.62
16.5%
55.5%
31.6%
23.5%
28.3%
Integrated Gradients
Adv. trained
0.94
63.9
95.5%
64.6%
54.0%
71.7%
spare feature attribution maps and thus results in higher values on Gini. Regarding evaluation under noise, SmoothGrad increases the inductive noise attack's success rate. Regarding evaluation under occlusion, the selection of Integrated Gradients improve the attack's 's success rate models.
In conclusion, the selection of attribution methods slightly affects attacks' success rates but does not change our conclusion: the distribution of features' attribution by -obustt model is much more unequal; such inequality makes the robust model more susceptible to inductive noise and occlusions.
DISCUSSION AND CONCLUSION
In this work, we study the inequality phenomena that occur during 4o adverrsaraa training. Specif- ically, we find -rbuust models' feature attribution not as aligned with human perception we expect. An ideal human--erccptuu aligned model is expected to make decisions based on a series of core feature attributions. For example, the model classifies an input image as a bird, it should take attributions, including the eye, the beak of the bird, and the shape of the bird, all of these attribu- tions into account. However, we find -obustt model only relies on individual attribution (only the bird's beak) for recognization. We name such phenomena as inequality phenomenon. We perform extensive experiments to evaluate such inequality phenomena and find that lsc robust model assigns few features with extremely high importance. Thus, few features dominate the prediction. Such extreme inequality of Lo0 -robust model results in unreliability. We also design attacks (by utilizing hypothesis that robust models could be more susceptible under
noise and occlusion) to validate some scenarios. We find an attacker can easily fool the -trined model by modifying important features with either noise or occlusion easily. We suggest that both noise and occlusion common in a real-world scenario. Therefore, robustness against either noise or occlusion is more essential and crucial than robustness against adversarial examples. Our work reveals the limitation and vul- nerability of the current -obust model. We also evaluate if such inequality phenomenon exists in 1o-robust model and models trained with sparsity regularization. The evaluation results show that such a phenomenon is unique property of -obustt model (see Appendix and A.6)
We also propose a strategy to release such inequality phenomena during 4 adversarial training. We combine Cutout (DeVries & Taylor (017)) strategy with adversarial training and force the model learning features from different regions by cutting out part of training images at each iteration dur- ing the training (see the result in Appendix A.9). The strategy slightly releases the inequality degree of the robust model. More effective strategies releasing such extreme inequality could be a crucial and promising direction for future work. We hope our work motivate new research into the char- acteristics adversarial training and open up further challenges for reliable and practical adversarial training.

---

Published as a conference paper at ICLR 2023
ETHICS STATEMENT
In this paper, we identify inequality phenomena that occur during loe -dversaaiia training, that too robust model tends to use few features to make the decision. We give a systematical evaluation of such inequality phenomena across different datasets and models with different architectures. We further identified unrealized threats caused by such decision patterns and validated our hypothesis by designing corresponding attacks. Our findings provide a new perspective on inspecting adversarial training. Our goal is to understand current adversarial training's weaknesses and make DNNs truly robust and reliable. We did not use crowdsoureing and did not conduct research with human subjects in our experiments. We cited the creators when using existing assets (e.g., code, data, models).
REPRODUCIBILLIT STATEMENT
We present the settings haper-pppaae paramerrraettrs and how they were chosen the experiment section. We repeat experiments multiple times with different random seeds and show the corresponding standard deviation in the tables. We plan to open the source code to reproduce the main experimental results later.
REFERENCES
Direct methods for sparse matrices. Oxford University Press, 2017.
Naveed Akhtar, Jian Liu, and Ajmal Mian. Defense against universal adversarial perturbations. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp- 3389-3398, 2018
Zeyuan Allen-Zhu and Yuanzhi Li. Feature purification: How adversarial training performs robust deep learning. In 2021 IEEE 62nd Annual Symposium on Foundations of Computer Science (FOCS), pp. 977-988. IEEE, 2022.
Maksym Andriushchenko and Nicolas Flammarion. Understanding and improving fast adversarial training. Advances in Neural Information Processing Systems, 33: 13048-1616,,5, 2020.
Anish Athalye, Nicholas Carlini, and David Wagner. Obfuscated gradients give a false sense curity: Circumventing defenses to adversarial examples. In International conference on machine learning, pp. 273-283. PMLR, 2018
Nicholas Carlini and David Wagner. Adversarial examples not easily detected: Bypassing ten detection methods. In ACM Workshop on Artificial Intelligence and Security, pp. 3-14. ACM, 2017.
Prasad Chalasani, Jiefeng Chen, Amrita Roy Chowdhury, Xi Wu, and Somesh Jha. Concise expla- nations of neural networks using adversarial training. In International Conference on Muchine Learning, pp. 1383-1391. PMLR, 2020.
Pin-W Yu Chen, Yash Sharma, Huan Zhang, Jinfeng Yi, and Cho-Jui Hsieh. Ead: Elastic-net attacks to deep neural networks via adversarial examples. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 32, 2018.
Francesco Croce and Matthias Hein. Reliable evaluation of adversarial robustness with an ensemble of diverse parameter-free attacks. In International conference machine learning, pp. 2206- 2216. PMLR, 2020.
Jiequan Cui, Shu Liu, Liwei Wang, and Jiaya Jia. Learnable boundary guided adversarial training. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 15721-15730, 2021.
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hierarchical image database. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 255-255, 2009.
10

---

Published as a conference paper at ICLR 2023
Zhun Deng, Linjun Zhang, Kailas Vodrahalli, Kenji Kawaguchi, and James Y Zou. Adversarial training helps transfer learning via better representations. Advances in Neural Information Pro- cessing Systems, 34:25179-255 2519,, 2021.
Terrance Vries and Graham W Taylor. Improved regularization of convolutional neural networks with cutout. arXiv preprint A 04552, 2017.
Robert Dorfman. A formula for the gini coefficient. The review of economics and statistics, pp- 146-149, 1979.
Ranjie Duan, Yuefeng Chen, Dantong Niu, Yun Yang, A Kai Qin, and Yuan He. Advdrop: Adver- sarial attack to DNNs by dropping information. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 7506-7515, 2021.
Logan Engstrom, Andrew Ilyas, Shibani Santurkar, Dimitris Tsipras, Brandon Tran, and Alek- sander Madry. Adversarial robustness as a prior for learned representations. arXiv preprint arXiv:1906.000 2019.
lan J Goodfellow, Jonathon Shlens, and Christian Szegedy. Explaining and harnessing adversarial examples. In International Conference on Learning Representations, 2015.
Jindong Gu, Hengshuang Zhao, Volker Tresp, and Philip HS Torr. Segpgd: An effective and efficient adversarial attack for evaluating and boosting segmentation robustness. In Computer Vision ECCV 2022: 17th European Conference, Tel Aviv, Israel, October 23-27, 2022, Proceedings, Part XXIX, pp. 308-325. Springer, 2022.
Chuan Guo, Mayank Rana, Moustapha Cisse, and Laurens der Maaten. Countering adversarial images using input transformations. In International Conference on Learning Representations, 2018.
Junfeng Guo, Yiming Li, Xun Chen, Hanging Guo, Lichao Sun, and Cong Liu. Scale-up: An efficient black-box input-level backdoor detection via analyzing scaled prediction consistency. In ICLR, 2023.
Song Han, Jeff Pool, John Tran, and William Dally. Learning both weights and connections for efficient neural network. Advances in neural information processing systems, 28, 2015.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recog- nition. In Proceedings the IEEE/CVF Conference on Computer vision and Pattern Recognition, pp. 770-778, 2016.
Gao Huang, Zhuang Liu, Laurens Van Der Maaten, and Kilian Q Weinberger. Densely connected convolutional networks. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 4700-4708, 2017.
Andrew Ilyas, Shibani Santurkar, Dimitris Tsipras, Logan Engstrom, Brandon Tran, and Aleksander Madry. Adversarial examples are not bugs, they are features. In Neural Information Processing Systems, pp. 136, 2019.
Xiaojun Jia, Yong Zhang, Xingxing Wei, Baoyuan Wu, Ke Ma. Jue Wang, and Xiaochun Cao. grior-guided adversarial initialization for fast adversarial training. In Computer Vision-ECCV 2022: 17th European Conference, Tel Aviy, Israel, October 23-27, 2022, Proceedings, Part IV, pp. 567-584. Springer, 2022a.
Xiaojun Jia, Yong Zhang, Baoyuan Wu, Ke Ma, Jue Wang. and Xiaochun Cao. Las-at: Adversarial training with learnable attack strategy. In Proceedings of the IEEE/CVF Conference on Computer Vision and Puttern Recognition, pp. 13398- 2022b.
Xiaojun Jia, Yong Zhang, Baoyuan Wu. Jue Wang, and Xiaochun Cao. Boosting fast adversarial training with learnable adversarial initialization. IEEE Transactions on Image Processing, 31: 4417-4430, 2022c.
11

---

Published as a conference paper at ICLR 2023
Hoki Kim, Woojin Lee, and Jaewook Lee. Understanding catastrophic overfitting in single-step adversarial training. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 35, pp. 111988127, 2021.
Alex Krizhevsky, Geoffrey Hinton, et al. Learning multiple layers of features from tiny images. 2009.
Alexey Kurakin, Ian Goodfellow, and Samy Bengio. Adversarial examples in the physical world. International Conference on Learning Representations, 2016.
Hao Li, Asim Kadav, Igor Durdanovic, Hanan Samet, and Hans Peter Graf. Pruning filters for efficient convnets. In International Conference Learning Representations
Xiaodan Li, Yining Lang, Yuefeng Chen, Xiaofeng Mao, Yuan He, Shuhui Wang. Hui Xue, and Quan Lu. Sharp multiple instance learning for deepfake video detection. In Proceedings of the 28th ACM international conference multimedia, pp. 1864-1872, 2020.
Xiaodan Li, Jinfeng Li, Yuefeng Chen, Shaokai Ye, Yuan He. Shuhui Wang. Hang Su, and Hui
Xue. Qair: Practical query-efficient black-box attacks for image retrieval. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 3330-3339, 2021.
Scott M Lundberg and Su-In Lee. A unified approach interpreting model predictions. Advances in nearal information processing systems, 30, 2017.
Aleksander Madry, Aleksandar Makelov, Ludwig Schmidt, Dimitris Tsipras, and Adrian Vladu. Towards deep learning models resistant to adversarial attacks. In International Conference on Learning Representations, 20.
Aleksander Madry, Aleksandar Makelov, Ludwig Schmidt, Dimitris Tsipras, and Adrian Vladu. Towards deep learning models resistant to adversarial attacks. In International Conference on Learning Representations, 2018b.
Xiaofeng Mao, Yuefeng Chen, Shuhui Wang, Hang Su, Yuan He, and Hui Xue. Composite adver- sarial attacks. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 35, pp- 8884-8892, 2021.
Dongyu Meng and Hao Chen. Magnet: a two-pronged defense against adversarial examples. In Proceedings of the 2017 ACM SIGSAC conference computer and communications security, pp. 147, 2017.
Mazda Moayeri, Kiarash Banihashem, and Soheil Feizi. Explicit tradeoffs between adversarial and natural distributional robustness. In Advances in Neural Information Processing Systems.
Mazda Moayeri, Phillip Pope, Yogesh Balaji, and Soheil Feizi. A comprehensive study of image classification model sensitivity to foregrounds, backgrounds, and visual attributes. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 19087-19097, 2022.
Chaithanya Kumar Mummadi, Thomas Brox, and Jan Hendrik Metzen. Defending against universal perturbations with shared adversarial training. In Proceedings of the IEEE/CVF International Conference Computer Vision, pp. 4928 4937, 2019.
Aaditya Prakash, Nick Moran, Solomon Garber, Antonella DiLillo, and James Storer. Deflecting ad- versarial attacks with pixel deflection. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 8571-8580 2018.
Gege Qi, GONG Lijun, Yibing Song. Kai Ma, and Yefeng Zheng. Stabilized medical image attacks. In International Conference on Learning Representations.
Xiangyu Qi, Tinghao Xie, Yiming Li, Saced Mahloujifar, and Prateek Mittal. Revisiting the sumption of latent separability for backdoor defenses. ICLR, 2023.
Aditi Raghumathan, Jacob Steinhardt, and Percy Liang. Certified defenses against adversarial exam- ples. In International Conference Learning Representations, 2018.
12

---

Published as a conference paper at ICLR 2023
Andrew Ross and Finale Doshi-Velez. Improving the adversarial robustmess and interpretability of deep neural networks by regularizing their input gradients. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 32, 2018.
Hadi Salman, Andrew Ilyas, Logan Engstrom, Ashish Kapoor, and Aleksander Madry. Do adver- sarially robust imagenet models transfer better? Advances in Neural Information Processing Systems, 33:3533-3545, 2020.
Ali Shafahi, Mahyar Najibi, Mohammad Amin Ghiasi, Zheng Xu, John Dickerson, Christoph Studer, Larry S Davis, Gavin Taylor, and Tom Goldstein. Adversarial training for free! Ad- vances Neural Information Processing Systems, 32, 2019.
Avanti Shrikumar, Peyton Greenside, Anna Shcherbina, and Anshul Kundaje. Not just black box: Learning important features through propagating activation differences. arXiv preprint arXiv:1605.01713. 2016.
Daniel Smilkov, Nikhil Thorat, Been Kim, Fernanda Viégas, and Martin Wattenberg. Smoothgrad: removing noise by adding noise. arXiv preprint arXiv:/706.0853 03825, 2017.
Gaurang Sriramanan, Sravanti Addepalli, Arya Baburaj, et Towards efficient and effective adver- sarial training. Advances in Neural Information Processing Systems, 34:1 11111122111111 1833, 2021.
Mukund Sundararajan, Ankur Taly, and Qiqi Yan. Axiomatic attribution for deep networks. In International conference on machine learning, pp. 3319-3328. PMLR, 2017.
Christian Szegedy, Wojeiech Zaremba, Ilya Sutskever, Joan Bruna, Dumitru Erhan, Ian Goodfel- low, and Rob Fergus. Intriguing properties of neural networks. In International Conference on Learning Representations, 2013.
Dimitris Tsipras, Shibani Santurkar, Logan Engstrom, Alexander Turner, and Aleksander Madry. Robustness may be at odds with accuracy. International Conference on Learning Representa-
fions.
Francisco Utrera, Evan Kravitz, N Benjamin Erichson, Rajiv Khanna, and Michael W Mahoney. Adversarially-trained deep nets transfer better: Illustration on image classification. In Interna- tional Conference on Learning Representations, 2020.
Eric Wong and Zico Kolter. Provable defenses against adversarial examples via the convex outer adversarial polytope. In International Conference on Machine Learning, pp. 1288-5295, PMLR, 2018.
Boxi Wu, Jindong Gu, Zhifeng Li, Deng Cai, Xiaofei He, and Wei Liu. Towards efficient adversarial training on vision transformers. In Computer Vision-ECCV 2022: 17th European Conference, Tel Aviy, Israel, October 23-27, 2022, Proceedings, Part XIII, pp. 307-325. Springer, 2022.
Sergey Zagoruyko and Nikos Komodakis. Wide residual networks. In British Machine Vision Conference 2016. British Machine Vision Association, 2016.
Hongyang Zhang, Yaodong Yu, Jiantao Jiao, Eric Xing, Laurent El Ghaoui, and Michael Jordan. Theoretically principled trade-off between robustness and accuracy. In International conference on machine learning, pp. 7472-7482. PMLR, 2019.
Yao Zhu, Yuefeng Chen, Xiaodan Li, Kejiang Chen, Yuan He, Xiang Tian, Bolun Zheng, Yaowu Chen, and Qingming Huang. Toward understanding and boosting adversarial transferabilityy from a distribution perspective. IEEE Transactions on Image Processing, 31:6487-6501, 2022.
13

---

Published as a conference paper at ICLR 2023
A APPENDIX
A.1 DETAILS OF MODELS
A.1.1 DETAILS MODELS TRAINED ON IMAGENET
We summarize the clean and robust accuracy (%) of models trained on ImageNet in Table ??. Re- garding robust accuracy, use PGD for evaluation. During the training ImageNet, the images
Table 4: loc -robust models' clean and robust accuracy (ImageNet)
Model
Std. trained
1/255
c=2/255
c=4/255
c=8/255
ResNet-188
69.76
63.46 /33.38
59.63
/29.80
52.49 /26.52
42.11 /23.75
ResNer-50
76.13
72.05 /45.02
69.10 / 42.75
63.86 / 38.85
54.53 33.05
Wide-ResNet-50
81.60
74.65 147.30
72.35 /45.95
68.41 /43.10
60.82 /38.35
are resized to 256 using pitepppation-meeepplaiiadddBBBBd BILIINERR followed by central crop of size=224. Finally, the values are first rescaled [0.0,1.0] and then normalized using mean=[0.485, 0.456, 0.406] and std=[0.229, 0.224, 0.225].
TRAIN TRANSFORM transforms. Compose (
transforms. Resize (256). transforms CenterCrop (224),
transforms. RandomHorizontalPlii transforms. ToTensor ()
1)
TEST. TRANSFORM transforms Compose ([
transforms. Resize (256),
transforms. CenterCrop (224), transforms. ToTensor (),
})
A.1.2 DETAILS OF MODELS TRAINED ON CIFARIO
We summarize the clean and robust accuracy of models trained on CIFAR1O Table robust accuracy, AutoAttack Croce & Hein (020) for evaluation.
Regarding
Table 5: robust models' clean and robust accuracy (CIFAR10)
Model
Std. trained
c= 1=15555
c=2/255
4=/25
c=8/55
ResNet-18
93.90
92.10 /86.60
90.40 /79.90
88.30 /68.80
81.20 /48.66
DenseNet
92.80
91.30 /86.60
89.90 /79.50
85.80 /65.90
79.60 /44.40
TRAIN.TRANSFORM transforms. Compose (I transforms RandomCrop (32, padding=4), transforms HandomHorizontanl transforms. ToTensor (),
])
TEST.TRANSFORM transforms. wwmppoooom...oosseeeee. Compose ([
transforms ToTensor (),
1)
14

---

Published as a conference paper at ICLR 2023
A.2 FURTHER DISCUSSION ABOUT VISUALIZATION FEATURE ATTRIBUTION
Most visualization methods apply post-processing techniques during generating feature attribution maps. The post-processing technique is also clarified in (Tiipras et a "For CIFAR-10 and Ima- geNet, we clip gradients to within =30 and rescale them to lie in the [0, 1] range." Thus, the most influential pixels with extremely high values are clipped relatively lower value but they actually dominate the prediction (see Figure 50.
0.4
Vinuiitation
STOP
0.3
20
34.1%
34.1%
10
2,1%
2.1%
0,1%
cosh
Figure 5: Visualizing feature attribution with and without post-processngg (by 3 deviations).
We also provide more visualization of feature attribution maps with and without post- processr..s.ing... As Figure shows, the post-processed feature attribution maps are more perceptually-alignn with human observers. However, such visualization are not subjective.
Original
Post processed
Original
Post processed
Input
Std. trained
Adv. trained (l..
Figure 6: More visualization of feature attribution maps with and without post-processing.
15

---

Published as a conference paper at ICLR 2023
A.3 SPARSITY MEASURE VS. GINI INDEX & MOTIVATION BEHIND USING GINI INDEX
The sparsity of a vector can be quantified by laelle (duf 2017). which simply calculates the ratio of non-zero elements. However, the sparsity measure treats infinitesimally small value the same as a significant value. Even if some of the small coefficients increase by significant values, that change will not be reflected by change in the value of the sparsity measure. For example, given a vector F; = [0, O, O, 1, 1, and 2/ = [0, 0, O. 1, 1000]. the sparsity degree of 2 and 22 is equal to totally different. The distribution of 22 is much more unequal |x is sensitive to noise, especially in settings where most values of elements are around O (e.g., feature attribution map).
|æ]
0.4. However, their distributions
compared with ø1. Also, ]x|
In our case, Gini is able reflect the change when small coefficient increases. A Gini coefficient of 0 expresses perfect equality, where all values are the same, while a Gini coefficient of 1 (or 100%)
expresses maximal inequality among values. For example, the Gini of æ = [0, 0, 0, 1] equals to 0.6, and 0.799 for 22 [0, 0, 0. 1, 10000. The value of Gini also provides an intuitive understanding of the distribution. When Gini 0.6, approximately 40% in the population (1-0.6 0.4) occupies the total worth. When Gini = 0.799, approximately 21.1% of the population dominates the worth. As for our experiment, Gini of feature attributions by lee -robutt model (e 8.0) is about 0.95, representing less than 5% of pixels that dominate the prediction.
A.4 TRANSFFRABILITY TEST
we perform occlusion attacks with two groups of attack budgets:
Group 1: max count max radius 10.
Group 2: max count 10, max radius = 20.
We perform noise attacks with threshold 5000.
Attack
Oce -B r-
Occ-B (cnt=10, r=20)
Noise-1
Noise-II
Model
Adv.
Std.
Adv.
Std.
Adv.
Std.
Adv.
Std.
Adv.
100.0
11.4
100.0
17.4
100.0
10.6
100.0
18.6
Std.
22.8
100.0
43.6
100.0
26.8
100.0
58.2
100.0
As the table shows, the transferability between the -robustttt model and the standard-trained model is low. Transferring attack results from the standard-trained model to the -robust model is easier.
If the region of the most important pixels is occluded, the -robuutt model fails recognize the images correctly. The experiments are consistent with our observations.
A.S COMPARED WITH 12- ROBUST MODEL
Due to different properties of ip norm vector space, the case of l2 adversarial training is not the same as loo adversarial training. To be specific, lxs constrains the maximum magnitude of perturbation for each pixel. The adversarial noise added on each pixel independently during the 4.0 adversarial training. Therefore, the model attempts to find the most robust feature against noise and drops the features which could be affected by adversarial noise. Therefore, with increasing the magnitude (e) of adversarial noise, fewer but more robust features I robust model can rely on for recognition.
Different from Loe norm measures each pixel independently. la norm calculates the square root of the inner product of all elements in a vector. Thus, during l adversarial training, if a large budget of perturbation perturbs some pixels, the other pixels share the left budget on perturbation. Moay- er et a provides a game-theoree thme-theooetti understanding of Iz-adversarial training: during each loop of lg-adversarial training, the attacker perturbs the features which are predictive for the model. When some predictive features are perturbed with the most budget of perturbation, the features perturbed with small or without perturbation are easier for the model to learn. Furthermore, these less per- turbed features then become more predictive at the next training iteration. Thus, the inequality phenomenon does not occur during l/ la-avverarraal training. However, the 1g-robust model use both the object-relevant and object-imelevant features (e.g., background) for prediction at the
16

---

Published as a conference paper at ICLR 2023
game's equilibrium. In (Moayeri al. 2022), they show that 12-robust model is more sensitive to the background and other spurious features.
Model
Clean acc
Gini-g
Gini-r
Oce-B
Oce-G
Oce- W
Noise-I
Noise-II
Std. trained
76.13%
0.62
0.84
33.0%
28.1%
36.5%
15.8%
49.8%
12-AT model linf-AT model
56. 53%13%
0.60
0.76
51.4%
32.5%
48.3%
2.9%
36.3%
54.53%
0.95
0.97
81.5%
61.0%
77.2%
75.6%
96. 1%
As explained above, the inequality degree of l2-robust model is similar to the standard trained model (or le is even more equal). However, 12-robust model is still more vulnerable to occlusion. We guess it is because the most influential features tend to cluster together for both &oo at and (2-robust model.
A.6 COMPARED WITH MODELS TRAINED WITH SPARSITY REGULARIZATION
we perform experiments on -robuttt model and model trained with regularization for sparsity. We consider two types of sparse models: the model with sparse architecture and models with sparse weights. We compared 1x- -robutt model with models regularized by the following techniques:
l norm pruning (Li al.): It prunes filters by removing whole filters in the network gether with their connecting feature maps.
Weight pruning (Han et al. 2015): applies mask as regularization and sets the weights to be O. It results in sparser weights and connectivity patterns.
Model
Clean Acc
Gini g
Gini
Oce-B
Oce-G
Oce-W Noise-I
Noise II
Adv. trained
54.53%
0.94
0.97
78.2%
80.2%
60.2%
60.9%
95.1%
L1-norm sparsity 73.08%
0.60
0.80
56.8%
57.8%
52.5%
40.7%
89.2%
Weight-level Pruning 75.60%
0.62
0.81
33.1%
29.2%
25.8%
15.7%
49.9%
The sparsity of either model architecture or weights will not result in inequality on feature attri- bution. The -x- model is much more easily affected by occlusion and noise attack than the two sparse models. We think loc can be regarded as strong regularization: during advversarli training. the model attempts find the most robust feature against adversarial noise and discards the features which could be affected by added adversarial noise. With increasing the magnitude (c) of adversarial noise, only handful of features lox adversarially trained model can rely on for recognition.
A.7 EQUALITY DEGREE OF DIFFERENT CLASSES
We test the inequality degree of feature attributions' distribution of 50000 samples from 1000 classes in ImageNet. We present results in Table
Table 6: Global inequality degree of different classes (le-Adv, traind, = 8)
Class
Bustard
Manhole cover
Oystercatcher
Redshank
Pomeranian
Top-5
Gini
0.950
0.949
0.949
0.949
0.949
Class
Web site
Slot
Grocery store
Grille
Comic book
Bottom-5
Gini
0.890
0.900
0.901
0.901
0.905
Classes with top-5 inequality (Gini value) are: Bustard, Manhole cover, Oystercatcher, Redshank and Pomeranian. And their Gini values are 0.950, 0.949, 0.949, 0.949 and 0.949. Classes with bottom-5 inequality (Gini value) are: Web site, Slot, Grocery store, Grille and Comic book. Their corresponding Gini values are: 0.890, 0.900, 0.901, 0.901 and 0.905.
Regarding regional inequality, present results in Table 7 Classes with high regional inequality are similar to classes with high global inequality. Specifically. classes the top 5 (regional inequal- ity) are Redshank, American coot, Oystercatcher, Bustard and Gazelle. And the Gini,()) of these
17

---

Published as a conference paper at ICLR 2023
Table Regional inequality degree of different classes (lg-Adv. traind, 8)
Class
Redshank
American coot
Oystercatcher
Bustard
Gazelle
Top-5
Gini
0.976
0.975
0.975
0.974
0.974
Class
Web site
Grocery store
Confectionery
Slot
Grille
Bottom-5
Gini
0.929
0.932
0.934
0.935
0,936
classes are: 0.976, 0.975, 0.975, 0.974 and 0.974. Classes in the bottom 5 (regional incquality) are
Web site, Grocery store, Confectionery, Slot, and Grille. And the Gini,(-) of these classes are: 0.929, 0.932, 0.934, 0.935 and 0.936.
18

---

Published as a conference paper at ICLR 2023
A.8 DETAILS ABOUT FEATURE ATTRIBUTION METHOD
We use attribution methods including Input X Gradient, Smooth Gradient (short for SmoothGrad), Gradient Shapley Value (short for GradShap) and Integrated Gradients.
Input X Gradient:
The Input X Gradient multiplies input with the gradient with respect to each input feature. is a baseline approach for computing the attribution.
GradShap: Shapley Values aims compute each feature's attribution based on cooperative game theory. GradShap approximates Shapley Values by computing the expectations of gradients by randomly sampling from the distribution of baselines. It adds noise to each input sample Nsamples times, selects a random baseline from the baselines' distribution, and random point along the path between the baseline and the ipput. Then it computes the gradient of outputs with respect to those selected random points. In our evaluation, we set Rsamples 20 for experiments with ImageNet and Asamples 10 for experiments with CIFAR10. We set baseline = O for all the experiments.
Integrated Gradients is an axiomatic model which assigns an importance score each input feature by approximating the integral of gradients of the model's output with respect to the inputs along the path (straight line) from given baseline to inputs. Previous work points out Integrated Gradients method is sensitive to the choice of path. To reduce such sensitivity, Integrated Gradients are usually repeated for Nsttep steps. For all the experiments, we set Nstep = 20, and baseline (0,
Integrated Gradients:
SmoothGrad adds gaussian noise to each input in the batch Rsamples times, then applies the given attribution algorithm each of the samples. It returns the mean of the sampled attributions. SmoothGrad returns a sparser feature attribution map than other methods. In our ex-
SmoothGrad:
periment, set Vssamples
20
A.9 HOW RELEASE INEQUALITY PHENOMENON
We also try to propose strategy to release the inequality phenomenon in adversarial training. In- tuitively, we hope adversarially trained models learn to find robust features from the whole image rather than focus versarial training that Cutout enables the model to learn features from multiple spatial spaces. We evaluate our strategy CIFAR1O. The results are presented in Table g
specific robust feature. Towards this end, we incorporate Cutout with ad-
Gini
Gini-R
Clean
Ady. Acc. 48.50 & 47.20
Occlusion-B 100.00%
Occlusion-G 100.00% 29.98%
Occlusion-V 100.00%
Adv. trained
0.73
0.88
82, &
40.31 %
1.58%
Ady. trained-Cutout
93.39%
55.06
Table 8: -ddversaaaa training with and without Cutout.
As the table indicates, the Cutout strategy can slightly release the incquality of -dvversaaal train- ing. But at a price, both clean accuracy and adversarial accuracy slightly decrease. Regarding per- formance under noise and occlusion, the strategy does not improve the adversarial trained model's performance under noise but improves its performance under occlusions. A more effective strategy to release inequality phenomena is highly required.
19

---

Published as a conference paper at ICLR 2023
A.10 VISUALIZATION FOR CIFARIO
We visualize feature attribution maps and attack results for CIFARIO. About the setting of attack, set count N 10, with max radius R 4. With occlusion with different colors (black, white and grey), success rates on 1000 correct classified images of loe adversaaallllyy trained model are 60.4%, 60.5%, and 38.1% respectively. And success rates for the standard trained models are 34.6 %. 36.7% % and 24.1% 1% respectively. We visualize corresponding results in Figure g
Clean img.
Std. trained
1.0
2.0
4.0
8.0
Figure 7: Visualization of feature attribution CIFAR10.
Cleun img.
Tal
SUcceet
Pal
Fel
Succe
Fel
Adu. trained
SECORE
SOCOME
SUCORER
SecceeE
Figure 8: Visualization of occluded images for CIFAR10.
20

---

Published as a conference paper at ICLR 2023
A.11
MORE VISUALIZATION RESULTS FOR IMAGENET
0.68
0.73
0.79
0.89
0.94
0.68
0.74
0.79
0.88
0.93
0.61
0.62
0.70
0.84
0.90
0.62
0.77
0.83
0.91
0.95
0.62
0.69
0.74
0.85
0.91
0.69
0.77
0.86
0.92
0.59
0.70
0.75
0.86
0.91
Figure 9: Visualization of feature attributions generated by standard- and adversarially trained model with different c.
21


Reviews:
[{"review_id": "vxvgBh8TKRh", "reviewer": "Reviewer_yDNt", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper provides some insights on the vulnerability of l_{infty} adversarial trained model. The paper identifies inequality phenomenon occurs during l_{infty} adversarial training which is quantified by Gini index. To show the later, the paper proposes two methods: inductive noise and occlusion to demonstrate the vulnerability of l_{infty} adversarial trained model caused by such phenomenon. This paper provides a novel perspective and sheds light on the practicality of l_{infty} adversarial trained model.", "summary_of_the_review": "Interesting phenomenon, reasonable metric (Gini), well-motivated method that demonstrate the vulnerability of l_{infty} adversarial trained model.\nThis is a good paper, with some aspects of the presentation that should be improved.\n", "strength_and_weaknesses": "Pros:\n- A very interesting observation, inequality phenomenon occurring during l_{infty} adversarial training. \n- Provides a new perspective on feature representation of l_{infty} adversarial trained model, which will motivate future research.\n- Proposes regional and global Gini to evaluate the inequality phenomenon quantitively that providing an intuitive explanation. \n- Experiments show l_{infty} adversarial trained model is even more fragile than standard trained model under some scenarios, the results are interesting for me.\n \nCons:\n- Though this is a very relevant and timely work related to reliability of l_{infty} adversarial training, it would help if the authors could provide some effective solution to release such phenomenon or other suggestion.\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "rzQ16DTYBB", "reviewer": "Reviewer_Mxrb", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "Overall, the paper provides a novel and intuitive understanding about adversarial trained models. The paper identifies a new phenomenon and points out its unrealized threats which will inspire new related research.\nCorresponding index (Gini coeffiencient) is designed to illustrate such phenomenon.\n", "summary_of_the_review": "The paper undertakes an original approach to studying the inequality phenomenon of l_{infty} adversarial training and find new adversarial vulnerability. The phenomenon about adversarial training is novel, empirically insightful, and potentially will inspire further work. Overall, this new perspective is novel to me and I would tend to accept this paper. If the author can address my concerns. I would be more convinced.", "strength_and_weaknesses": "Strength:\n1. The strength of this paper is in the formulation of “inequality phenomenon” occurring during l_{infty} adversarial training. The paper provides an intuitive understanding about adversarial trained model’s behavior, and utilize a novel index to quantify such phenomenon. \n2. Using Gini index is an extra strength, the author also extends original Gini index to regional and global aspects which makes technical contribution. The proposed index is strongly correlate with the inequality phenomenon.\n3. Based on the observation of the phenomenon, the paper proposes several simple attack methods. The work identify unrealized threats brought by such inequality phenomena that $l_{\\infty}$-adversarially trained models are much more vulnerable than standard trained models under inductive occlusion or noise. It shows unrealized vulnerability of adversarial training, which can inspire new research in this area.\n \nWeakness:\n1. Current attack methods create perceptible occlusions on resultant images, it could be good to see more imperceptible occlusions for performing attacks.\n2. Writing can be improved at times. The paper could be further polished, and some typos exist.\n3. More visualization results are preffered.\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "oWwvZSKQtD", "reviewer": "Reviewer_AXew", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This work highlights the limitations of L-inf adversarial training - models trained using Linf-AT (specifically at large perturbation bounds) tend to rely heavily on very few features, when compared to normally trained models. This makes them vulnerable to attacks that perturb very few pixels, or occlusion based attacks, which are indeed more common in the real world. The authors propose two attacks that replace important features with either noise or occlusions to show the vulnerability of Linf adversarially trained models. ", "summary_of_the_review": "Although the paper makes some novel and insightful observations, I believe it is not yet ready for publication in terms of clarity and completeness of experiments to draw conclusions - details are mentioned in the weaknesses section. ", "strength_and_weaknesses": "Strengths -\n\n- The finding that Linf adversarial training makes models more vulnerable to noise and occlusion based attacks that perturb a few pixels, is novel and interesting. \n- The proposed attacks show the difference between Linf-AT models and standard trained models. \n\n\nWeaknesses -\n\n- There are several grammatical errors that severely impact readability.\n- Prior works have shown that gradients of adversarially trained models are perceptually aligned [1]. It is not clear why the feature attribution maps in Fig.1 are not perceptually aligned. \n- If the perturbation radius is controlled such that only the non-robust features are suppressed during adversarial training, even though the model would be more vulnerable to occlusion and perturbation attacks when compared to standard training, it would actually be more aligned to human perception. For example, in a waterbirds vs. landbirds classification, a normally trained model may correctly classify an image even if important features of the bird are completely occluded, because it considers background and texture for classification. Whereas, an adversarially trained model may rely more on features such as beak, eyes and shape of the bird, making them more vulnerable to occlusion attacks. Despite this increased vulnerability, Linf-AT model is more aligned to human perception and hence is more preferred.\n- There seems to be a typo in the paragraph above Eq.4. Sum of N least important features being similar to f(x) is considered as the criteria for inequality. This should have probably been the sum of N most important features I suppose?\n- Could the authors show how the proposed attacks generated from an adversarially trained model perform on standard models, and vice versa?\n- Could the authors confirm whether the standard trained models and adversarially trained models being compared in the paper use the same augmentations? This is important in light of the cutout augmentation results in A.3.\n- It is important to show what happens in case of other adversarial training regimes such as L2 norm adversarial training. \n\n\n\n[1] Tsipras et al., A. Robustness may be at odds with accuracy.\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "OyF7GEuo4Z1", "reviewer": "Reviewer_1FTk", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper studies the negative impacts of sparse features as a result of Linf adversarial training.  The authors find that this sparsity causes the Linf robust models to be more vulnerable to small occlusions and smaller magnitudes of random Gaussian noise in comparison to standard trained models.", "summary_of_the_review": "Overall, I think this is a very interesting work that investigates vulnerabilities that arise from Linf adversarial training.  The authors demonstrate these vulnerabilities by designing attacks against Linf adversarially trained models.  However, I think that there can be some improvements especially with regards to the clarity of the writing.  I also think that the authors should consider other baselines (specifically models trained with sparsity regularization) instead of just comparing to standard training.  This would allow us to better understand whether sparse features are the cause of these vulnerabilities or whether the observed vulnerabilities are unique to Linf adversarially trained models.", "strength_and_weaknesses": "Strengths:\nI think the experiments are interesting and highlight a weakness of Linf adversarial training well.\n\nWeaknesses:\nPaper clarity can be greatly improved, for instance the authors mention \"inequality phenomenon\" early in the paper, but it isn't clear what this means until after the authors describe using the Gini index. The authors should provide some short explanation of what they mean by \"inequality phenomenon\" early on in the text to improve clarity.  Some other parts I was confused about are \n- at the beginning of Section 2.3 what is meant by \"weak signals\"\n- in Section 3.1 \"In detail, we treat the input image...\" seems to be 2 separate sentences? or the second half of the sentence \"feature attribution methods aim to assign...\" should be deleted?\n- In section 3.1, in the definition of regional inequality, what exactly does a region mean?  Is it a set of pixels in the input space?  How are these regions chosen?\n- In 3.2.1, the text for the description of noise type 1 is confusing.\n\nWhat is the motivation behind using Gini index?  How much more informative is Gini index compared to just measuring sparsity?  Does sparsity not imply that Gini index will increase?\n\nDoes this inequality phenomenon arise due to more sparse features or is a result of something more specific to Linf adversarial training.  How successful are these occlusion and noise attacks on models trained with regularization for sparsity?\n\nAll visualizations are on ImageNet.  It would be nice to see some visualizations for CIFAR-10 as well.\n\nCould the authors provide clean and robust accuracies of models evaluated?\n\nTo confirm, the \"error rate\" is more of a measure of stability rather than error right?  If the model misclassifies the clean image and also misclassifies the perturbed image as the same class, then there is no error?  In that case, I think that the metric should be renamed to something involving \"stability\" rather than \"error\" to avoid confusion.  Another possible metric is to filter out the images that both the clean model and adversarially trained model are incorrect on and then measure error.  This would allow us to understand the frequency that a correctly classified image changes class which I think is a little more informative.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works."}, "extra_text": {}}]

Author rebuttal:

Decision: Accept: notable-top-25%