Distilling Robust and Non-Robust Features in Adversarial Examples by Information Bottleneck
Authors: Junho Kim
Byung-Kwan Lee
Yong Man Ro
Venue: NeurIPS 2021 Poster (2021)
Keywords: Adversarial Examples
Information Bottleneck
Feature Disentanglement
Feature Interpretation

Abstract: Adversarial examples, generated by carefully crafted perturbation, have attracted considerable attention in research fields. Recent works have argued that the existence of the robust and non-robust features is a primary cause of the adversarial examples, and investigated their internal interactions in the feature space. In this paper, we propose a way of explicitly distilling feature representation into the robust and non-robust features, using Information Bottleneck. Specifically, we inject noise variation to each feature unit and evaluate the information flow in the feature representation to dichotomize feature units either robust or non-robust, based on the noise variation magnitude. Through comprehensive experiments, we demonstrate that the distilled features are highly correlated with adversarial prediction, and they have human-perceptible semantic information by themselves. Furthermore, we present an attack mechanism intensifying the gradient of non-robust features that is directly related to the model prediction, and validate its effectiveness of breaking model robustness.

Distilling Robust and Non-Robust Features in Adversarial Examples by Information Bottleneck
Junho Kim," Byung- K Kwan Lee," Yong Man Ro
School Electrical Engineering Korea Advanced Institute of Science and Technology (KAIST)
farkimjh, leebk, ymarsskaist... Ckaist. ac ac.kr
Abstract
Adversarial examples, generated by carefully crafted perturbation, have attracted considerable attention in research fields. Recent works have argued that the exis- tence of the robust and non-robust features is a primary cause of the adversarial examples, and investigated their internal interactions in the feature space. In this paper, we propose a way of explicitly distilling feature representation into the ro- bust and non-robust features, using Information Bottleneck. Specifically, we inject noise variation each feature unit and evaluate the information flow in the feature representation to dichotomize feature units either robust or non-robust, based on the noise variation magnitude. Through comprehensive experiments, we demonstrate that the distilled features are highly correlated with advensarial prediction, and they have numan-pprcepple present attack mechanism intensifying the gradient of non-robust features that is directly related to the model prediction, and validate its effectiveness of breaking model robustness.
semantic information by themselves. Furthermore, we
Introduction
Deep neural networks (DNNs) have achieved remarkable performances in a wide variety of machine learning tasks. Despite the breakthrough outcomes, DNNs are easily fooled from adversarial attacks, with crafted perturbations 222 4 6. These perturbations are imperceptible human eyes, but simply adding them to clean images (i.e., adversarial examples) can effectively deceive classifiers. Such vulnerability affects security problems g 9 [T] [1], bringing in the weak reliability of DNNs.
Previous works have broadly investigated the reason for the widespread of such adversarial examples. Goodfellow et al. (2) have argued that adversarial vulnerability is induced from the excessive linearity nature of DNNs in sigh-dimensional spaces. Several works [2] have regarded the primary cause of the adversarial examples as statistical variation with aberrations in data manifold. Schmidt et al. [3 have suggested that the pervasiveness the examples should not be considered as a drawback of training methods for DNNs, since the available dataset may not be large enough to train them robustly.
In recent years. Tsipras et al. 14 have suggested an intriguing analysis that the disagreement between standard and adversarial accuracy stems from differently trained feature representation. In this literature, Ilyas ef al. [15] further have demonstrated the adversarial examples are inevitable results of standard supervised training and arisen from well-generalized features the dataset. They have analyzed the adversaria examples are originated from brittle and unintelligible features (i.e., non-robust features) that are arbitrarily manipulated with the imperceptible noise, and shown that the
Equual contribution. Corresponding author.
35th Conference on Neural Information Processing Systems (NcurIPS 2021).

---

robust features still can provide precise accuracy even in the existence of adversarial perturbation. They have argued that the non-robust features cannot show reliable accuracy in the adversarial setting and could provoke incomprehensible properties.
Nonetheless, the underlying reason for the existence and pervasiveness of adversarial examples cannot derive common consensus in the research field and still remains unclear [16]. To clarify where the adversarial brittleness truly comes from, we need figure out how the robust and non-robust features in data manifold subtly manipulate feature representation and fool model prediction, by directly handling them in the feature space. To address it, we propose a way to precisely distill intermediate features into robust and non-robust features by employing Information Bottleneck (IB) [17 18 [9]. In the sense that semantic information is included in the units of intermediate feature representation 120, 21 (3), we utilize the bottleneck to regulate the information flow in the feature space by explicitly adding noise to the feature units. Then, we estimate how each feature unit contaminated with the noise affects model prediction with assigned information. Based on the prediction sensitivity of the noise intervention, we assort the feature units either robust or brittle, and disentangle the feature representation into robust non-robust features, respectively.
Through extensive analysis of the distilled features, we corroborate that the pervasiveness of the adversarial brittleness is derived from the non-robust features, and they have a high correlation with the adversarial prediction. In addition, in order to understand the semantic information of distilled features, we directly visualize them in the feature space and provide their visual interpretation. Consequently, we reveal that both of the robust and non-robust features indeed have semantic information in terms of numan-perception by themselves. Based on our observation, we theoretically describe the negative impact of the non-robust features for the model prediction and introduce an approach of amplifying the gradients of non-robust features to break the model prediction.
In this paper, our contributions can be summarized into three-fold as follows:
We propose a novel way to explicitly distill intermediate features into the robust and non- robust features using Information Bottleneck, and interpret the disentangled features in terms of numan-perception by directly visualizing them in the feature space.
By analyzing how the distilled features affect the intermediate feature representation under adversarial perturbation, we demonstrate that the non-robust features are highly correlated with the adversarial prediction.
We present an attack mechanism manipulating the non-robust features by strengthening their gradients, and validate its effectiveness of breaking model prediction.
2 Distilling Robust and Non-robust Features in Intermediate Feature Space
Problem Setup and Notations. Let X denote clean images and Y denote (one-hot encoded) target labels corresponding to the clean images. Then, adversarial examples Xade can be created by the following equations: max F(x,Y)[[([C
+ 6), Y)), where 8 denotes an adversarial perturbation, and L denotes a pre-defined loss for machine learning tasks. The adversarial examples can be made by Xadv X + 6. When given model is adversarially trained against PGD attack [7], it can be written as follows:
min max F(x,Y) [C( + 8), Y)), 11
where w represents the parameters of f. which are learned to be robust against adversarial attacks. Here, Il-llxo < 7 describes Los norm, and y-ball means the perturbation magnitude. In this paper, we adversarially train the model on y 0.03 for the standard adversarial attack. Note that once adversarially trained, the parameters of the model f are no longer covered
As notation of variables that will use in this paper, Z and Z indicate the intermediate features of the model f such that Z = -f(X) and Z = f((Xadv), where f((-) describes I-th layer outputs of the given model. Similarly, fi+ represents subsequent network after the 1-th layer, thus intermediate
features can be propagated to the last output layer, such that Ý ft+ (Z) and Yude = f(+ (2). The
Previous works [4] [5] have demonstrated that the distinction between the robust and non-robust features arises in adversarial settings. In the sense that adversarially trained networks learn robust representation [24]. we set the robust classifier as default. Please the analysis of the standard training in Appendix F.

---

L
Given
Oz
Parameters
B + Lx(o)
C
X
ft
fi+
LCB
Z
Noise
W
inservention
0
Optimizing
7, f(0) +
E~N((0,1)
Parameters
Informative Feature
Figure Diagrams of our Information Bottleneck (IB), minimizing LCE + BL, to find noise variation that estimate information flow in the intermediate features Z. Here, dz represents nature feature variation of intermediate features Z for each unit, and e indicates Gaussian noise sampled from (0, I). More implementation details are described in Appendix A.
model f can be expressed as f = Ji+ o Jt. satisfying V = f(X) for the given clean images X. Also, Yado f(Xade) is denoted by model propagation of the adversarial examples Xade- Note that we designate 1-th layer as the last convolutional layer, and regard 1+ as the rest of the layers the model.
2.1 Information Bottleneck for Distilling Informative Features
In adversarial settings, we focus on separating robust and non-robust features in the intermediate layer. Recall that robust features are literally robust the noise (variation) and invariant the existence of the adversarial perturbation, but non-robust features are not. Our approach aims to distill feature units that affect target prediction under the noise perturbation in the intermediate feature space.
We follow that the semantic information is inherently included in the feature units of DNNs 20 22 25 From this perspective. utilize Information Bottleneck (IB) to distill the robust and non-robust features on the given intermediate features Z. Information Bottleneck [17 18] proposed to encode maximally informative representation for target labels, restraining input information, concurrently. Using the bottleneck, we suggest a way to assess feature importance and quantify information flow for the target prediction. The objective function of IB can be written follows:
max (ZZ X), 2
where I denotes mutual information, and 3 represents the degree of restraining input information. The first term [Z, Y) allows the intermediate features Z to be predictive on the target label Y, and second term I [2, encourages to compress the information of the given images in the bottleneck. Here, the second term requires a true feature probability p(2) = p(Z X))((X) (XXXX)XX to expand it, but is computationally intractable due to a high dimensional dependency of the dataset probability p(X). Thus, several works [18 19 modified the IB's objective function make possible to Iearn DNNs without the true feature probability as follows: min Loe + BL, (sce Appendix B). In this formulation, Leg indicates cross- cross-entropy loss, and L, represents information loss computed by KL divergence [26) between feature likelihood p | X) and an approximate feature probability (Z). It is radically a closed-form approximation for the true feature probability p(Z).
Firstly, we deliberately inject noise variation into Z to estimate the prediction sensitivity of each feature unit along the noise intervention. do so, we newly design an approximate feature probability using noise variation such that yo (Z) = (X)()) ??). Then, we sample random variables from qo (Z) and define informative features Z, as follows:
2, = f((X) ta.c.
where Z₁ ~ (Z). Note that the operator denotes Hadamard product, and stands for Gaussian noise sampled from (0. I). Here, the noise variation measures correlation between intermediate features and model prediction based on fact that robustness means high correlation on model prediction and non-robustness are opposite [4
15 in adversarial settings. Since the correlation
Fxx,y) (Y. [(X)) output layer can be expressed a variance measure Jox(Y,((X)) we consider the correlation in intermediate layer the noise variation. From this perspective, if feature unit is

---

highly predictive despite large noise variation (high correlation), the unit can robustly predict target labels, while a non-robust unit cannot.
Once the informative features 2 are acquired from the noise variation a, we propagate Z, to the last output layer and estimate the feature importance of each unit for model prediction. Then, we deal with the information loss C in order to alleviate feature heterogeneity between Z and Z,. Through aforementioned modification [8 19 our objective function can be written as follows:
min Lx(a) = lo fit (f(() + +0.0) +,3 DKL(P(Z | x) 11 20 (Z)),
(4)
CoB
Cr
where the feature likelihood p(Z X) set to (X), o2). Here, a= indicates inherent feature variation of the intermediate features Z for each unit. The second term L, makes the informative features Z, resemble the intermediate features Z, while minimizing the cross-entropy LCg- This
1], where & denotes an index of =x + +log of dz
second term can be written as LJ = 2 2===1
of
the noise variation = [0i, 02.-.. ac] (the optimizing parameters) and the feature variation a= - [=1: O221" [zc] (the given parameters). Here, only of the variation is updated such that 4- o (x). In brief, we summarize overall procedure of our bottleneck concept in Fig.
Moreover, we mention that ,3 in Eq. (4) controls the amount of information that flows into the feature representation. Specifically, when 3 is set to zero, IB loss is equivalent cross-entropy loss, which means that Z, can accommodate even unimportant features to predict target labels. In contrast, excessively large 3 only focuses compressing input information, thus IB may cannot filter out important features to predict target labels. Accordingly, we empirically control 3 to distill informative features Z based on the noise variation (Please section 3.4 for analysis of information flow).
2.2 Separating Informative Features by Tolerance of Feature Variation
After optimizing the informative features Zr, we compare the noise variation a for the informative feature units and dichotomize each unit either robust or non-robust based their prediction sensitivity. We set the criterion for comparison as T = max(o") o²). Here, I' represents the maximum tolerance of the noise variation. is reasonable choice to set 7' as criterion, because it indicates the maximal variation with respect to the changes of the given image X, in the feature space. In the following procedure, explicitly disentangle intermediate features Z into the robust Z, and non-robust features Znr-
Firstly, once the noise variation is larger than the maximum tolerance T in specific units, it indicates that their corresponding features are highly predictive on the model prediction, despite the noise intervention. Thus, we define their conjunction as robust features. On the other hand, if the variation of a specific unit is smaller than T, their corresponding features can be represented as non-robust features. This is because the small variation behaves strict restriction to retain model prediction of target labels. We assume that once a strong adversarial perturbation comes in, the feature variation of non-robust features becomes to be larger than acceptable tolerance, thereby easily breaking the model classifier and leading to misclassified prediction. In this respect, we sort the noise variation according to their magnitude, and cluster them by assigning robust or non-robust channel indexes to each feature unit. The robust channel index, i, = [ir, fr2s" ... irel can be computed as follows:
{
[1 0² > T
irc L(o T)
(5)
0² ST
where 1(-) represents the indicator function. The non-robust channel index inr is simply reversed from the robust channel index such that inzg = i Then, we estimate robust features 2, by multiplying the robust channel index to the intermediate features element- wisely such that Z, i, Zz Similarly, non-robust features Znr are presented as Z,r = inr Z. In this way, the intermediate features Z are fully disentangled into the two types of feature representation satisfying Z = Z, Z
To sum it up, we regard the robust features Z, that have the larger noise variation as invariant features from the adversarial perturbation. On the other hand, non-robust features Zar are considered as easily manipulated features, which harmonize the smaller noise variation. Now, we analyze their impacts to the robustness by expanding the model prediction of Z7 to Taylor approximation (sce Appendix C.)

---

Table 1: Classification accuracy of model performance attacked by FGSM (2]. PGD 7. and CW 4 on VGG-16 22 and WRN-28- [3], adversarially trained with ? 0.03 for CIFAR- 10. SVHN and Imageem.aeNet. We selectively propagate each feature (i.e., intermediate features (Int.), robust (R.), and non-robust features (NR.)) to measure classification accuracy.
CIFAR-10
SVHN
Tiny-ImageNet
Model Example
Int. Acc
R. Acc
NR. Acc
Int.
Acc
R. Acc
NR. Acc
Int. Acc
R. Acc
NR. Acc
Clean
79.73
99.87
34.82
90.35
99.76
57.09
33.98
83.11
7.50
FGSM 2
51.28
99.58
22.82
63.71
98.72
40.12
17.45
77.74
4.94
VGG
PGD 7
44.71
99.38
20.64
48.92
97.91
32.18
16.13
77.59
4.69
CW 4
40.32
99.85
13.66
33.26
99.60
16.75
12.00
75.69
4.03
Clean
82.56
98.66
44.67
93.53
99.44
70.43
43.13
96.35
6.07
FGSM 2
56.43
96,80
31.65
73.93
97.90
53.96
20.38
91.58
3.01
WRN
PGD
51.63
96.61
29.44
61.09
96.33
45.79
18.84
90.37
2.83
CW
45.47
97.74
17.28
40.61
97.58
22.78
13.51
95.78
2.06
2,
Z
2,
Zur
(a) Clean Examples
(b) PGD Examples
Figure 2: The result of t-SNE plot 31 in CIFAR-I0 dataset for VGG-16 network. Each cluster
indicates high-dimensional distributions of feature representation for 10 object labels in CIFAR-10 dataset. Additional t-SNE results for other adversarial attacks illustrated in Appendix D.
with its convergence of local minima as follows:
a Oo, fi+
1 Onr
+0 ()))
+ t..
+)+
+ (x()
(6)
where robust noise variation o, = i, a and non-robust noise variation our far a. Since the variation of the robust features does not degrade the model prediction, when our is small enough, the erroncous term A in Eq. (6) closes to zero. That is, the model retains having significant robustness against the adversarial perturbation interrupting robust channel index in Z. Conversely, once force our to increase, its output becomes inaccurate for the robust prediction (i.e., first term in Eq. 6 due to high A. Here, we theoretically demonstrate how the brittleness of non-robust features affects the model robustness. We will thoroughly analyze the properties of the two distilled features by empirically showing the robustness of Z, and brittleness of Znr in the following sections.
Analysis of Distilled Features and Visual Interpretation
3.1 Property of Distilled Feature Units under Adversarial Perturbation
After we distill the robust and non-robust features using the bottleneck concept, our next question is How will the target prediction change under the adversarial attacks? In our posit, if the bottleneck successfully disentangles the robust and non-robust channel index (i.e., and in,)) from the given examples, we should identify the consequential classification accuracy changes under the adversarial perturbation. That is, after applying the robust index the attacked feature representation, the selected adversarial features with i, denoted by 2, (i.e., Z, = i, Z) should have invariant accuracy changes for the target labels. Here, i, is robust channel index obtained from the given clean examples X, and Z represents the intermediate features of the adversarial examples, which means 2 = =[(Xad))) Contrarily, the selected features satisfying Znr inr will show inaccurate accuracy due their brittleness under the existence of adversarial perturbation.
5

---

Clean Img
Int. Feature
R. Feature
NR. Feature
Adv Img
Int. Feature
R. Feature
NR. Feature
CIFAR 10
Bird
2: Bird
Z,: Bird
Zur: Deer
Pade: Deer
Z: Deet
Zr: Bird
Z Deer
A
SVHN
Y:4
2:4
Zy:4
Zar:
Pacte: 1
2:
2P
Zür:
Tiny Imageehee
Y: Ladybugs
Ladybugs
Zy: Ladybugs
Zu Sulfur Butterlly
Peds: Sulfur Butterfly
Sulfur
Z Ladybugs
2 Sulfur Butterfly
Butterfly
(a) Clean Examples
(b) PGD Examples
Feature visualization 25 for the intermediate feature (Int.), robust feature (R.), and non-robust feature (NR.). The class labels under each image indicate the predicted results of the corresponding features propagated by (-) Note that the visualization of the non-robust features displays semantic similarity of the misclassified classes of the adversarial examples. Please see more visualization results in Appendix E.
Figure
In Table T analyze evaluation results of the disentangled features under standard attack algo- 33 34]. As aforementioned, we apply the robust
rithms 4 in publicly available datasets [3] and non-robust channel index optimized from the clean examples to the adversarial features Z, and estimate their accuracy (i.e., fi+ (Zz) and fi4 (Znr)) As in the table, Z, still shows constant robust accuracy regardless of the adversarial perturbation, even the high-confidence adversarial attack 44) On the other hand, we can find that the classification accuracy of Zur steeply degrades as the attacks get stronger, which coincides with the properties of the robust and non-robust features. To further support our experiments, we illustrate the correlation between the disentangled features and true labels, using 2D t-SNE plot [1 In the case of clean examples as in Fig. (()),, the robust features 7, exhibit separable clusters on the target labels, while the non-robust features 7nr show partially disorganized tendency. When adversarial perturbation 7 exists, the attacked features Znr represents more collapsed t-SNE visualization as shown in Fig. (b). Notably, sustain highly clustered results even in the attacked condition.
observe that Z, still
3.2 Feature Visualization of Robust and Non-robust Features
We have identified the existence of the robust and non-robust features using the bottleneck. Then, wonder about way interpreting the semantic information in the feature space with respect to human-perception. Analyzing semantic representation of DNNs is wide research area to understand their decision 20 22 35). In adversarial settings, several studies [14] 36 argued
loss gradients in the input space. Engstrom et al. (24] further endeavored to interpret robust feature representation using feature visualization (25 37 38 In this manner, explore whether the disentangled feature representation from the bottleneck indeed has human-percepprbbe information in the intermediate feature space.
that robust classifier has more meaningful (i.e., perceptually--lignel
Feature visualization is optimization-based method that maximizes specific activation of feature units [38 39), such that X' = argmax (X))), where (-(-) indicates feature activation of i-th unit in the (-th layer. To understand what conjunction of the robust and non-robust feature units truly interacts with the target labels, we optimize each distilled feature and create their visual explanations. We adopt direct visualization method [25] that has various regularization techniques (e.g., frequency penalization and transformations) to yield better representative visual quality.
We employ 2 and in for the adversarial examples Xade. which is obtained by optimizing Lx.a. (a) instead of Eq. (4). Then, we define Z and 2g as robust and non-robust features the adversarial examples, satislying Ze = ii 7 and - "n Z. After distilling robust and non-robust features with their corresponding index, we maximize the selected feature unit activation, respectively. The feature visualization results of distilled features illustrated in Fig.

---

Table 2: The prediction accuracy of the non non-robust features 2 for attacked labels Yau-- The input Z is the non-robust features of the corresponding attack methods. To clearly show the correlation between ad- versarial examples and the non-robust features, we evaluate the accuracy under the condition of success- fully attacked examples (i.e.) Y # Yudv).
R.
Adv. Img
Information Bottleneck
Distilligg Features
Z, Pomegranate
Y: Pomegranate
Pedu: Bell Pepper
Attack
CIFAR-I0
SVHN
Imageem
High Correlation
Z Bell Pepper
VGG
WRN
VGG
WRN
VGG
WRN
Figure 4: An example of highly correlated ad-
FGSM [2]
92.78
94.35
94,90
96.13
63.39
60.82
PGD 7
93.43
95.06
96.21
96.44
65.84
63.64
versarial prediction with NR. Both Yodu and
CW 9
93.72
94.42
95.75
97.65
55.68
56.84
Zür output same prediction "Bell Pepper".
As in the figure, we can clearly recognize the semantic information of true labels Y and attacked predictions Yodo in the intermediate features (Z and Z). Interestingly, what we can observe is: (i) the distilled features have semantic information by themselves and maintain their information, even under the adversarial perturbation, (ii) when the adversarial perturbation exists, the brittleness of non-robust features is intensified and reflected onto Z. Thus, the visualization of Z and Z% looks similar, and they manipulate the target prediction to same adversarial prediction. Unlike the previous work [15] that has argued the non-robust features solely have incopprchensible property, the visualization of the distilled features from our bottleneck represent recognizable outputs even for the adversarial examples, and provides decisive key interpret the cause of adversarial examples.
3.3 Adversarial Prediction is Highly Correlated with Non-robust Features
We have observed that the non-robust features optimized from the bottleneck are brittle and easily manipulated under the adversarial perturbation, while robust features maintain substantial prediction results for the target labels. Then, if the primary cause of the adversarial examples indeed belongs to non-robust features, it is natural to examine the correlation between the classification outputs of the non-robust features and the adversarial prediction induced by adversarial attacks. Accordingly, we identically apply our IB loss on the adversarial examples Xado 2 4. and find their corresponding robust and non- non-robust channel index using Eq. (5).
To enlighten the correlation of non-robust features and the adversarial prediction, we evaluate the model prediction of " for the attacked labels Yadv that can be written as follows: y = fi+ (Za,) We set the condition of Yodo successfully attacked labels (i.e., Y * fXXadv) to definitely show the relationship between the prediction Y; of non-robust features and the adversarial prediction Yads of adversarial examples. In Table 2 we summarize the accuracy of y for the successfully attacked label Yado in standard attack methods. Generally, we can observe that the non-robust features are highly predictive on Yado in standard low dimensional datasets such as CIFAR-I0 and SVHN. Even the non-robust features are remarkably correlated with the attacked prediction. A brief explanation of the highly correlated example is described Fig. 4
in large dataset (i.e.,
3.4 Bottleneck Controls Information Flow of Robust and Non-robust Features
In this analysis, we will investigate how the bottleneck affects the information flow of the robust and non-robust features and clarify their relation. Recall that the bottleneck refines informative features from the given image samples, and 3 regulates the total amount of the information that flows into Z. We will compare classification accuracy for the robust and non-robust features along & value and analyze the changes of information flow assigned to each disentangled feature.
In Fig. 5] as .9 value increases, the accuracy of the robust features is getting higher and decreases after a specific threshold. As theoretically mentioned in 2.1 we can infer that suitable choice of the 3 can filter out robust feature units in the intermediate layer. For the excessive B value, we can observe that the bottleneck cannot accurately disentangle adversarial features. For example, the accuracy of the robust and non-robust features are reversed after 3 = 5.0 in the particular networks of CIFAR-10 10 and SVHN datasets. In addition, as in Fig. 5] and (b), the accuracy of the non-robust features progressively increases. Such results indicate few robust feature units that not distilled
7

---

Table 3: Comparing attack performance for FGSM 22 BIM 40), PGD M. CW 4. AutoAttack
(AA) ІЛ, FAB 42), and non-robust feature attack denoted by NRF We adversarially train VGG-16
and WRN-28- 10 on Te norm 7 0.03 perturbation for CIFAR-10, SVHN, and Tiny-ImageNet with
PGD adversarial training (ADV) and advanced defense methods: TRADES 44 and MART
VGG-16
WRN-28-18
Dataset
Method
Clean
FGSM
BIM
PGD
CW
AA
FAB
NRF
Clean
FGSM
BIM
PGD
CW
AA
FAB
NRF
ADV
79.7
51.3
46.5
44.7
40.3
42.0
40.9
27.4
82.6
56.4
52.8
51.6
45.5
49.8
49.0
17.1
CIFAR-IO
TRADES
78.2
54.5
51.7
50.9
43.0
49.5
46.3
31.2
83.0
57.9
55.0
53.9
46.7
52.4
49.8
26.8
54.2
52.2
51.7
50.6
83.4
56.0
54.7
52.8
50.2
19.6
ADV
90.4
63.7
52.1
48.8
33.3
39,9
41.6
12.6
93.5
73.9
64.8
61.1
40.7
55.5
56.6
13.4
SVHN
TRADES
90.4
65.3
59.0
57.0
44.8
53.5
50.0
14.3
93.9
72.9
63.9
60.4
42.0
55.0
55.4
10.1
MART
90.5
65.1
59.7
57.8
46.4
53.0
47.0
94.1
73.0
64.5
61.1
42.3
55.4
56.0
ADV
34.0
17.5
16.5
16.1
12.0
15.4
12.
43.1
20.4
19.3
18.8
13.5
18.1
142
5.3
Tmy-InageNeet
TRADES
38.7
20.1
19.1
18.7
13.9
17.8
13.3
7.8
47.2
26.7
25.6
25.2
17.4
24.4
17.7
9.6
MART
38.4
20.6
19.5
19.1
14.1
18.3
14.7
9.2
48.5
27.4
26.1
25.7
17.5
25.0
17.8
9.9
CIFAR- 10 Dataset
Bidak Anc
Bahad Act
800
Preditaion ArznnNy
Rabea
Preficiin
300
0- WisH Tatool
64
6.1
0.5
0U7
1.0
3.0
1/8
30.5
0.3
3.5
0U7
2.0
3.
2.0
7.0
13.9
0.3
03
0.2
60
1.0
34
74
16.2
Budnasl Auc
Datae Art
000
SVHN Dataset
Prediction Accumey
Prwdiatin keaney
Raber
300
Wite Tabeat
64
617
160
45
R77
10
3.5
555
0.3
233
0.5
6.7
1.0
340
5.0
74
36.2
(c) # of Assigned Channels Figure 5: The accuracy of robust and non-robust features along 33 value, and the number of assigned channels in the 1-th feature representation. Note that each color the lines (c) corresponds to the same color in the bar plots of (a) and (b). The total number of channels is equivalent to the size of C.
(a) VGG-16
(b) WRN-28-10 10
from IB are gradually flowing into the non-robust side and dichotomized as non-robust units, thus producing more higher accuracy. It coincides with the analysis of the number of the assigned robust and non-robust channels in Fig. c)... We can observe that the number of robust channels constantly diminishes, whereas that of the non-robust channels increases. It indicates the bottleneck delicately weighs the information flow that should be precisely assigned to the robust or non-robust units.
From our analysis section, in conclusion, we have demonstrated the existence of the distilled features using the bottleneck: the robust and non-robust features the intermediate feature representation. In addition, we have revealed that easily manipulated property of non-robust features is the primary cause of adversarial examples through their high correlation with the adversarial prediction. Based on the fact that the non-robust features break the model prediction, we will suggest an effective way of enhancing the adversarial attack, utilizing the gradient of non-robust features in section
Amplifying Brittleness of Non-Robust Features for Attack
In this section, we intentionally increase the non-robust noise variation our to make model classifier fooled based on Eq. (6). Note that increasing our induces high erroneous term A, thereby going to deviate from the robust prediction. However, this variation dur is merely an optimizing parameter that cannot be controlled manually. Thus, in order to secondarily have the effect of enlarging dur- alternatively utilize the gradients of the non-robust features directly connected with the model
8

---

Table 4: Comparison of classification accuracy for adversarial examples generated by maximizing (1) or minimizing (L) the magnitude of robust (9,) or non-robust feature gradients (Gnr).
VGG-16
WRN-28-10
Dataset
Method
Clean
| la
Oue laa
|| la
[ L
Clean
Gar laa
1| la 4|
| ll 4
ADV
79.7
27.4
67.5
35.9
74.6
82.6
17.1
74,8
28,9
79.5
CIFAR-10
TRADES
78.2
31.2
71.6
38.2
77.1
83.0
26.8
73.9
30.3
79.9
MART
73.5
31.4
63.8
39.6
69.1
83.4
19.6
74.3
24.5
79.4
ADV
90.4
12.6
71.9
20.8
71.5
93.5
13.4
88.0
15.6
93.4
SVHN
TRADES
90.4
14.3
68.4
27.3
82.3
93.9
10.1
88.1
13.2
93.3
MART
90.4
16.1
66.2
31.6
84.9
94.1
8.1
87.7
9.0
91.5
ADV
34.0
6.7
26.1
9.7
29.0
43.1
5,3
38.9
15.5
39.9
Tiny-lmageNel TRADES
38.7
7.8
30.7
11.8
33.8
47.2
9.6
40.9
16.9
44.4
MART
38.4
9.2
29.0
13.4
32.5
48.5
9,9
41.3
17.2
45.7
prediction. The gradients of non-robust features in adversarial examples can be described as follows:
Gar ①
TZnr Lrase (s (f(X + o),
where we define a baseline loss as Lsase (f()))(X))))) = ||0|| + c. max(max( ((X); 0),
FFY
instead of cross-entropy loss due to its empirical effectiveness of attack performance 4 In addition, we use a technique of changes of variables 4 from to w for generating an imperceptible yet powerful perturbation, such that 5 2 (tanh(w) + X. It serves to smooth out projected gradient descent that clips prematurely to prevent adversarial examples falling into the extreme image domain.
To compute the gradient practically, we firstly calculate 2 Lbases and multiply az rule. Here, the latter gradient equals to non-robust channel index inr, because the intermediate features of adversarial examples can be re-written as: 7 i, Zy inr Znr. and the derivative of 7. over Znr equals to hn-- Thus, the gradients of the non-robust features 9nr can be simplified as fnr .22 Lbase. Using the gradient, we suggest an attack to non-robust features (NRF) by optimizing the following objective:
@ Z by chain aZ.r
2
min Lrase (achas([[( (f(X +0, Anr
az Lbase
In Table NRF shows more effective attack performance than the other standard adversarial attacks in 7 43 since NRF strengthens the gradients of non-robust features that contains the same effect of increasing our to disturb accurate model prediction.
Moreover, we conduct an ablation study on the gradients of robust and non-robust features to probe their influence of prediction changes in Table 4 As expected, maximizing Gn. shows more effective attack performance than minimizing it. because maximizing Gur has an alternative effect of increasing the non-robust noise variation our (i.e., large erroneous term A f) in Eq. (6). Whereas, manipulating the gradients of robust features G, shows less attack performance than controlling Gnr- since it cannot directly handle brittle features in the intermediate feature representation. Especially, seems difficult to break model prediction, while weakening the gradient of robust features, whose noise variation a, have invariance of target prediction even under the adversarial perturbation.
5 Related Work
Various works have been tried to figure out the reason for adversarial vulnerability in the intermediate feature representation. Inkawhich et al. [45] analyzed how the intermediate features are changed by adversarial attacks and measured layer- and class-wise feature distributions. Engstrom et al. [24] pointed out that there is a shortcoming of DNNs and their embedding, that is, the primary features used in DNNs are contrasting with what human uses. Also, they argued that the robust optimization to learn robust features could address this shortcoming by encoding high-level representations of input data. Jacobsen et al. [46] argued that the reason for adversarial vulnerability lies in invariant characteristics of DNNs to task-relevant features. This invariance makes most regions of input space brittle to adversarial attacks so that the classifiers become relying
few highly predictive features. In addition, recent works 14 15 have suggested that the existence of the vulnerability is on the non-robust features, which are inherently included in the data and have unrecognizable properties.

---

Our work is in line with the concept of the non non-robust feature. However, unlike the aforementioned works that directly generated the robust and non-robust datasets to analyze their properties, we reveal that the robust and non-robust features can be completely disentangled in the feature space using Information Bottleneck, and they have semantic information by themselves in fact.
Conclusion and Discussion
Conclusion. To understand where the adversarial brittleness comes from, we have investigated information flow in the intermediate feature space, using Information Bottleneck. By estimating the feature importance of each feature unit based on the added noise variation, we introduce a novel method to explicitly distill the robust and non-robust features in the feature representation. Through extensive analysis of the distilled features, we figure out how the feature units interact with target labels, and corroborate that the non non-robust features are highly correlated with the adversarial prediction. In addition, we directly visualize the distilled features the intermediate feature space and reveal that they have recognizable semantic information by themselves. Based on the properties of the distilled features, we suggest non-robust feature attack (NRF) utilizing the brittleness the feature space.
The adversarial examples, including our work, can potentially cause negative impacts on various machine learning applications, such as autonomous driving [47] and medical image process 48 However, by analyzing the nature of robust and non-robust features that can be primary cause of the adversarial examples, our work contributes important societal impacts in this research field. In this paper, we propose a way of utilizing the gradient of brittle features in the intermediate feature representation (NRF). In future works, we hope to bridge the gap of deploying distilled features into diverse applications such as adversarial detection and defense strategies. Moreover, we would like to note that the last convolutional layer is set to distilling criterion for Information Bottleneck, since the high-level concepts are included in the higher layer of DNNs. In this sense, it seems reasonable choice, but how the robust non-robust features in the lower dimension interact with the much higher dimensional feature units can be another undisclosed future direction to investigate.
Discussion.
Acknowledgments and Disclosure of F Funding
This work was conducted by Center for Applied Research in Artificial Intelligence (CARAI) grant funded by DAPA and ADD (UD190031RD).
References
[1] C. Szegedy, W. Zaremba, I. Sutskever, J. Bruna, D. Erhan, I. Goodfellow, and R. Fergus, "Intriguing properties of neural networks," in International Conference on Learning Representations (ICLR), 2014.
[2] J. Goodfellow, J. Shlens, and C. Szegedy, "Explaining and hamessing adversarial examples," in Interna- tional Conference on Learning Represeniations (ICLR), 2015.
[3] P.-Y. Chen, H. Zhang, Y. Sharma, J. Yi, and C.-J. Hsieh, "Zoo: Zeroth order optimization based black- box attacks to deep neural networks without training substitute models," ACM Workshop on Artificial Intelligence and Security, 2017.
[4] N. Carlini and D. Wagner, "Towards evaluating the robustness of neural networks," in 2017 IEEE Sympo- sium Security and Privacy (SP), pp. 39-57, IEEE Computer Society, 2017.
[5] N. Papernot, P. McDaniel, I. J. Goodfellow, S. Jha, Z. Y. Celik, and A. Swami, "Practical black-box attacks against machine learning." ACM ASIA Conference Computer and Communications Security (ACM ASIACCS), 2017.
[6] K. Eykholt, I. Evtimov. E. Fernandes, B. Li, A. Rahmati, C. Xiao, A. Prakash, T. Kohno, and D. Song. "Robust physical- world attacks on deep learning visual classification," in Conference on Computer Vision and Pattern Recognition (CVPR), pp. 1634,1634, 2018.
[7] A. Madry, A. Makelov, L. Schmidt, D. Tsipras, and A. Vladu, "Towards deep learning models resistant to adversarial attacks." in International Conference on Learning Representations (ICLR), 2018.
10

---

[8] G. Apruzzese, M. Colajanni, L. Ferretti, and M. Marchetti, "Addressing adversarial attacks against security systems based on machine learning," " in International Conference on Cyber Conflict (CyCon), vol. 900,
pp. 1-18. IEEE, 2019.
[9] X. Wang. J. Li, X. Kuang, Y. Tan, and J. Li, "The security of machine learning in an adversarial setting:
A survey." Journal of Parallel and Distributed Computing, vol. 130. pp. 12 22,,,32111 2019.
[10] Y. E. Sagduyu, Y. Shi, and T. Erpek, "lot network security from the perspective of adversarial deep learning." in International Conference on Sensing, Communication, and Networking (SECON), pp. 1-9, 2019.
[11] I. Rosenberg. Shabtai, Y. Elovici, and I.. Rokach, "Adversarial learning in the eyher security domain," arXiv preprint arXiv:2007.02407 2020.
[12] A. Shafahi, W. R. Huang, C. Studer, Feizi, and Goldstein, "Are adversarial examples inevitable?," in International Conference on Learning Representations (ICLR), 2018.
[13] L. Schmidt, S. Santurkar, D. Tsipras, K. Talwar, and A. Madry, "Adversarially robust generalization requires more data," in Neural Information Processing Systems (NeurlPSS, 2018.
[14] D. Tsipras, S. Santurkar, I.. Engstrom, A. Turner, and A. Madry, "Rohustness may he at odds with accuracy," in International Conference on Learning Representations (ICLR), 2019.
[15] A. Ilyas, S. Santurkar, D. Tsipras, L. Engstrom, B. Tran, and A. Madry, "Adversarial examples are not bugs, they are features," in Neural Information Processing Systems (NeurIPS), vol. 32, 2019.
[16] L.. Engstrom, Gilmer, G. Goh, D. Hendrycks, A. Ilyas, A. Madry, R. Nakano, P. Nakkiran, S. Santurkar, B. Tran, D. Tsipras, and E. Wallace, "A discussion of "adversarial examples are not bugs, they are features"," Distill, 2019. https:/ddsttill.pubb/
[17] N. Tishby, F, C. Pereira, and W. Bialek, "The information bottleneck method," in Proc, of the 37-th Annual Allerton Conference Communication, Control and Computing, pp. 368-377, 1999.
[18] A. Alemi. I. Fischer, J. Dillon, and K. Murphy. "Deep variational information bottleneck," in International Conference on Learning Representations (ICLR), 2017.
[19] K. Schulz, L. Sixt, F. Tombari, and T. Landgraf. "Restricting the flow: Information bottlenecks for attribution." " in International Conference Learning Representations (ICLR). 2019.
[20] M. D. Zeiler and R. Fergus, "Visualizing and understanding convolutional networks," in European Confer- ence on Computer Vision (ECCV), pp. 818-833, Springer, 2014.
[21] J. Yosinski, J. Clune, T. Fuchs, and H. Lipson, "Understanding neural networks through deep visualization," in International Conference on Machine Learning Workshop on Deep Learning, Citeseer.
[22] D. Bau, B. Zhou, A. Khosla, A. Oliva, and A. Torralba, "Network dissection: Quantifying interpretability of deep visual representations," " in Conference on Computer Vision and Pattern Recognition (CVPR), pp. 6541-6549, 2017.
[23] C. Olah, A. Satyanarayan, I. Johnson, S. Carter, L. Schubert, K. Ye, and Mordvintsev, "The building blocks of interpretabilityy" Distill, 2018. https://distill.pub/00 8/building- blocks.
[24] L. Engstrom, A. Ilyas, Santurkar, D. Tsipras, Tran, and A. Madry, "Adversarial robustness as prior for learned representations," arXiv preprint arXiv:1906. 2019.
[25] C. Olah, A. Mordvintsev, and L. Schubert, "Feature visualization." " Distill, vol. no. 11, p. e7, 2017.
[26] S. Kullback and R. A. Leihler, "On information and sufficiency," The annals of mathematical statistics, vol. 22, no. 1, pp. 79-86, 1951.
[27] C.-J. Gabrie,, Y. Ollivier, L. Bottou, B. Schölkopf, D. Lopez-Paz, "First-order adversarial vulnerability neural networks and input dimension," in International Conference on Machine Learning (ICML), pp. 5809-5817, PMLR, 2019.
[28] C. Qin, J. Martens, S. Gowal, D. Krishnan, K. Dvijotham, A. Fawzi, S. De, R. Stanforth, and Kohli, "Adversarial robustness through local linearization," in Neural Information Processing Systems (NeurIPS), vol. 32, 2019.
[29] K. Simonyan and A. Zisserman, "Very deep convolutional networks for large-scale image recognition," in International Conference Learning Representations (ICLR), 2015.
11

---

[30] S. Zagoruyko and N. Komodakis, "Wide residual networks." in British Machine Vision Conference (BMVC). 2016.
[31] L. Van der Maaten and G. Hinton, "Visualizing data using t-sne.," Journal of machine learning research, vol. 9, no. 11, 2008.
[32] A. Krizhevsky, G. Hinton, et al., "Learning multiple layers of features from tiny images," 2009.
[33] Y. Netzer, T. Wang, A. Coates, A. Bissacco, B. Wu, and A. Y. Ng, "Reading digits in natural images with unsupervised feature learning," 2011.
[34] Y. Le and X. Yang, "Tiny imagenet visual recognition challenge," " CS 231N, vol. 7. 7, 2015.
[35] R. Fong and A. Vedaldi, "Net2vec: Quantifying and explaining how concepts are encoded by filters in deep neural networks," in Conference on Computer Vision and Pattern Recognition (CVPR), pp. 8730-8738, 2018.
[36] S. Kaur, J. Cohen, and Z. C. Lipton, "Are perceptually-aligned gradients general property of robust classifiers?," in NeurlPS Workshop: Science Meets Engineering of Deep Learning (NeurIPS SEDL., 2019.
[37] D. Erhan, Y. Bengio, A. Courville, and P. Vincent, "Visualizing higher-layer features of a deep network," University of Montreal, vol. 1341, no. 3, p. 1, 2009.
[38] A. Nguyen, J. Yosinski, and J. Clune, "Understanding neural networks via feature visualization: A survey," in Explainable Al: interpreting, explaining and visualizing deep learning, pp. 55-76, Springer, 2019.
[39] A. Mahendran and A. Vedaldi, "Understanding deep image representations by inverting them," in Confer- ence Computer Vision and Pattern Recognition (CVPR). pp. 5188-5196, 2015.
[40] A. Kurakin, I. J. Goodfellow, and S. Bengio, "Adversarial machine learning at scale," in International Conference on Learning Representations (ICLR), 2017.
[41] F. Croce and M. Hein, "Reliable evaluation of adversarial robustness with an ensemble of diverse parameter- free attacks," in International Conference on Machine Learning (ICML), 2020.
[42] F. Croce and M. Hein, "Minimally distorted adversarial examples with a fast adaptive boundary attack," in Proceedings of the 37th International Conference Machine Learning (H. D. III and A. Singh, eds.), vol. 119 of Proceedings of Machine Learning Research, pp. 2196-2205, PMLR, 13 13-18 Jul 2020.
[43] H. Zhang, Y. Yu, J. Jiao, E. Xing, L. El Ghaoui, and M. Jordan, "Theoretically principled trade-off between robustness and accuracy," International Conference on Machine Learning (ICML), pp. 7472-7482, PMLR, 2019.
[44] Y. Wang. Zou, Yi, J. Bailey, X. Ma, and Q Gu, "Improving adversarial robustness requires revisiting misclassified examples," in International Conference on Learning Representations (ICLR), 2020.
[45] N. Inkawhich, K. Liang, L. Carin, and Y. Chen, "Transferable perturbations of deep feature distributions," in International Conference on Learning Representations (ICLR), 2019.
[46] J. Jacobsen, J. Behrmann, R. Zemel, and M. Bethge. "Excessive invariance causes adversarial vulnera- bility," International Conference on Learning Representations (ICLR), 2018.
[47] V. Banks, K. Plant, and N. A. Stanton, "Driver error or designer error: Using the perceptual cycle model to explore the circumstances surrounding the fatal tesla crash on 7th may 2016," Safety Science, vol. 108, pp. 278-285, 2018.
[48] V. Antun, F. Renna, C. Poon, B. Adcock, and A. C. Hansen, "On instabilities deep learning in image reconstruction and the potential costs of ai," Proceedings of the National Academy of Sciences, 2020.
12


Reviews:
[{"review_id": "uMsCw-PSZZW", "reviewer": "Reviewer_jAXE", "summary": "This paper focuses on robust and non-robust features in adversarial examples. This paper proposes a method of distilling robust and non-robust features via information bottleneck. Through analysis, this paper has shown the high correlation between the distilled features and the adversarial prediction. Based on that, the authors design a new attack that intensifies the gradient of non-robust features.", "questions": "", "limitations": "", "rating": 6, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Pros:\n\n1. This paper proposes a new way to distill robust and non-robust features via information bottleneck.\n2. The detailed method are well illustrated and sounds feasible.\n3. Extensive analysis on the distilled features is provided and the figures or tables are well presented.\n4. The current experimental results have shown the strong performance of the proposed attack.\n\nCons:\n\n1. The motivation for using the information bottleneck seems to be unclear. Why we need the information bottleneck? Why information bottleneck can help to distill robust and non-robust features precisely? It is better to clearly discuss more in the former part of this paper.\n2. It is better to discuss more the difference between this paper and [1] since the latter also provides a way to deal with robust and non-robust features. What is the relationship between the definitions in the two papers of robust/non-robust features? What is the unique significance of using information bottleneck to distill robust/non-robust features? Since the latter paper has already demonstrated the high correlation between the prediction and robust/non-robust features and proposed an easy but effective way to figure out these features.\n\nMinor Concerns/Questions:\n\n1. Can the authors verify the effectiveness of the proposed attack method on a larger dataset? Can the authors discuss the impact of dataset size on the performance of the attack method?\n\n[1]. Ilyas, Andrew, et al. \"Adversarial Examples Are Not Bugs, They Are Features.\" Advances in Neural Information Processing Systems 32 (2019): 125-136.", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "It is encouraged to discuss more how to utilize the distilled robust features.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "6"}, "extra_text": {}}, {"review_id": "XodPYkPvvd", "reviewer": "Reviewer_Yvhe", "summary": "This work utilizes an Information Bottleneck (IB) to distill robust and non-robust features in neural networks (NNs). Experiments are performed to show that the distilled features are correlated with adversarial prediction and are human-perceptible. Finally, an attack mechanism intensifying the gradient of non-robust features is proposed.", "questions": "", "limitations": "", "rating": 8, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Utilizing IB to capture relevant information is not a novel technique, but distilling robust and non-robust features in the intermediate layers of networks with it sounds interesting. The authors define the robustness and non-robustness of units by their correlation to the model prediction and separate informative features into robust and non-robust units by their tolerance of feature variation, which is a novel perspective to study the robustness of NNs. Experiments along with visualizations can well support their claims. There is also question: What is the connection between the robustness and informativeness? Does Figure 2 implies that non-robust features are less informative than robust features?", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Please refer to the main review.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "5"}, "extra_text": {}}, {"review_id": "UDREBiMJmUf", "reviewer": "Reviewer_LQdF", "summary": "This paper categorizes the intermediate features of a DNN into robust ones and non-robust ones. Through visualization, the non-robust ones display different semantic information from the true label, and thus potentially leads to the existence of AE.  The paper proposes a new AE generation algorithm targeting the non-robust feature, and beats the common attack baselines.", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "I like the overall idea of the paper. It delves deep into the potential cause of AE and shows an interesting perspective.\n\nHowever, this paper suffers from the poor writing. There's no apparent grammar error, but the paper is almost un-readable because of wordy expressions, numerable inline definitions and lacking of concise algorithmic description. The algorithmic steps are buried in definition of terms and lengthy details. Here are some of my suggestions.\n\n-- Present the definition of IB, the definition of robust and non-robust feature and the algorithm of distinguishing them in separate definition boxes or separate paragraphs. Never use inline definition for key terms.\n\n-- Instead of immediately adding comments to the parameters in an equation, wait for the main content be defined and put analysis together. There's an analysis section, so the analysis should all go there... For each aspect of the analysis, add a paragraph header if subsection header takes too much space.\n\n-- Make sure all terms in the key equations are defined. For example in Equation 3), the term f_{l+} is not defined. \n\n-- Use concise expression. For example, starting from Line 97 \"Here, ...\" to the end of that paragraph, the entire part can be replaced by \"Since finding the exact value of the second term is computationally prohibitive, we use the heuristics in [18, 19].\" Save the space for **your** contribution. \n\n-- Make the caption count. For example for the caption of Figure 3, explicitly say the non-robust feature displays semantics similar to the misclassified label of AE.\n\nOverall, I like the contribution of this paper. However, in order for the community to better understand the paper and to more conveniently refer to the formulation in this work, I want to see a careful revision in writing. Therefore, I'm giving a score of 5 for now. \n\n============ Post Rebuttal Response =======================\n\nThanks for the response. It's unfortunate that Neurips doesn't allow uploading an updated draft. I will raise my score to 6 and trust the authors to make another pass over the writing thoroughly.", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "They are adequately addressed.", "review_text": "", "extra_scores": {"ethical_concerns": "No concern.", "needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "3"}, "extra_text": {}}, {"review_id": "MjUnjebF9nd", "reviewer": "Reviewer_CPc8", "summary": "The paper proposes to explicitly separate intermediate representation into robust and non-robust categories by information bottleneck, and demonstrate that the non-robust feature is related to adversarial prediction. An attack mechanism based on enforcing non-robust features gradient is proposed. The experimental results show the effectiveness of the proposed approach.\n", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Information bottleneck(IB) has been widely used for feature disentanglement. It’s an interesting idea to connect IB to separate robust and non-robust features under adversarial settings.  Overall, the paper is well-organized. Hopefully, the authors can help address the following questions. \n\n\n**1.** In line 119, how did you calculate the inherent feature variation $\\sigma_z$? Is it obtained by computing the variance of the same unit along the batch size dimension within a batch? I think it needs to be clarified since its maximal value directly determines the robust and non-robust index.\n\n\n**2.** Amplifying the brittleness of non-robust features is proposed as an attack. It is a typical white-box and untargeted attack, of which the applicable scenario is limited these days. Usually, we evaluate the effectiveness of an attack by attacking a baseline defense approach. There are many defense works against adversarial attacks[1]. To make the paper stronger, I think the authors should try at least one defense strategy to verify the proposed attack.  \n\n\n[1] [Towards Deep Learning Models Resistant to Adversarial Attacks.](https://arxiv.org/abs/1706.06083)  \n\n\n\n**3.**  The channel index acts like a mask to filter out non-robust features, which is highly dependent on the dataset used. On the other hand, most of the experiments are done on CIFAR-10 and SVHN, which both are small datasets and nicely distributed. I wonder about the transferability to larger datasets of the proposed method\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "The authors have adequately addressed the limitations and potential negative societal impact of their work.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "3"}, "extra_text": {}}, {"review_id": "95VC3yOgmdg", "reviewer": "Reviewer_xNdo", "summary": "This study aims at distilling feature representations (mainly the last convolution layer) into robust and non-robust features using information bottleneck. For every layer, the feature maps are partitioned into robust and non-robust for each individual example. After finding this, the paper first illustrates that selectively propagating robust features of adversarial examples built for a robust network will not degrade the accuracy but propagating non-robust features does which to some extent illustrates that the information bottleneck method has correctly identified the robust and non robust features. They also illustrate that the adversarial class label has high correlation to the Non-robust features.\n", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The paper has some interesting insights but they all correspond to what is expected and it doesn’t provide much new understanding of robustness. Also, the paper's clarity can be significantly improved, for example, there are parts which are referenced before and then introduced after (for example, Z^a_{nr} is referenced in section 3.2 and figure 3 but is defined for the first time in section 3.3). After reading the paper with care, I am still not sure if I understood the experiments completely. The paper extends the idea of input data having robust and non-robust features to deeper “learned” representations of a Neural Net. While in the original adversarial examples are features not bugs paper, the robust and non-robust features co-existed in each channel, in this paper the authors classify each channel as robust or non-robust. I do wonder whether more straightforward methods exist for this partitioning purpose — possibly, simply using the ell_2-norm of the gradient of the loss w.r.t. the feature map —  which can result in similar qualitative results? \n\nOne interesting factor seems to be from the fact that the non-robust deep features are also perceptually aligned. However, I am not sure if this is true in general, i.e., when we do not use regularizations used for visualizations and we partition deep features for naturally trained models. Do the authors have an intuition for that case? I do wonder if robust and non robust features of a naturally trained net are also perceptually aligned? The finding that the non-robust feature's visualizations for a natural example on an adversarially trained model visually look like the adversarial class is not shocking and is expected due to the perceptual alignment of robust networks where if you make an adversarial example with a large epsilon for them, the adversarial example looks like an image from the adversarial class. \n\nI do have the following other questions which could help me understand if I understood the paper good or not:\n\n* Is there a reason for using a robust net in Table1? Does the network have to be adversarially trained to have clearly separable R and NR features?\n* The NRF attack does not seem to be ell-infitiny bounded, is this correct?  If yes, comparisons of that with other attacks in table 3 does not seem to be fair.", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "NA", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "5"}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper utilizes the information bottleneck mechanism to disentangle robust and non-robust features in the representation space of a deep neural network. The paper shows a high correlation between the (wrong) prediction on an adversarial example and the identified non-robust features for the example. Finally, the paper proposes a new adversarial attack that aims to enlarge the gradient for the non-robust features to degrade the model performance.

Even though robust and non-robust features have been discussed in the context of adversarial examples before, this paper proposed an interesting and novel approach to disentangle these features in the representation space. The experiments in the paper do demonstrate a clear connection between the non-robust features and the performance of the model on adversarial examples. Furthermore, the identified adversarial attack is shown to be very effective in rendering the model useless. Thus, this paper will be a valuable addition to NeurIPS 2021.

There is some room for improvement in the writing of the paper. Please address the following issues in the revised version. Instead of formally introducing many notations, key definitions, and algorithmic details, they appear as part of various paragraphs. There is no discussion of the computational complexity issues, especially in the context of the proposed attack. Furthermore, how feasible is it to perform the proposed attack (NRF) as it requires first disentangling the robust vs. non-robust features and then accordingly perturbing those? Does NRF provide any guidance to train more robust models? Interestingly, it appears from Table 4, that model degradation is quite significant even when robust features are perturbed. In fact, perturbing the robust features is more effective than all baselines considered in Table 3 (obviously, except NRF). This point is glossed over in lines 284-288. Addressing these aforementioned issues will certainly improve the overall quality and impact of this paper. 
Decision: Accept (Poster)