Clustering Effect of Adversarial Robust Models
Authors: Yang Bai
Xin Yan
Yong Jiang
Shu-Tao Xia
Yisen Wang
Venue: NeurIPS 2021 Spotlight (2021)
Keywords: Adversarial robust models
hierarchical clustering effect
domain adaption tasks

Abstract: Adversarial robustness has received increasing attention along with the study of adversarial examples. So far, existing works show that robust models not only obtain robustness against various adversarial attacks but also boost the performance in some downstream tasks. However, the underlying mechanism of adversarial robustness is still not clear. In this paper, we interpret adversarial robustness from the perspective of linear components, and find that there exist some statistical properties for comprehensively robust models. Specifically, robust models show obvious hierarchical clustering effect on their linearized sub-networks, when removing or replacing all non-linear components (e.g., batch normalization, maximum pooling, or activation layers). Based on these observations, we propose a novel understanding of adversarial robustness and apply it on more tasks including domain adaption and robustness boosting. Experimental evaluations demonstrate the rationality and superiority of our proposed clustering strategy. Our code is available at https://github.com/bymavis/Adv_Weight_NeurIPS2021.

Clustering Effect of (Linearized) Adversarial Robust Models
Yang Bai Xin Yan Yong Jiang!25 Shu-Tao Xia Yisen Wang
'Tsinghua Berkeley Shenzhen Institute, Tsinghua University
Tsinghua Shenzhen International Graduate School, Tsinghua University
3 Key Lab. of Machine Perception, School of Artificial Intelligence, Peking University *Institute for Artificial Intelligence, Peking University SPCL Research Center of Networks and Communications, Peng Cheng Laboratory, China
Abstract
Adversarial robustness has received increasing attention along with the study of adversarial examples. So far, existing works show that robust models not only obtain robustness against various adversarial attacks but also boost the performance in some downstream tasks. However, the underlying mechanism of adversarial robustness is still not clear. In this paper, interpret adversarial robustness from the perspective of linear components, and find that there exist some statis- tical properties for comprehensively robust models. Specifically, robust models show obvious hierarchical clustering effect on their linearized sub-networks, when removing replacing all non-linear components (e.g., batch normalization, max- imum pooling, or activation layers). Based on these observations, we propose a novel understanding of adversarial robustness and apply on more tasks including domain adaption and robustness boosting. Experimental evaluations demonstrate the rationality and superiority of our proposed clustering strategy. Our code is available at httpss//gitubb.co/bymaawsss Laaeaaiis//dyWeegihhht NeurlPS20211
1 Introduction
Nowadays, deep neural networks (DNNs) have shown a strong learning capacity through a huge number of parameters and diverse structures [6 [. Mcanwhile, adversary has raised increasing security concerns on DNNs due to the observation of adversarial examples (i.e., the examples mislead classifier when crafted with human imperceptible but carefully designed perturbations) [K3 Adversarial robustness and defense techniques have thus become crucial for deep learning, yet many adversarial defense techniques found with deficiencies such as obfuscated gradient [2]. Up to now, the widely accepted techniques to improve adversarial robustness are adversarial training variants 35 36 by training DNN after data augmentation with the worst-case adversarial examples, Recent study has found that not only do robust models show moderate robustness under vast attacks. but also they can surprisingly boost the downstream tasks (e.g., domain adaption task with subpopulation shift) 29
A
However, the underlying mechanism of adversarial robustness is still not clear. Existing studies have explored adversary on some specific components of DNNs, such as batch normalization [12), skip connection [2] or activation layers 14 4 which yield some enlightening understanding. On the other hand, despite variation in DNNs' structure, the components can be roughly divided into linear and non-linear ones. The non-linear components tend be instance-wise. For example, adversarial
'Equal contribution. "Corespondence to: Yisen Wang
and Shu-Tao Xia (xiast@s.ssinghuu.cdu..o)
35th Conference on Neural Information Processing Systems (NcurIPS 2021).

---

CIFAR-10
Non-animal
Animal
(a) Class Hierarchy (b) VGGI6-STD
(c) VGG16-AT
(d) VGGI6-AT+C
Figure I: Correlation matrices of VGG-16 on CIFAR-10: b) 'STD' and c) 'AT' indicate standard and adversarial training, d) 'A++C' indicates applying hierarchical clustering in adversarial training. Robust models tend show clustering effect aligned with a) the class hierarchy of CIFAR-10, which is enhanced in d) our clustering adversarially trained model.
and benign examples tend to present different patterns activation components, once the optimization done and model evaluation mode is activated, they are shared and fixed independently from inputs. Nowadays, the intrinsic relationship of adversary with linearity has been studied. Some ascribes the adversary to linear accumulation of perturbations from inputs to final outputs, dubbed 'accidental steganography" [K] Some show that linear backward propagation enhances adversarial transferability [4 [4] Nevertheless, such mentioned linearity analyses are all on the original non-linear models, which hinders deeper understanding on linearity directly. Further
4 In contrast, for those linear
essential exploration is encouraged.
In this paper, we thus study adversarial robustmess the statistical regularity of linear components. We provide a novel insight on adversarial robustness by showing the clustering effect well aligned with class hierarchy. As shown in Fig. [d), given data set, superclasses defined by grouping semantically similar subclasses together to form a hierarchy, egg., in CIFAR-10, 'Cat' and 'Dog' represent different subclasses/fine labels, despite both being the same superclassesccoarse labels 'Animal'. Based on this, we conduct backward propagation on linearized sub-networks of robust and non-robust DNNs (through adversarial/standard training respectively) to extract the corresponding implicit linear expression. In this fashion, we could obtain Dimpt X Doutput linear matrix W. which extracted from inputs to predictive outputs. D Dmpptt represents for the dimension of one input data, specifically defined as width x height x channel. Doutput represents for the dimension of one output vector, which is also the number of classes. We further study the correlation of vectors in W by matrix C, whose shape is Doutput x Doutput- The details of W and C are introduced in Sec. 3 Then we find that robust models tend show hierarchical clustering effect in correlation matrix C. which is surprisingly consistent with class hierarchy. For example, in Fig. on CIFAR- 10, the fine labels (0,1,8,9) belong to 'Non-animal', others belong to 'Animal.. The matrix C' of robust model shows block clustering: values are close to '+1' in the two superclasses 1,.9, and (2,3,4,5,6,77 7), and values are close to '-1' across these two superclasses. Such clustering effect enhanced by our clustering regularization penalty in Fig. [(d] This phenomenon could be semantically explained as subclasses from the same superclass share more feature similarities. Such hierarchical clustering effect can give novel and insightful explanation on the superiority of robust models in some observed downstream tasks, as they can extract more semantic and representative features. To further confirm our findings, we enhance the hierarchical clustering effect in both adversarial robustness and downstream tasks, e.g., domain adaption proposed in 29 22). The improvements of experimental results demonstrate that the superiority of robust models is closely related to their hierarchical classification capability. Our contributions are summarized as follows:
To the best of our knowledge, we are the first to systematically analyze the statistical regularity of adversarially robust models (through adversarial training) compared to non- robust models (through standard training) on their linearized sub-networks.
We present an intriguing phenomenon of hierarchical clustering effect in robust models, and provide a novel yet insightful understanding of adversarial robustness. The clustering effect aligned with class hierarchy demonstrates more semantic and representative feature extraction capacity of robust models, which benefits a lot in various tasks.
Based on the observations, we propose a plugged-in hierarchical clustering training strategy to generally enhance adversarial robustness and investigate some intriguing adversarial attack findings. Besides adversarial-relatee study, we further explore some downstream tasks with

---

the understanding of hierarchical clustering, e.g., domain adaption with subpopulation shift. Experimental results show that the clustering effect and hierarchical classification learned by robust model benefits the task as well.
2 Related Work
2.1 Adversarial Attack and Defense
Attack Given a clean example æ, its true label y, a DNN model F with parameters 0 and its loss function L. the goal of adversarial attack is to find adversarial example z' that is close clean example on pixel level but can fool F to give an incorrect prediction. Fast Gradient Sign Method (FGSM) | adds perturbations clean example z by one step with step size 0), 0),
(1) 24 adds perturbations on clean example by steps with smaller step size a. After every step (k-th) attack, adversarial example is projected into the e-ball (x)) around clean example (function II):
x²
x + essinninnv
Projected Gradient Descent (PGD)
[I -~ + asign (
.0).y)). (0))))))
There are other attack techniques, such as white-box CW attack 6 black-box attacks 21 3 and
adaptive AutoAttack 9
Defense Many adversarial defense techniques have been proposed since then, such as input pre- processing [19], defensive distillation 24 model compression [TC), and adversarial training [23).
Among them, adversarial training variants are assumed to be most effective in comprehensive attack evaluations. They solve min-max problem as:
min max (FF(x'.).y)) (x)
(3)
The inner maximum problem often generates adversarial examples by FGSM or PGD, while the outer minimum problem optimizes the worst-case loss. Some defense technique improves robustness through wiss feature clustering clustering.
[| 30
which is different from our instance-wise feature
2.2 Lincarity Exploration in Adversary
Beside attack or defense techniques, the cause and understanding of adversary studied. The lin- earity is one essential perspective with no doubt. Previous studies attribute the existing of adversarial examples to the linearity of DNNs [K] and observe the adversarial transferability enhancement on more linear models 34 14] Some works study linearity from another perspective by focusing on the properties of weight layers, e.g., their norms, variances and orthogonality. Lp norm regularization works by pulling examples far away from the decision boundary, which comes as the side-effect of adversarial robustness 377 [8] Spectral norm 7 is useful in adversarial defense, which is
defined as a, = || and is used to constrain the Lipschiz constant a DNN F on x (assuming æ||2
Lipschiz continuous) [ 8) F(x)]l2 S a||z||2 < o; |2 The weight scale shifting issue is also discussed in adversary [2]. that is, scale weights can be shifted between layers without changing the input-output function specified by the network, which could affect the capacity to regularize models. Then one weight scale shift invariant regularization is proposed and improves adversarial robustness. Moreover, the orthogonality could help improve generalization and adversarial robustness [5] by inducing uncorrelated features. However, these experiments are all conducted on the original non-linear models instead of the linear components directly, which hinders deeper understanding on linearity.
Observations on Linear Components
The robust models perform completely differently from non-robust ones especially under compre- hensive adversarial attacks. However, the statistical regularity on linear components of these robust models is yet under little exploration. Different from related works, we explore linear components directly. To be specific, we extract weight matrix expression of linearized sub-networks, estimating the linear propagation from inputs outputs.
3

---

3.1 Extracting a Linear Weight Matrix
Given DNN FF composed of L linear layers and non-linear components, the output y F(x) could be expressed as
y = 9 x 91-1(W1-1 ... (1(1 x a + bb)).... bb-1) by):
where 9i is series of non-linear components (e.g., activation function ReLUs), w; and b; the expressions of linear ones. Then a corresponding linear sub-network of this original one could be extracted with the same weights and architectures, yet it removes activation function (e.g., ReLU layers) and batch normalization layers, and further replaces maximum pooling layers with average pooling layers. This output of linear network ylinear Flinear (a)) is expressed
ylieear wL X (WL-1 x (.:
bj)... + b + PL-
(5)
Thus in Finear, each input a multiplied by an instance-agnostic total weight
W wt x WL-1 ...WI Σ bi... x Wj...
(6)
However, such total weight expression W in DNNs is difficult compute directly, because the multiplication of convolution layers is infeasible. Instead, we propose to compute W by applying backward propagation on the linear sub-network. To be specific, given a random input and a pre-trained DNN F. W is computed as the gradient propagated on linear sub-network Fiinear following Algorithm ! For non-linear model, as the activation part is example-wise, is difficult to analyse the original network with non-lineat components. In contrast, the linear component is generally applied on all examples and example-agnos
So extract linear expression from the original non-linear model to approximate and analyse the model performance. Intuitively, although the linearized network has limited capacity, we think that the class-wise directions (the extracted linearized weight vectors) should still represent some class-wise directions, which holds a connection with the feature space of the non-linear models.
Algorithm 1 Extracting the Linear Weight Matrix W
Input: A random input æ, pre-trained DNN model F with D Doutput classes
Output: Weight matrix W
Get the corresponding linear sub-network Flinear from F
Conduct forward propagation as: Viinear Flinear (2)
for i range(Dnutpin) do
Vlincar backward()
W|: = x. æ.grad
end for return W
3.2 Observation on Weight Clustering Effect
As indicated, though instance-wise performance varies on one specific DNN with non-linear com- ponents, it shares the same linear sub-network (i.e. parameters and linear structures) and thus the same implicit weight matrix W. As the implicit matrix W could estimate linear output scores Vlinear of any input o when forward propagating on the linear sub-network, hints some class-wise linear amplifications. After extracting W with shape Dmputt x Doutput, we further explore the correlation across classes by normalizing weight vectors in W (to avoid the class-wise scale variance) and computing correlation matrix C' following
WT
W
C1,5
1l2
|IW
In such fashion, the matrix C. whose shape is D Doutput x D Doutput, has all elements in [-1, 1]. C(i.)) represents for cosine value of two weight vectors corresponding to class and j on linear sub-networks, which also represents for the correlation of class-wise weight vectors. If the element C(i.j) is close 1, it means that i-th and j-th class-wise weight vectors are strongly positive correlated the linear weight space, which also demonstrates that class i and j are more likely to be positively related with

---

PISTIEITE FERESSAAIIOO
(a) STL-AT
(b) CIFARIO-AT
(c) CIFAR20-AT
(d) ImageNer10-AT
(e) STL-STD
(f) CIFARIO-STD (g) CIFAR20-STD (h) ImageNetIO-STD.
Figure 2: Correlation weight matrix C' of ResNet-18 on different data sets. 'AT' indicates adversarial training. 'STD' indicates standard training. Robust models tend show weight clustering effect aligned with class hierarchy.
each other. In contrast, for the element C(i.j) close to -1, it means that i-th and j-th class-wise weight vectors are strongly negative correlated in the linear weight space, which also demonstrates that class and j are more likely to be negatively related with each other.
Datasets and Setups Beside Fig. [ we show more clustering effect across data sets using STL-10 [ and two reconstructed data sets CIFAR-20 and ImageNet-10, composed of 20/10 subclasses from 4/2 superclasses in CIFAR- -EFAR--10 and TinylmageNet (27l The details of constructed data sets are in Appendix A For robust models, we adversarially train ResNet-18 18 [15] using PGD-10 (c = 8/255 and step size 2/ /255) with random start. The non-robust models are standard trained. Both models trained for 200 epochs using SGD with momentum 0.9, weight decay 2e-4, and initial learning rate 0.1 which is divided by 10 the 75-th and 90-th epoch.
Result Analysis We plot the correlation matrix C' in Fig. 2 Similar to Fig. D on CIFAR-10, 10, the similar block clustering effect is observed robust models and aligns well with class hierarchy. To be specific, in Fig. 2(a) and 2(e) on STL-10, fine labels (0,2,8.9) belong to 'Non-animal" others belong to 'Animal', and the matrix of robust model shows block clustering: values close '+1' in the two superclasses (0,2,8.9) 9) and (1,3,4,5,6,7), values close to '-1' across these two superclasses. Such phenomena hint at the fundamental connection of adversarial robustness with weight clustering effect. That is, subclasses in the same superclass are positively related to each other, which is in contrast with those in different superclasses. This hierarchical classification property is more semantic with data sets, demonstrating that robust model has a better representative feature extraction capacity and thus outperforms standard model various tasks.
Discussion order to analyse the connection of weight clustering effect on linearized models with their original models, specifically take the linear model model is itself. We additionally train two linear models with initial learning rate 0.01 and 100 epochs by standard training PGD-10 adversarial training. The detailed architecture of the model shown Appendix [F] The final accuracies are listed in Table D Though the model capacity restricts their accuracies, the robust model still shows clustering effect. Fig. 3 indicates that the clustering effect of linear weight matrix generally holds even on a simplified linear model.
example whose linearized
Table 1: Accuracy (%) on CIFAR-10 two linear models. 'ST' indicate adversarial//tanddar training. The model capacity restricts their accuracies.
Linear Model
(CIFAR-10) 10)
AT
STD
CIFARIO-AT
(b) CIFARIO-STI
Clean ACC
29.21% 20.80%
39.53%
Figure 3: Correlation weight matrix C' of linear models on CIFAR-10. 'AT' model shows clustering effect.
PGD-20 ACC
4.36%
5

---

Explorations with Clustering Effect
simple and direct idea is to apply the linear weight clustering effect as a penalty. However, the time cost and computational cost are too expensive. Based on our understanding that robust models learn better class hierarchy of one data set and thus better semantic features, we further add the penalty in feature space by evaluating class-wise feature distance matrix D of original non- non-linear DNNs on CIFAR-I0. Given data pairs (ar,, Vi), their features Zi, we compute the class-wise feature centers Zj for class j following
Zj mean 2; ( = j)
F(a. b) represents the distance of class-wise feature centers Zu and Zz which is normalized with the largest class-wise distances for better visualization.
Zoll2
F(a
Zn
2el22
c,cpa
Result Analysis We adopt the same basic setting as Sec. 3.2 and find a similar clustering effect on CIFAR-10 in Fig- The models are the same with those in Fig. Robust models consistently show the better clustering effect compared with non-robust models. That is, the centers of subclasses the same superclass lic closer to each other in feature space. The feature clustering effect also aligns well with class hierarchy. Such effect further confirms our understanding that robust models could extract more semantic and hierarchical features even in their original non-linear feature space.
Further Clustering Enhancement Given any unknown data set, we first extract its class hierarchy from pre-trained robust model, e.g., trained by adversarial training. To be specific, we compute matrices W and C following Sec. 3.2 We then formulate C' to an approximate matrix Cop with all items set to '-1" or '+1', i.e., in Cop. subclasses in the same superclass block have values '+1', else have values '-1', With the extracted class hierarchy, a regularization loss is designed to minimize outputs within the same superclass. Take the data set CIFAR- 10 for example, subclasses are divided into (,,1, and (2,3,4.5,6,7) in Cop. The regularization loss defined in Eq. I0. The hierarchical instance wise clustering training strategy is proposed following Algorithm
Fu = F,+(( [ 0, 1,8,9
Fi = E,F(,) 1(j 2,3,4,5,6,7
(10)
(x))(a) Σ [F(x); Foll2
Σ j=266,5,66,7
[F(x), Fill2.
Algorithm 2 Enhancing the Hierarchical Clustering Effect
Input:
A random input æ, pre-trrined robust DNN model FF (e.g., CIFAR-10)
Output: Robustness Enhanced Model FF
Compute the weight correlation matrix C following Algorithm | and Eq.
Compute an approximate matrix Cop with all items '+1/-1' based on C and extract class hierarchy
Compute regularization loss Creg (z) with feature clusters following Eq. 10
with Lreg Retrain F from scratch using the original robust loss L(x, y) (e.g., adversarial training loss) added
return F
4.1 Domain Adaption Case
Recent study shows that robust models tend to perform better in downstream tasks, especially domain adaption with subpopulation shift 20 However, further analyses are insufficient. We study from two perspectives, 1) verifying how robust models outperform in domain adaption with different settings and 2) exploring a further improvement by enhancing the clustering effect. Specifically, we randomly divide a hierarchical data set into source data and target data according to their subclasses. That is, source and target data share the same superclass yet different subclasses, dubbed subpopulation shift, In the training phase, we use fine labels of source data and map fine labels to coarse ones. In such a fashion, the model returns fine and coarse labels. Only coarse label evaluated in the test phase as the target test data has different fine labels from source data. Both coarse and fine accuracies are

---

(a) STL-AT
(b) CIFARIO-AT
(c) CIFAR20-AT
(d) ImageNetlO-AT
(e) STL-STD
(f) CIFARIO-STD
(g) CIFAR20-STD (h) ImageNet1O-STDD
Figure 4: Feature distance matrix F of ResNet-18 on different data sets. 'AT" indicates ad advessaiaal training. 'STD' indicates standard training. Robust models tend show feature clustering effect aligned with class hierarchy, which is similar to C.
evaluated source domain. We enhance the feature clustering effect by penalty loss defined in Eq. [0 To better utilize hierarchical features extracted by pre-trained models, we also finetune parameters on target data to achieve further improvement. Different models, different constructed data sets and different adversarial training settings (i.e. adversarial robustness levels) are evaluated and analyzed.
4.1.1 Different Models
Datasets and Setups The data set CIFAR-100 is composed of 20 superclasses, and each superclass composed of 5 subclasses. Based on that, we construct our own data sets by randomly sampling from CIFAR- 100, e.g., our default CIFAR-22 20 is composed of 20 subclasses from 4 superclasses. The details of all constructed data sets are shown Appendix A We adversarially train ResNet- 18, DenseNet-122 121 [17] and AlexNet 20 with PGD. If not specifically mentioned, PGD-5 is adopted (5 steps, step size 1/255, e=2/255). The robust/non-robust ResNet- 18 and DenseNet- 121 are trained with SGD 25 and initial learning rate 0.1. The robust/non-robus AlexNet
trained with SGD) and a initial learning rate 0.01. The learning rates decay with a factor of 0.1 at the 75 and 90 epoch. Other hyper-parameters
the same with Sec. 3.22 Moreover, finetune the pre-trained models on
target data for 20 epochs with a learning rate 0.01.
Results Analysis We evaluate the models on our default CIFAR-20. As shown in Table when comparing the coarse accuracies on target and source data, robust models tend to decline less than non-robust ones. Take pre-trained ResNet-18 18 for example, the coarse accuracy robust model directly declines by 5.44% (from 87.94% to 82.50%) but declines by 10. 10.18% (from 90.93% to 80.75%) on non-robust model. When considering finetuned models, the superiority of robust models outstands. For example, the coarse accuracy on finetuned robust ResNet-18 increases by 6.46% (from 87.94% 44.40%) while increases by 3.32% (from 90.93% 94.25%) finetuned non-robust ResNet- 18. When applying the clustering penalty defined in Eq. 10 coarse accuracy is improved by 2.25% on robust R8sNet-18 (from 82.50% 84.75%) even without finetuning. Experimental results demonstrate that features extracted by robust models have non non-trivial impact on domain adaptation. Such superiority keeps consistent when finetuning with the feature centers of target training data, Thus our understanding is confirmed that a better hierarchical classification learned by robust models benefits a lot in downstream domain adaption task.
4.1.2 Different Data Sets
Datasets and Setups For CIFAR- 100, we randomly select 4 to 6 superclasses, select 4 subclasses from each superclass for training and apply the left one for test. We also select 5 superclasses, select 3 subclasses from each superclass for training and apply the left 2 subclasses for test. When selecting 4 superclasses and subclasses, we typically design two extreme data set construction methods, i.e. the most different and similar data set. The 'different" 4 superclasses are selected index 1, 2, 5, 14 (fish, flowers, household electrical devices, people). The 'similar" 4 superclasses are selected as index 8, 11, 12, 16, which are all animals. The reconstructed data set ImageNet-20 is composed of
7

---

Table 2: Accuracy (%) CIFAR-20 across various models. 'Coarse' and 'Fine' indicate the ground- truth labels are coarse or fine respectively. 'FT' indicates the pre-trained models are finctuned on target data. 'R' indicates robust models (adversarial training). 'NR' indicates non-robust models (standard training). '+C" indicates robust models with an enhanced clustering effect defined in Eq.
10
Source Domain Fine
Target Domain
Coarse
Coarse
Coarse-FT
AlexNet
NR
85,50
65.37
71.75
86.70
R
84.75
64.50
72.75
87.50
R+C
90.68
65.56
75.25
89.25
R8sNet-18
NR
90.93
70.62
80.75
94.25
R
87.94
68.75
82.50
94.40
R+C
88.25
69.62
84.75
94.75
DenseNet-123 121
NR
92.31
73.56
77.25
95,50
R
90.18
72.87
81.00
95.75
R+C
91.37
73.87
85.25
96.50
00.5
15.5
Dage Minco
Thipe
Sours Accamoon
S0uce ACOUROOIM
Souic Acc 10777
(a) Different Data Sets
(b) Different CIFAR-4*4
Figure 5: Comparisons of target accuracy across different data sets. Different colors indicate different data set construction methods. Different shapes indicate different training methods, i.e. adversarial or standard training. Circle indicates non-robust model (NR), square and start indicate robust model without/with an enhanced clustering effect (R/R+C) respectively. Pentagon, diamond and triangle indicate finetuning NR/R/R+C models respectively. The legend in format A * B: A indicates the number of selected superclasses, B indicates the number of selected subclasses from each superclass for training. The points located in left and upper represent for better performances they have lower source accuracies yet higher target accuracies.
20 subclasses from 4 superclasses in TinylmageNet. The details conduct experiments ResNet-18 18 in this section.
of datasets are in Appendix A We
Results Analysis As shown Fig. 5 we find that experimental results on different data sets show different yet consistent performances: robust models show great superiority on those more hicrarchical data sets, which can be boosted with an enhanced clustering effect. In Fig. [5(a] our constructed data sets from CIFAR-100 and TinylmageNet show superiority with robustness and an enhanced clustering. Moreover, a better class hierarchy of data set enhances domain adaption tasks. For example, orange points (i.e. the 'different' data set) in both two subfigures in Fig. 5(b) tend to present lower source accuracies yet higher target accuracies, especially compared with lightseagreen points (i.e. the 'random' data set). For 'similar' data sets, because the class hierarchy is not obvious, target accuracy drops lot. However, the royalblue start point (i.e. 'R+C' version model trained on 'similar' data set) still achieves the highest target accuracies among three models trained 'similar' data set, We show the specific numerical results in Appendix
4.1.3 Different Robustness Settings
Datasets and Setups We evaluate 18 with different robustness settings (i.e. different PGDs in adversarial training) on random CIFAR-4x4 defined in Appendix The PGD details are given in Appendix B
Results Analysis As shown Fig. we could observe a trade-off between adversarial robustness and accuracies on target data. For example, in the polylines of both two subfigures, purple points show higher accuracies on target data than brown ones. However, brown points are of better robustness as their accuracies source data much lower. Royalblue points show the relatively bad performance
8

---

on target data yet the best performance on source data. The phenomena demonstrate trade-off between robustness and domain adaption performances. The better performance on target domain could only be achieved with both robustness and moderate accuracies on source data. Furthermore, robust model with an enhanced clustering effect has significantly results than the original robust model. We show the specific numerical results in Appendix In addition, the time costs results with different robustness settings are shown in Appendix D
750 115 738 arwaayy
753
733
Figure 6: Comparisons of target accuracy across different adversarial robustness levels (i.e. different adversarial training settings). Different colors indicate different PGD attack settings in adversarial training. Circle and square indicate robust model without/with an enhanced clustering effect (R/R+C), Pentagon and diamond indicate finetuning R/R+C model respectively. The points located in left and upper represent for better performances as they have lower source accuracies yet higher target accuracies.
4.2 Adversarial Case
We further conduct extensive experiments explore such intriguing clustering effect along with other adversarial phenomena. The experiments are designed from two perspectives, 1) adversarial robustness enhancement and 2) attack confusion matrix study. The results show that adversarial robustness could be improved by enhancing the hierarchical clustering effect, and attack success rates highly related to class hierarchy well.
4.2.1 Adversarial Robustness Enhancement
Datasets and Setups We adversarially train robust ResNet-18 CIFAR-10. The basic training setting is the same with Sec. 3.2 The PGD-20 attack applied with c = 8/255 and step size 0.003.
Results Analysis Although the main exploration of clustering effect is studied not only in robustness improvement, we conduct comprehensive robustness evaluations following the statc-of-the- attack strategy with adaptive AutoAttack [9] We find that the clustering enhancement actually boosts adversarial robustness in Table 3 for both best and last checkpoints. For black-box attacks, the surrogate model is standard trained VGG-16. A'tttack is conducted 1,000 random test data and with a maximum query 20,000. Note that, the improvement against Auto/Atack is 1.31% 1.3,% which shows that the clustering alignment with coarse labels learned by robust models could indeed boost adversarial robustness. An interesting phenomenon is that when enhancing clustering effect, both natural and robust accuracies are improved because the clustering effect improves feature representation. The natural accuracy by standard training improves as well (from 92.75% to 93.54 3445)) with clustering effect. Therefore, the trade-off between adversarial robustness and natural accuracy still exists. In addition, we count the training time of two models (AT and AT+C). The AT model costs 191.20s per epoch, while the AT+C model costs 193.66s. using one GPU 1080X with batch size 128 of ResNet-18 18 on CIFAR-10 10. The time cost slightly increases with the clustering regularization, which is negligible. More results are shown in Appendix E
Table 3: Robustness (accuracy (%) on various attacks) on CIFAR- 10, based on the best/last checkpoint of ResNet-18. 'AT' indicates adversarial training. '+C' indicates applying our hierarchical instance- wise clustering training strategy. The best results are boldfaced.
White-box
Black-box CW
Adaptive
Defense
Natural
FGSM
PGD-20
CW
PGD-20
NAttack
AutoAttack
Best Last
84,20
63.32
50.12
50.97
81.80
84.80
48.21
46.58
AT
84.27
60.46
46.50
48.97
79.13
79.87
45.05
41.90
Best
85.43 85.53
64.11
52.54
52.72
82.27
85.22
50.51
47.21
AT+C
Last
62.11
47.78
49.95
80.21
78.98
47.51
43.21

---

Smebbrice (b) UNTARGET-STD
UNTARGET-AT
TARGET-AT
(d) TARGET-STD
Figure Attack confusion matrix M under different settings. 'AT'STD'' indicate adversar-
ial/standard training. Robust models tend to show clustering effect aligned with class hierarchy.
4.2.2 Attack Confusion Matrix
To further analyze how class hierarchy matters in adversary, evaluate the attack confusion matrix M of adversarial examples in Fig. whose shape is Doutput x Doutput. M(, j) is defined as attack success rate (ASR) of adversarial examples being attacked from the ground-trun truth label i to the target label j.
Datasets and Setups We train robust and non-robust ResNet-18 on CIFAR-10. The basic setting is the same with Sec. 3.2 The PGD-20 attack is applied with 8/ 8/255 and step size 0.003. For target attack, target labels are set the other 9 classes except for the ground-truth one.
Results Analysis Experimental results show that the attack confusion matrix M performs in another blocking pattern especially on robust models, which is well-aligned with class hierarchy as well as weight correlation matrix and the feature distance matrix F. That is, the difficulty attacking robust models shows distinguishing performance for subclasses in or out of the same superclass. For example, against untarget PGD-20 attack in Fig. a). the ASRs on adversarially trained ResNet- 18 show that classes in (2,,,4,5,6,7) could decline to almost zero from one superclass (0.1,8,9) to the other (2,3,4,5,6,7). Different superclasses are harder to attack especially robust models, demonstrating that the hierarchical classification capacity is enhanced by adversarial robustness. Such phenomenon is also consistent with our hierarchical clustering observations on robust models.
casier to be attacked from one other. However, the ASRs
Conclusions
In this paper, we provide novel view of linear model exploration by extracting an implicit linear matrix expression. Then we surprisingly find an intriguing clustering effect adversarially robust models, which is well-aligned with class hicrarchy. Based on such observations, we first give an insightful explanation and understanding of robust models, which have a better capacity in extracting more representative features and semantic information. Extensive experiments show that enhancing the hicrarchical clustering effect not only improves model robustness, but also benefits other downstream domain adaption task. Overall, our observations and findings motivate a deeper understanding on adversarial robustness.
Acknowledgement
Yisen Wang partially supported by the National Natural Science Foundation China under Grant 62006153, and Project 2020BD006 supported by PKU-Baidu Fund. Shu-Tao Xia is supported in part by the National Key Research and Development Program of China under Grant 2018YFB 1800204, the National Natural Science Foundation of China under Grant 62171248, the R&D Program of Shenzhen under Grant JCYJ201 80508 52204044.
References
[1] Motasem Alfarra, Juan C Pérez, Adel Bibi, Ali Thabet, Pablo Arbeláez, and Bernard Ghanem. Clustr: Clustering training for robustness. arXiv e-prints, 2020.
[2] Anish Athalye, Nicholas Carlini, and David Wagner. Obfuscated gradients give a false sense of security: Circumventing defenses to adversarial examples. In ICML, 2019.
10

---

[3] Yang Bai, Yuyuan Zeng, Yong Jiang, Yisen Wang, Shu-Tao Xia, and Weiwei Guo. Improving query efficiency of black-box adversarial attack. In ECCV, 2020.
[4] Yang Bai, Yuyuan Zeng, Yong Jiang, Shu-Tao Xia, Xingjun Ma, and Yisen Wang. Improving adversarial robustmess via channel-wise activation suppressing. In ICLR, 2021.
[5] Nitin Bansal, Xiaohan Chen, and Zhangyang Wang. Can we gain more from orthogonality regularizations in training deep networks? 2018.
[6] Nicholas Carlini and David Wagner. Defensive distillation is not robust to adversarial examples. arXiv preprint 1607.0411
[7] Moustapha Cissé, Piotr Bojanowski, Edouard Grave, Yann Dauphin, and Nicolas Usunier. Parseval networks: Improving robustness adversarial examples. In ICML, 2017.
[8] Adam Coates, Andrew Ng. and Honglak Lee. An analysis of single-layer networks unsuper- vised feature learning. In Proceedings of the Fourteenth International Conference Artificial Intelligence and Statistics, 2011.
[9] Francesco Croce and Matthias Hein. Reliable evaluation of adversarial robustness with an ensemble of diverse parameter-free attacks. 2020.
[10] Nilaksh Das, Madhuri Shanbhogue, Shang- Tse Chen, Fred Hohman, Siwei Li, Li Chen, Michael E Kounavis, and Duen Horng Chau. Compression to the rescue: Defending from adversarial attacks across modalities. In KDD, 2018.
[11] Jacob Devlin, Ming-Wor Wiig-Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. In NAACL, 2019.
[12] Angus Galloway, Anna Golubeva, Thomas Tanay, Medhat Moussa, and Graham W Taylor. Batch normalization is a cause of adversarial vulnerability, 2019.
[13] Ian J Goodfellow, Jonathon Shlens, and Christian Szegedy. Explaining and harnessing adversar- ial examples. In ICLR, 2015.
[14] Yiwen Guo, Qizhang Li, and Hao Chen. Backpropagating linearly improves transferability of adversarial examples. 2020.
[15) Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. arXiv preprint arXiv: 1512.03385, 2015.
[16] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In CVPR, 2016.
[17] Gao Huang, Zhuang Liu, Laurens Van Der Maaten, and Kilian Q Weinberger. Densely connected convolutional networks. In CVPR, 2017.
[18) Daniel Jakubovitz and Raja Giryes. Improving dnn robustness to adversarial attacks using jacobian regularization. In ECCV, 2018.
[19] Xiaojun Jia, Xingxing Wei, Xiaochun Cao, and Hassan. Foroosh. Comdefend: An efficient image compression model to defend adversarial examples. In CVPR, 2019.
[20] Alex Krizhevsky, Ilya Sutskever, and Geoffrey E Hinton. Imagenet classification with deep convolutional neural networks. In NewrIPS, 2012.
[21] Yandong Li, Lijun Li, Liqiang Wang, Tong Zhang, and Boqing Gong. Nattack: Learning the distributions of adversarial examples for improved black-box attack on deep neural networks. 2019.
[22] Ziquan Liu, Yufei Cui, and Antoni B Chan. Improve generalization and robustness of neural networks via weight scale shifting invariant regularizations. arXiv preprint arXiv:2008. 02965, 2020.
11

---

[23] Aleksander Madry, Aleksandar Makelov, Ludwig Schmidt, Dimitris Tsipras, and Adrian Vladu. Towards deep learning models resistant to adversarial attacks. 2018.
[24] Nicolas Papernot, Patrick McDaniel, Xi Wu, Somesh Jha, and Ananthram Swami. Distillation as defense to adversarial perturbations against deep neural networks. In S&P, 2016.
[25] Ning Qian. On the momentum term in gradient descent learning algorithms. Neural Networks, 15,), 1999.
[26] Kevin Roth, Yannic Kilcher, and Thomas Hofmann. Adversarial training generalizes data- dependent spectral norm regularization. 2019.
[27] Olga Russakovsky, Jia Deng, Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma. Zhiheng Huang, Andrej Karpathy, Aditya Khosla, Michael S. Bernstein, Alexander C. Berg, and Fei-Fei Li. Imagenet large scale visual recognition challenge. arXiv preprint arXiv: 1409.0575, 2014.
[28] Hadi Salman, Andrew Ilyas, Logan Engstrom, Ashish Kapoor, and Aleksander Madry. Do adversarially robust imagenet models transfer better? arXiv preprint arXtv:2007.08489 2020.
[29] Shibani Santurkar, Dimitris Tsipras, and Aleksander Madry. Breeds: Benchmarks for subpopu-
lation shift. arXiv preprint arXiv:2008. 04859, 2020.
[30] Qi Tian, Kun Kuang, Kelu Jiang, Fei Wu, and Yisen Wang. Analysis and applications of class-wise robustness in adversarial training. In KDD, 2021.
[31] Yisen Wang, Xuejiao Deng, Songbai Pu, and Zhiheng Huang. Residual convolutional networks for automatic speech recognition. arXiv preprint 1702.0777 2017.
[32] Yisen Wang, Xingjun Ma, James Bailey, Jinfeng Yi, Bowen Zhou, and Quanquan Gu. On the convergence and robustness adversarial training. In ICML, 2019.
[33] Yisen Wang, Difan Zou, Jinfeng Yi, James Bailey, Xingjun Ma, and Quanquan Gu. Improving adversarial robustness requires revisiting misclassified examples. In ICLR, 2020.
[34] Dongxian Wu, Yisen Wang, Shu-Tao Xia, James Bailey, and Xingjun Ma. Skip connections matter: On the transferability of adversarial examples generated with resnets. In ICLR, 2020.
[35] Dongxian Wu, Shu-Tao Xia, and Yisen Wang. Adversarial weight perturbation helps robust generalization. In NeurlPS, 2020.
[36] Jia Xu, Yiming Li, Yong Jiang, and Shu-Tao Xia. regularization. In ICIP, 2020.
Adversarial defense via local flatness
[37] Ziang Yan, Yiwen Guo, and Changshui Zhang. Deep defense: Training dons with improved adversarial robustness. 2018
[38] Hongyang Zhang, Yaodong Yu, Jiantao Jiao, Eric P Xing, Laurent El Ghaoui, and Michael I Jordan. Theoretically principled trade-off between robustness and accuracy, 2019.
12


Reviews:
[{"review_id": "gvf_aRlWg4v", "reviewer": "Reviewer_92hs", "summary": "Adversarial robustness has received a lot of attention along with the study of adversarial data. So far, almost works have shown that robust classifier not only outperform under comprehensive adversarial attack evaluations but also boost the performance in some downstream tasks. However, the underlying mechanism of adversarial robustness is still not clear.\n\nIn this paper, they investigate adversarial robustness from the perspective of linear components. From this very novel perspective, they find that there are some statistical properties for comprehensively robust classifiers. Specifically, robust classifiers show an obvious hierarchical clustering effect on their linear sub-networks. Based on the above investigation, this paper applies this on more tasks, such as robustness boosting and domain adaption.", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Pros:\n\n1. There are rare works to deeply think about the underlying mechanism of adversarial robustness. In this paper, they consider understanding this from the perspective of linear components. From this very novel perspective, they find that there are some statistical properties for comprehensively robust classifiers. Specifically, robust classifiers show an obvious hierarchical clustering effect on their linear sub-networks. This contribution is novel and has a significant impact on the field of adversarial machine learning.\n\n2. They present an intriguing phenomenon of hierarchical clustering effect in robust models and provide a novel yet insightful understanding of adversarial robustness. The clustering effect aligned with class hierarchy demonstrates more semantic and representative feature extraction capacity of robust models, which benefits a lot in various tasks.\n\n3. The paper is technically sound and well written. The proposed solution is based on a very interesting investigation. Figures are very informative. \n\n4. Besides adversarial-related study, this paper further explores some downstream tasks with the understanding of hierarchical clustering. Experimental results show that the clustering effect and hierarchical classification learned by robust model benefits the task as well.\n\nCons:\n\n1. The major concerns include the setup of experiments. The domain adaptation is a well-developed area and problem. This paper said their contributions to the domain adaptation field but did not explain details in such experiments. In domain adaptation, we focus the performance on the target domain. It is very confusing what kind of domain adaptation tasks is considered in Table 2. Why is there accuracy on the source domain? This is not the one cared about in the field.\n\n2. I noticed many hyper parameters used in the experiments like dynamically changed learning rate. What are the motivations to design such ways to adjust hyper parameters? \n\n3. It is unclear why the normalization in Eq. (8) is necessary. More explanations should be given.\n\n4. In the defense part (Table 1), which method does this AT mean? It would be better to see the effectiveness of “+C” with more adversarial training methods. \n\n--------------------------------------------\nMy concerns are addressed well. Thus, I will increase my score by 1 point.", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "The authors have discussed limitations and potential social impacts well.", "review_text": "", "extra_scores": {"ethical_concerns": "No.", "needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "7hrs"}, "extra_text": {}}, {"review_id": "bSr6FVfGPg-", "reviewer": "Reviewer_cKPT", "summary": "This paper introduces the \"clustering effect\" of linear components of adversarial robust models. To be specific, the authors propose to remove the non-linear functions in networks and further integrate all linear components into a single weight matrix W in order to directly study the effect of linearity in adversarial models. By conducting experiments with W, the authors find that adversarially trained models tend to show clustering effects in both weight and feature dimensions. This effect further benefits some downstream tasks such as adversarial attack and transfer learning and can be enhanced with additional supervision. In all, the phenomenon showed in this paper is interesting and deserves more and deeper research.\n", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "  - Originality: It seems novel -- I don't see the proposed method before. \n  - Quality: Though the phenomenon shown in the paper is interesting, first of all, the setting is relatively simple, especially considering that the superclasses in most experiments are only animals and non-animals. Also, I do not see a convincing reason for ignoring fine-level accuracy in transfer learning from the authors. It would be good to explain why just compute the coarse-level accuracy. Second, the authors only point out this phenomenon and conduct further experiments on its applications without deep analysis on the reason behind it. I would expect more theoretical analysis on why linear components of the networks can show such clustering effects when trained with adversarial examples. Third, the experimental results are not always consistent and sometimes the improvement is also trivial which further enlarge my concern about the paper. For example, in Figure 5(a), we can see those non-robust models perform the worst as we would expect in certain situations but perform the best at other times. Also in Table 2, the performance after fine-tuning might be affected by the upper bound the backbone can achieve which can not fully support the idea that robust models can benefit more from fine-tuning.\n  - Clarity: I have two questions that need to be further answered. First of all, how the proposed regularization loss is added to the training pipeline should be stated more clearly. According to Algorithm 2, after computing the regularization loss before retraining the model, is the feature clusters and the loss fixed or it is computed by the running mean of the samples? Besides, why the performance of AlexNet with R+C in Table 2 is so much higher than NR and R? (91 v.s. 85). I am wondering if this is not an expected phenomenon so some models might not be trained well enough. The writing of the paper needs to be improved greatly. Some paragraphs are hard to follow such as Section 4.1.1 and 4.2.1. There also exist lots of typos that need to be fixed. In the end, I would like to suggest the authors re-draw Figures 5 and 6. It is a little bit hard to distinguish all the shapes and colors without annotations on the graph when the only guidance is the text in the main body.\n  - Significance: I think the phenomenon observed in this paper is quite interesting and deserves more research. It might be a good start point for this area if the paper can be updated.", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "The paper does not discuss the limitations. I think the most critical weakness is the too simple setting of the experiments and lack of deeper analysis. Please also see the main review for other weaknesses.\n", "review_text": "", "extra_scores": {"ethical_concerns": "N/A", "needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "5"}, "extra_text": {}}, {"review_id": "XeF8Rt_hDql", "reviewer": "Reviewer_7cFv", "summary": "The paper analyses adversarially robust models in terms of their clustering behavior with respect to the learned weights as well as with respect to the class-wise feature representations. For this purpose, the CNNs are linearized, i.e. non-linearities (ReLUs) are removed from the trained network such that it can be represented by a single weight matrix W, where each column corresponds to one class. The paper comes to the conclusion that hierarchical clustering of these representations is an indicator for adversarial robustness. Consequently, it formulates a regularization term which encourages such clustering behavior. The resulting models are evaluated in terms of adversarial robustness and in the context of domain adaptation. ", "questions": "", "limitations": "", "rating": 5, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The paper makes an interesting observation in terms of analyzing adversarially robust models. While adversarial robustness is classically hard to grasp besides when evaluating against adversarial attacks, the proposed hierarchical clustering can be directly measured and implemented into a loss in a straight forward way. Therefore, the paper offers in my understanding an interesting new perspective onto the issue. \nThat said, several aspects are not clear in the paper. \n\nFrom a theoretical perspective, it remains unclear why the analysis of the linearized network should extend to the original network with non-linearities. However, is this correlation holds, I wonder, from a practical perspective, why the feature space analysis is not executed on the linearized network as well. More importantly, in Figure 3, we see the evaluation of several trained models in terms of their linearized weight clustering. However, in Figure 4, only one model is shown in terms of feature clustering. The clustering effect appears to be much weaker here. What would the same evaluation look like for the other models in Fig.3? Also, in Fig 3. what are the accurcies on test/train/val set for the 8 models? Are these in a comparable range for each pair of models?\n\nThe experiments show that the additional proposed clustering loss can boost the robustness in terms of accuracy under attack for different attack methods, including the adaptive autoAttack method as well as a black-box attack. Similarly, the results on domain adaptation can be improved. Yet I object to the timing analysis in lines 213, 214. The model with the proposed loss only needs slightly more time to train than the original model with adversarial training. Yet, the entire model has to be fully trained before, in order to determine the clusters to be used in the regularization. Thus, the training time of the adversarially robust model is increased by more than a factor of 2. \nThe results in Tab 1 indicate that the accuracy under attack improves with the proposed procedure. Yet, the reported clean accuracy for these models is increased in a similar range (1.2 - 1.3 %) as well. So it is unclear whether the increased accuracy under attack might be caused by better training in general. The drop between clean accuracy and accuracy under attack remains about constant. Therefore, these results are not convincing to me.\nThe results in the domain adaptation task are more convincing. Yet, they are not well motivated in the context of the paper.\nIn neither of the evaluations, the proposed approach is compared to the state-of-the-art, but only relative improvements are reported. Would they also hold for the SotA model in terms of adversarial robustness such as:\nhttps://arxiv.org/abs/2010.00467\nhttps://arxiv.org/abs/2002.08619\nhttps://arxiv.org/abs/2011.11164\nhttps://arxiv.org/abs/2002.11242\netc... ?\n\n \nThe paper is overall well structured but the language is sometimes hard to understand. (e.g. line 48,49: D_input represents for the dimension...)\n \n\nMinor: \nIntro, line29: if you say \"on the other hand\", there should be \"on the one hand\" somewhere before\nplease define \\delta in eq. 4\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "The limitations and societal impact are not well discussed in the paper. With respect to the societal impact, I would assume that this is tolerable as the paper makes a conceptual contribution to the analysis of adversarial robustness, probably without a direct societal impact. Potential benefit would be, if the model works in practice , that CNNs could be applied in more safety relevant applications.\nWith respect to the limitations, please refer to the weaknesses I listed above. The model needs a significantly larger training time that the basic adversarially robust model. A comparison to the state of the art is missing and it is unclear whether the relative improvement would translate to a better model.\n", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "3"}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper presents an interesting study of adversarial models that seems novel and creative.
The phenomenon identified for the linear components is intriguing, though it seems it remains speculative to fully understand its implications on the original network. Also, the experimental results could be stronger.
Nevertheless, given the novelty of proposal in an area that has received significant attention, this seems like an interesting contribution. 
Decision: Accept (Spotlight)