Representation Quality Explain Adversarial Attacks
Authors: Danilo Vasconcellos Vargas
Shashank Kotyan
Moe Matsuki
Venue:  (2020)
Keywords: Representation Metrics
Adversarial Machine Learning
One-Pixel Attack
DeepFool
CapsNet

Abstract: Neural networks have been shown vulnerable to adversarial samples.  Slightly perturbed input images are able to change the classification of accurate models, showing that the representation learned is not as good as previously thought. To aid the development of better neural networks, it would be important to evaluate to what extent are current neural networks' representations capturing the existing features. Here we propose a way to evaluate the representation quality of neural networks using a novel type of zero-shot test, entitled Raw Zero-Shot. The main idea lies in the fact that some features are present on unknown classes and that unknown classes can be defined as a combination of previous learned features without representation bias (a bias towards representation that maps only current set of input-outputs and their boundary). To evaluate the soft-labels of unknown classes, two metrics are proposed. One is based on clustering validation techniques (Davies-Bouldin Index) and the other is based on soft-label distance of a given correct soft-label.
Experiments show that such metrics are in accordance with the robustness to adversarial attacks and might serve as a guidance to build better models as well as be used in loss functions to create new types of neural networks. Interestingly, the results suggests that dynamic routing networks such as CapsNet have better representation while current deeper DNNs are trading off representation quality for accuracy.

Under review as a conference paper at ICLR 2020
REPRESENTATION QUALITY EXPLAINS ADVERSARIAL ATTACKS
Anonymous authors Paper under double-blind review
ABSTRACT
Neural networks have been shown vulnerable to adversarial samples. Slightly perturbed input images are able to change the classification of accurate models, showing that the representation learned is not as good as previously thought. To aid the development of better neural networks, would be important to evaluate to what extent are current neural networks' representations capturing the existing features. Here we propose a way to evaluate the representation quality of neural networks using novel type of zero-shot test, entitled Raw Zero-Shot. The main idea lies in the fact that some features are present on unknown classes and that unknown classes be defined as combination of previous learned features. To evaluate the soft-labels of unknown classes, two metrics are proposed. One is based clustering validation techniques (Davies-Bouldin Index) and the other is based on soft-label distance of given correct soft-label. Experiments show that such metrics are in accordance with the robustness to adversarial attacks and might serve as a guidance to build better models as well as be used in loss functions to create new types of neural networks. Interestingly, the results suggests that dynamic routing networks such as CapsNet have better representation while current deeper DNNs are trading off representation quality for accuracy.
INTRODUCTION
Adversarial samples are slightly perturbed inputs that can make neural networks misclassify. They are carefully crafted by searching for variations in the input that, for example, could decrease the soft-labels of the correct class. Since they were discovered some years ago (Szegedy) 2014), the number of adversarial samples have grown in both number and types. Random noise was shown to be recognized with high confidence by neural networks (Nguyen et al. 2015), universal perturbations, that can be added to almost any image to generate an adversarial sample, were shown to exist Moosavi-Dezfoolll et al.) 2017), and the addition of crafted patches was shown to cause networks to misclassify (Brown et al. (017). Only one pixel is enough to make networks misclassify (Su et al. 2017). Such attacks can also be easily transferred to real-world scenarios (Kurakin et al. 2016).44 (Athalye & Sutskever] 2018), which confers a big issue well security risk for current deep neural networks' applications.
Albeit the existence of many defences, there not any known learning algorithm or procedure that can defend against adversarial attacks consistently. Many works have tried to defend by hiding or modifying the gradients to make neural networks harder to attack. However, recent paper shows that most of these defences fall into the class of obfuscated gradients which have their shortcomings (e.g., they can be easily bypassed by transferable attacks) (Athalye al. 2018). Additionally, the use of an augmented dataset with adversarial samples (named adversarial training) is perhaps one of the most successful approaches to construct robust neural networks (Goodfellow et al. (2014) ().uHu al. 2015), (Madry et al.) 2018). However, it is still vulnerable to attacks and has strong bias in the type of adversarial samples used in training (Tramèr et al. 2018).
This shows that deeper understanding of the issues is needed to enable more consistent defences to be created. Few works focused on understanding the reason behind such lack of robustness. In (Goodfellow et al., 2014), ) it is argued that Deep Neural Networks's (DNN) linearity one of the main reasons. Recent investigations reveal that attacks are changing where the algorithm is paying attention (Vargas & Su 2019), other experiments show that deep learning neural networks learn false

---

Under review as a conference paper at ICLR 2020
structures that are easier to learn rather than the ones expected (Thesing al. 2019) and an accuracy and robustness trade-off for models were shown to exist (Tsipras et al. 2019).
In this paper, we propose methodology of how to evaluate the representation of machine learning methods. Based on these metrics, we reveal a link between deep representations' quality and attack susceptibility. Specifically, we propose test called Raw Zero-Shot and two metrics evaluate DNN's represenaations.
1.1 RECENT ADVANCES IN ATTACKS AND DEFENSES
DNNs were shown vulnerable to many types of attacks. For example, the output high confidence results to noise images (Nguyen et al. 2015), universal perturbations in which single perturbation can be added to almost any input to create an adversarial sample are possible MMoosavi-Dezfood
et al. 2017), the addition of image patches Moreover, the vulnerability can be exploited even with a single pixel, i.e., changing a single pixel is often enough to make DNNs misclassify (Su et al. (201)). Most of these attacks can be transformed into real-world attacks by merely printing the adversarial samples (Kurakin et al.) 2016). Moreover, crafted glasses (Sharif et al. 2016) or even general 3d adversarial objects (Athalye & Sutskever 2018) can be used as attacks.
also make them misclassify (Brown al. 2017).
Although many defensive systems were proposed to tackle the current problems, there is still no consistent solution available. Defensive distillation in which a smaller neural network squeezes the content learned by the original DNN was proposed (Papernot al.] 2016). However, was shown not to be robust enough (Carlini & Wagner 2017). Adversarial training was also proposed as a defence, in which adversarial samples are used to augment the training dataset (Goodfellow eet al. 2014).(Huang et al.) 2015), (Madry et al., 2018). With adversarial training, DNNs increase slightly in robustness but not without bias towards the adversarial samples used and while still being vulnerable to attacks in general (Tramer et al.) 2018). There many recent variations of defenses in which the objective is to hide the gradients (obfuscated gradients) (Ma et al.) 2018). 8). (Guo et al. 2018) (Song et al. 2018). However, they can be bypassed by various types of attacks (such attacks not using gradients, transfer of adversarial samples, etc.) (Athalye et al.) 2018). ,. (Uesato et al. 2018).
There couple of works which are trying to understand the reason behind such lack of robustness. To citep some, in (Goodfellow et al. 2014). it is argued that the main reason may lie in DNNs' lack of non-linearity Another work argues that the perturbations cause change in the saliency of images which makes the model switch the attention to another part of it (Vargas & Su (2019). False structures that are easier learn were also shown related the problem (Thesing al., 2019)) Moreover, in (Tippaa et al. 2019), the accuracy and robustness trade-off was shown to exist.
1.2 ZERO-SHOT LEARNING
Zero-Sho learning is a method used to estimate unknown classes which do not appear in the training data. The motivation of Zero-Shot learning transfer knowledge from training classes to unknown classes. Existing methods approach the problem by estimating unknown classes from an attribute vector defined manually. Attribute vectors are annotated to both known and unknown classes, and for each class, whether an attribute, such as "colour" and "shape", belongs to the class or not is represented by 1 or O. For example, in (Lampert et al., (209) the authors proposed Direct Attribute Prediction (DAP) model, which learns each parameter for estimating the attributes from the target data. It estimates an unknown class of the source data which is estimated from the target data by using these parameters. Based on this research, other zero-shot learning methods have been proposed which uses an embedded representation generated using a natural language processing algorithm instead of manually created attribute vector (Zhang & Saligrama 2016: Fu et al. 2015 Norouzi ctt al. 2013 Akata et al. 2015 Bucher et al., 2016).
In (Zhang & Saligrama| 2015). different approach to estimate unknown classes is proposed. This method constructs the histogram of known classes distribution for an unknown class. In this approach, it is assumed that the unknown classes are the same if these histograms generated in the target domain and the source domain are similar. This perspective is similar to our approach because our method approach to represent an unknown class as the distribution of known classes. However, our objective
2

---

Under review as a conference paper at ICLR 2020
not estimating the unknown class, and do not the source domain. Our objective here is to analyze DNNs' representation by using this distribution.
ON THE LINKS BETWEEN ROBUSTNESS EVALUATION AND REPRESENTATION
In the canonical classification setting, the goal of a classifier is to achieve low expected loss:
(z.y)~ [(x, 0)
E
Robustness against adversarial attacks a slightly different setting. To achieve high robustness in this setting, a classifier should have lower adversarial loss in noise E A
E [C + 8, 0)]
(a.,))~~
Considering Mean Squared Error (MSE), we have:
E [(x 0) (r,0)~
E [1(x 6) h(x + [))7] (m.y)~
+
E (1(x ÿ(x 8)) (x.9)~
where h(x) = [[hn(x)] is the expected behavior of the prediction when averaged over many datasets, f(x) is the ground-truth and ((x) (x)) is the output after learned given dataset D. For robustness to increase, adversarial training requires that datasets should have many noisy samples, i... x+ D. However, the more noise is added to images the more D becomes close all possible images M-N,
lim D = P-RMM M
(3)
However, fxx is undefined for D E M in which y C' for the set of known classes C. Even small amount of noise may be enough to cause & C and thus (xx undefined. Would be possible evaluate the robustness and/or the quality of a model without well defined y?
To answer this question we take into account an ideal representation and the representation learned by the model a:
[[( 8; h( 6; B)))))(((((r)))) 8; 0(x 6; #))
Interestingly, although is undefined, 2 represents the features learned and well defined for any input. Moreover, by considering learned classes be clustered in z space, unsupervised learning evaluation can be used to evaluate even without a well defined We use here a famous clustering analysis index to evaluate clusters in by their intracluster distance. 2, is also possible to evaluate the representation of known and unknown classes which should share some features. Moreover, we hypothesize here that unknown classes should evaluate z with less bias because direct map from input to output is inexistent. Any projection of the input in any of the feature maps the output layer could be used as z. To take the entire projection into account, we use here z as the final projection of the input to the classes, i.c. is the soft label array
3 RAW ZERO-SHOT
In this paper, we propose to evaluate the representation learned by conducting experiments over the labt-labels of the image in unknown classes. This is based on the hypothesis that if model is
Here the the relationship between error in noise and adversarial samples please refer to (Gilmer et al. 2019)
noise be the adversarial loss instead of the worst-case error, for discussion of
Alternatively, f(x)) could be defined for any noise if an additional unknown class defined, such as with an
OpenMax layer (Bendale Bouli 2016).

---

Under review as a conference paper at ICLR 2020
capable of learning useful features, unknown class would also trigger of these features inside the model. We call this type of test over unknown classes and without any other information, Raw Zero-Shot (Figure D.
Feature(x)
Class(y)
train
test (soft-label)
bear
[1,0,0]
[0.98,0.01,0.01]
zebra
[0,1,0]
[0.2,0.8,0]
bird
[0,0,1]
[0.1,0.03,0.87]
bear 60% zebra 39%
[0.6, 0.39, 0.01]
Figure 1: Raw Zero-Shot Illustration.
The Raw Zero-Shot is supervised learning test in which only of the n classes are shown to the classifier during training. The classifier also has only - possible outputs. During testing, only unknown classes are presented to the classifier. The soft-labels outputted for given unknown class is recorded, and the process is repeated for all possible classes, removing different class each time.
To evaluate the representation quality, metrics computed over the soft-labels are used. These metrics are based on different hypothesis of what defines feature or a class. In the same way that there different types of robustness, there also different types of representation quality. Therefore, metrics are somewhat complementary, each highlighting a different aspect of the whole. The following subsections define two of them.
3.1 DAVIES-BOULDAY METRIC CLUSTERING HYPOTHESIS
Soft labels of a classifier compose space in which given image would be classified as a weighted vector concerning the previous classes learned. Considering that a cluster in this space would constitute a class, we can use clustering validation techniques to evaluate the representation (Figure 2
DBI Metric
Raw Zero-Shot
Suftlubels (N-1 daass)
(N-1 clacsifier Model)
Clecs unimwee for model
Visualisatio SoD-label Diacersional
Sofl-labwbsspeer
Hatogram of Sofl-iswel (N-3 class)
Compare between histogram
N- Classsiffer Model
Class is knowa for the model
Delete class latt-label distibutikn cormalae
I_ Illttggra Sott-shhets (K-1 class)
Histugram Seft-lablla (Ndios)
Amalgam Metric
Figure 2: Illustration of both DBI and Amalgam metrics.
Here we choose for simplicity one of the most used metric in internal cluster validation, Davies- Bouldin Index (DBI). DBI is defined as follows:
4

---

Under review as a conference paper at ICLR 2020
1/2
DBI =
cn 2
(4)
nle ;=1
which en the centroid of the cluster, is one soft-label and ne is the number of samples.
3.2 AMALGAM METRIC AMALGAM HYPOTIESSIS
If DNNs can learn the features present in the classes, it would be reasonable to consider that the sof-labels also describe a given image as a combination of the previously learned classes. This is also true when image contains unknown class. Similar to vector space in linear algebra, the soft-labels can be combined to describe unknown objects in this space. This is analogous to how children describe previously unseen objects combination of previously seen objects. Differently from the previous metric, here we are interested in the exact values of the soft-labels. However, what would constitute the correct soft-labels for given unknown class needs to be determined.
To calculate the correct soft-label of a given unknown class (amalgam proportion) automatically, we use here the assumption that accurate classifiers should output good approximation of the amalgam proportion already. Therefore, if classifier is trained in the n classes, the soft-labels of the remaining n I classes the amalgam proportion (Figure illustrates the concept). Consequently, the Amalgam Metric (AM) is defined as:
Th
nh ej
h':
jo h
(5)
n
lh;
h
AM
n 1
in which, e' is the normalized (such that they sum one) soft-label from the classifier trained over classes and is the soft-labels from the classifier trained over n classes.
4 RAW ZERO-SHOT EXPERIMENTS
Here, we conduct Raw Zero-Shot experiments to evaluate the representation of DNNs. To obtain results over a wide range of architectures, chose evaluate CapsNet (a recently proposed completely different architecture based on dynamic routing and capsules) (Sabour et al. (017). ResNet state-of-the-arr architecture based skip connections)(He et al.J 2016)). Network in Network (NIN) (an architecture which uses micro neural networks instead of linear filters) (Lin al. 2013), All Convolutional Network (AllConv) (an architecture without max pooling and fully connected (Syers)(Springenerr et al. 2015) and LeNet (a simpler architecture which is also historical mark) (LeCun et al. 1998). All the experiments are run over the CIFAR dataset by using training dataset with all the samples of one specific class removed. This process is repeated for all classes, removing the samples of different class each time.
To analyze the correlation between representation metrics and robustness against adversarial attacks, we conducted adversarial attacks on all the architectures tested using the most well known algorithms such Carlini (Carlini & Wagneri (207)), Fast Gradient Method (FGM) (Goodfellow et al. 2014). Basic Iterative Method (BIM ) Kurakin et al.) 2016). DeepFool -Moosavi-Dezfooli et al. 2016).
Projected Gradient Descent Method (PGDM) (Madry et al. 2018 (Table
For all tests, is fixed to the corresponding value given in the table. However, different methods have different meanings for e: (a) for pixel attacks, is the maximum number of pixels to be changed, (b) for threshold attacks, E the maximum amount of change allowed per pixel, (c) for FGM, BIM and PGDM, is the attack step size (input variation) and (d) for DeepFool, is the overshoot parameter.
Table 2 shows the ranking based on the attack accuracy and their required perturbation (Table IT]. In general, CapsNet is shown be the most robust, AllConv and DenseNet follow with proper
5

---

Under review as a conference paper at ICLR 2020
Model
WideResNNet
DenseNet
ResNet
NetInNet
AllConv
CapsNet
LeNet
Accuracy (in %)
NewtonFool
51.00
50.17
$1.80
53.45
62.33
75.00
66.24
PGD
94.90
94.10
94,34
91.84
89.24
79.31
88.78
Virtual
39.14
43.68
46.05
47.07
56.72
52.27
58.87
FGM
78.69
78.19
81.91
83.11
81.32
87.54
85.28
Deep Fool
44.47
48,00
49.32
51.91
62.76
68.75
61.91
BIM
98.18
97.98
97.65
97.26
96.95
96.70
94.12
Carlini
94.39
65.50
95.99
86.44
88.68
57.64
77.30
Average
71.54
68.23
73.87
73.01
76.86
73.89
76.07
L2 Score
NewtonFool
2564.0167
2675.4001
2657. 19999091
2556.8375
2555.900 9087
2451.6315
2518.5794
PGD
935.1888
934.5662
934.9485
942.2018
946.7355
962.7172
952.4294
Virtual
2656.4656
2673.4656
2629.6419
2631.2242
2620.5943
2595.9619
2579.7693
FGM
2497.5955
2534.9078
2504.6836
2515.7904
2523,1862
2506.7086
2509.134
Deep Fool
2812.1814
2627.0271
2602.9897
2583.3581
2582.5377
2490.3162
2568.3051
BIM
3328.8821
3296.0967
3295.5864
3324.4652
3404.1802
3312.3621
3566.806
Carlini
2422.023
2549.4311
2430.7893
2412.361
2411.7259
2345.2674
2377.3322
Average
2428.0504
2470.1279
2436.6498
2423.7483
2434.9812
2380.7093
2438.9079
Table I: Attack Accuracy (percentage of successful attacks correct classified samples) and Mean
L, score (L.2 difference between original sample and adversarial one) the CIFAR-10 test dataset for the different architectures.
Model
Rank (Acc)
Rank (L2)
Diff
WideResNet
DenseNet
ResNet
6
/
NIN
4
5
I
AllConv
2
3
CapsNet
LeNet
Overall ranking for both accuracy (Acc) and L2 attacks. The rankings are obtained by ordering the average accuracy and L2 for all attacks.
Table 2:
placement while the remaining networks vary depending on the perspective used analyze (i.e., higher accuracy or lower L2). These robustness rankings will be used in the next sections to verify the relationship of robustness against adversarial samples and metrics to evaluate the representation quality.
4.1 EXPERIMENTS ON DBI METRIC
Table 3 shows the results with the DBI metric (the smallest the better) and the respective ranking of each neural network. According to this metric, CapsNet possesses the best representation of all networks tested. LeNet is considered the second-best neural network regarding representation, followed by AllConv, NIN and then deeper neural networks. DBI metric matches exceptionally well with both accuracy and Lg ranking based on robustness against adversarial samples. Most differences in Hamming distance lies in the exact places in which both accuracy and L,2 rankings differ. Notice that DBI metric does not use anything related attacks and still arrives at similar rankings. further demonstrate the correlation between DBI and adversarial attacks, a Pearson correlation of the DBI for each network shown in Table This table suggests that DBI and adversarial attacks have statistical significant correlation.
The fact that LeNet and other relatively more uncomplicated networks achieve a high representation quality which is at odds with accuracy may extremely unlikely. However, discussed in (Tsipras et al., (2019). accurate models can trade-off robustness for accuracy. DBI suggests that this trade-off happens because the representation quality has worsened. Interestingly, LeNet and other

---

Under review as a conference paper at ICLR 2020
Model
DBI
Ranking
Diff
Diff
AM
Ranking
Diff
Diff
(Acc)
(L2)
(Acc)
(L2)
WideResNet
0.58110.4
4
0
249.481135.66
4
0
DenseNet
0.60=0.14
6
4
2
116.76+1000.
5
3
1
ResNet
0.63+0.13
2
1
281.29
1
0
NIN
0.62+0009
1
2
203.15
4
0
1
AllConv
0.64
4
1
2
101.12:154.99
4
1
CapsNet
0.23+0001
0
0
624.48+62.23
LeNet
0.51
2
4
1
144.10=75.18
3
3
0
Table Mean DBI, the rank regarding this representation metric and the Hamming distance the robustness rankings against adversarial samples for each neural network. AM value for each of the different architectures and their respective ranking. The Hamming distance of the DBI's and AM's rankings for both the accuracy and L2 robustness ranking against adversarial samples are also shown.
Model
Newton Fool
PGD
Virtual
FGM
Deep
BIM
Carlini
Pearson Correlation between DBI and Accuracy (p-value)
WideResNet
-0.48 (0.16)
0.01 (0.98)
-0.43 (0.22)
-0.35 (0.32)
-0.42 (0.23)
-0.00 (0.99)
-0.21 (0.57)
DenseNet
-0.51 (0.13)
-0.19 (0.60)
-0.51 (0.13)
+0.54 (0.11)
-0.51 (0.13)
-0.28 (0.43)
-0.51 (0.13)
ResNet
-0.29 (0.41)
-0.30 (0.40)
-0,29 (0.42)
-0.24 (0.50)
-0.28 (0.44)
-0.26 (0.46)
-0.18 (0.62)
NetlnNet
-0.49 (0.15)
-0.41 (0.24)
-0.51 (0.14)
-0.38 (0.27)
(0.11)
-0.68 (0.03)
-0.50
AllCony
-0.61 (0.06)
-0.27 (0.45)
-0.64 (0.05)
-0.45 (0.20)
-0.68 (0.03)
-0.60 (0.07)
-0.70 (0.02)
CapsNet
-0.71 (0.02)
0.12 (0.75)
-0.80 (0.01)
-0.45 -0.45 (0.20)
-0.70 (0.02)
-0.12 (0.75)
-0.70 (0.02)
LeNet
-0.60 (0.06)
-0.00 (0.99)
-0.54 (0.10)
-0.28 (0.43)
-0.58 (0.08)
-0.13 (0.72)
-0.61 (0.06)
Pearson Correlation between DBI and L2 Score (p-value)
WideResNet
-0.58 (0.08)
-0.61 (0.06)
-0.62 (0.06)
-0.64 (0.05)
-0.62 (0.06)
0.668 (0.03)
-0.69 (0.03)
DenseNet
-0.51 (0.13)
-0.68 (0.03)
-0.58 (0.08)
-0.71 (0.02)
-0.61 (0.06)
-0.76 (0.01)
-0.74 (0.02)
ResNet
-0.55 (0.10)
-0.60 (0.06)
-0.56 (0.09)
(0.10)
-0.55 (0.10)
-0.65 (0.04)
-0.66 (0.04)
NetlnNet
-0.64 (0.05)
-0.71 (0.02)
-0.59 (0.07)
-0.66 (0.04)
-0.61 (0.06)
(0.01)
-0.76 (0.01)
AllCony
(0.02)
-0.65 (0.04)
-0.67 (0.03)
(0.01)
-0.71 (0.02)
(..51)
-0.75 (0.01)
CapsNet
(0.01)
-0.78 (0.01)
-0.81 (0.00)
-0.81 (0.00)
(0.01)
(0.4
-0.73 (0.02)
LeNet
-0.57 (0.08)
-0.59 (0.07)
+0.57 (0.09)
-0.60 (0.07)
(0.09)
0.56 (0.09) -0.60 (0.07)
Table 4: Pearson correlation between DBI and both 1,2 score as well as accuracy of attacks
simple networks are also easier to attack (low-rank accuracy) but need more perturbation to achieve the same accuracy (high rank L2). Therefore, LeNet and other simple networks might be casier to attack because the search space is less complicated (less obfuscation (Athalye et al.) (0188). However, this does mean they are less robust. Alternatively, DBI suggests, LeNet and other simple networks might have achieved relatively good representations but without high accuracy.
To enable visualization of this metric, plotted in Figure 3 projection into two dimensions of all the points in the decision space of unknown classes. All this is done while preserving the high-dimensional distance between the points. Here we use the Isomap (Tenenbaum et al. 2000) to achieve this effect. It can be easily observed that CapsNet's 'ss results for unknown classes are more clustered and thus form better-defined cluster than other architectures.
4.2 EXPERIMENTS ON AMALGAM METRIC
In this section, the AM for all the networks is evaluated, which is based on the similarity of soft-labels for networks that were trained in all classes. The results shown Table almost the same representation ranking as the robustness rankings related to L2. Interestingly, although both DBI and AM differ widely concept and calculation procedure, the rankings are both similar and close to the L2. This further suggests that both metrics agree what would be good representation quality and can be used to evaluate representation newer methods. A visualization as well as the Pearson correlation of the metric is included in the supplementary works.
7

---

Under review as a conference paper at ICLR 2020
1010
Visualization of the results in Table 3 using topology preserving two-dimensional projection with Isomap. Each row shows the Isomap of one architecture. From top to bottom: WideResNet, DenseNet, ResNet, NIN, AllConv, CapsNet, LeNet.
Figure 3:
4.3 PARTS OF A WHOLE
In (Agarwal et al. 2019). adding loss force features be close to the feature centroid was shown to be beneficial against adversarial attacks. This consistent with the proposed DBI metric, demonstrating that both soft-label space well other feature spaces benefit from projections nearby positions. At the same time, we further support the existence of trade-off between accuracy and robustness for deeper DNNs first point out in (Tsipras al. 2019). Other types of networks, such CapsNet seem to avoid it to some extent. Therefore. the trade-off is shown to vary with the architecture and computing dynamics of model. Lastly, we hypothesize that representation bias may hold in which features learned are only invariant to the dataset and not unseen classes.
5 CONCLUSIONS
Here we proposed the Raw Zero-Shot method to evaluate the representation of classifiers. In order to score the soft-labels, metrics were formally defined based on different hypothesis of representa- tion quality. Results suggest that the evaluation of the representation of both metrics (DBI and AM) are linked with the robustness of neural networks. In other words, quickly attacked neural networks have a lower representation score. Interestingly, LeNet scores well in both metrics, albeit being the least accurate. LeNet is followed up by AllConv and NIN, which are less complicated/profoum than other models which suggest that deeper architectures might be trading-off off representation quality for accuracy. These results shown here further support the claim that there is trade-off between accuracy and robustness in current deep learning (Tsipras al. 2019)
Thus, the proposed Raw Zero-Shot was able to evaluate the representation quality of state-of-the-an DNNs and show their shortcomings concerning adversarial attacks, explaining many of the current problems. It also opens up new possibilities for both the evaluation (i.e. as a quality assessment) and the development (e.g., a loss function) of neural networks.
8

---

Under review as a conference paper at ICLR 2020
REFERENCES
C. Agarwal, A. Nguyen, and D. Schonfeld. Improving robustness to adversarial examples by encouraging discriminative features. In 2019 IEEE International Conference on Image Processing (ICIP), pp. 3801-3505 Sep. 2019. doi: 10.1 D0.119//CIP:22111888000 88800601..
Zeynep Akata, Scott Reed, Daniel Walter, Honglak Lec, and Bernt Schiele. Evaluation of output embeddings for fine-grained image classification. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 2927-2936, 2015.
Anish Athalye and llya Sutskever. Synthesizing robust adversarial examples. In ICMI., 2018.
Anish Athalye, Nicholas Carlini, and David Wagner. Obfuscated gradients give a false sense of security: Circumventing defenses to adversarial examples. In ICMI., 2018.
Abhijit Bendale and Terrance Boult. Towards open set deep networks. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 1563-1572, 2016.
Tom B Brown, Dandelion Mané, Aurko Roy, Martín Abadi, and Justin Gilmer. Adversarial patch. arXiv preprint arXiv: 171.v:1712.09665, 2017.
Maxime Bucher, Stéphane Herbin, and Frédéric Jurie. Improving semantic embedding consistency by metric learning for zero-shot classiffication. European Conference on Computer Vision, pp. 730-746. Springer, 2016.
Nicholas Carlini and David Wagner. Towards evaluating the robustness of neural networks. In 2017 IEEE Symposium on Security and Privacy (SP), pp. 39-57. IEEE, 2017.
Yanwei Fu, Yongxin Yang, Tim Hospedales, Tao Xiang, and Shaogang Gong. Transductive multi-label zero-shot learning. arXiv preprint 1533 2015.
Justin Gilmer, Nicolas Ford, Nicholas Carlini, and Ekin Cubuk. Adversarial examples are a natural consequence of test error in noise. In International Conference Machine Learning, pp. 2280- 2289, 2019.
lan J Goodfellow, Jonathon Shlens, and Christian Szegedy. Explaining and harnessing adversarial examples. arXiv preprint arXiv:1412.6572, 2014.
Chuan Guo, Mayank Rana, Moustapha Cisse, and Laurens van der Maaten. Countering adversarial images using input transformations. In ICLR, 2018
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference computer vision and pattern recognition, pp. 770-778, 2016.
Ruitong Huang, Bing Xu, Dale Schuurmans, and Csaba Szepesvári. Learning with a strong adversary. arXiv preprint 033004451, 2015.
Alexey Kurakin, Ian Goodfellow, and Samy Bengio. Adversarial examples the physical world. arXiv preprint arXiv:1607.02533 2016.
Christoph Lampert, Hannes Nickisch, and Stefan Harmeling. Learning detect unseen object classes by between-class attribute transfer. In 2009 IEEE Conference on Computer Vision and Pattern Recognition, pp. 951-958 IEEE, 2009.
Yann LeCun, Léon Bottou, Yoshua Bengio, Patrick Haffner, et al. Gradient-basse learning applied to document recognition. Proceedings of the IEEE, (1111)):27822222 1998.
Min Lin, Qiang Chen, and Shuicheng Yan. Network in network. arXiv preprint arXiv:1312.44 2013.
Xingjun Ma, Bo Li, Yisen Wang. Sarah M Erfani, Sudanthi Wijewickrema, Grant Schoenebeck. Dawn Song, Michael E Houle, and James Bailey. Characterizing adversarial subspaces using local intrinsic dimensionality. arXiv preprint arXiv: 100Iiv1801.000 001002113,, 2018

---

Under review as a conference paper at ICLR 2020
Laurens der Maaten and Geoffrey Hinton. Visualizing data using t-sne. Journal of machine learning research, 9NNov):2579 2605, 2008.
Aleksander Madry, Aleksandar Makelov, Ludwig Schmidt, Dimitris Tsipras, and Adrian Vladu. Towards deep learning models resistant adversarial attacks. In ICLR, 2018.
Seyed-Mohsen Moosavi-Dezfoolle Alhussein Fawzi, and Pascal Frossard. Deepfool: simple and accurate method to fool deep neural networks. In Proceedings of the IEEE Conference Computer Vision and Puttern Recognition, pp. 2574-2582, 2016.
Seyed-Mohsen Moosavi-Dezzoolli Alhussein Fawzi, Omar Fawzi, and Pascal Frossard. Universal adversarial perturbations. In 2017 IEEE Conference Computer Vision and Pattern Recognition (CVPR), pp. 86-94. IEEE, 2017.
Anh Nguyen, Jason Yosinski, and Jeff Clune. Deep neural networks are easily fooled: High confidence predictions for unrecognizable images. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 427-436, 2015.
Mohammad Norouzi, Tomas Mikolov, Samy Bengio, Yoram Singer, Jonathon Shlens, Andrea Frome, Greg S Corrado, and Jeffrey Dean. Zero-shot learning by convex combination of semantic embeddings. arXiv preprint arXiv: /3 13/2.5650, 2013.
Nicolas Papernot, Patrick McDaniel, Xi Wu. Somesh Jha, and Ananthram Swami. Distillation as defense to adversarial perturbations against deep neural networks. In 2016 IEEE Symposium on Security and Privacy (SP), pp. 582-597. IEEE, 2016.
Sara Sabour, Nicholas Frosst, and Geoffrey Hinton. Dynamic routing between capsules. In Advances in neural information processing systems, pp. 3856-3866, 2017.
Mahmood Sharif, Sruti Bhagavatula, Lujo Bauer, and Michael K Reiter. Accessorize a crime: Real and stealthy attacks on state-of-the-art face recognition. In Proceedings of the 2016 ACM SIGSAC Conference on Computer and Communications Security, pp. 1528-1540 ACM, 2016.
Yang Song, Taesup Kim, Sebastian Nowozin, Stefano Ermon, and Nate Kushman. Pixeldefend: Leveraging generative models to understand and defend against adversarial examples. In ICLR, 2018.
JT Springenberg, A Dosovitskiy, Brox, and M Riedmiller. Striving for simplicity: The all convolutional net. In ICLR (workshop track), 2015.
Jiawei Su, Danilo Vasconcellos Vargas, and Sakurai Kouichi. One pixel attack for fooling deep neural networks. arXiv preprint arXiv: 1710.0886, 2017.
Christian et al. Szegedy. Intriguing properties of neural networks. In In ICLR. Citeseer, 2014.
Joshua B Tenenbaum, Vin De Silva, and John C Langford. A global geometric framework for nonlinear dimensionality reduction. science, 290(5500):2319-22 239-2333,, 2000.
Laura Thesing, Vegard Antun, and Anders C Hansen. What do ai algorithms actually learn?-on false structures in deep learning. arXiv preprint arXiv: 01478, 2019.
Florian Tramèr, Alexey Kurakin, Nicolas Papernot, Ian Goodfellow, Dan Bonch, and Patrick Mc- Daniel. Ensemble adversarial training: Attacks and defenses. In ICLR, 2018.
Dimitris Tsipras, Shibani Santurkar, Logan Engstrom, Alexander Tumne, and Aleksander Madry. Ro- bustness may be at odds with accuracy. In International Conference on Learning Representations, 2019.
Jonathan Uesato, Brendan O' O'Donoghue, Pushmeet Kohli, and Aaron Oord. Adversarial risk and the dangers of evaluating against weak attacks. In International Conference on Machine Learning, pp. 5032-5041, 2018
Danilo Vasconcellos Vargas and Jiawei Su. Understanding the one-pixel attack: Propagation maps and locality analysis. arXiv preprint arXiv: arXiv:1902.029497 2019.
10

---

Under review as a conference paper at ICLR 2020
Ziming Zhang and Venkatesh Saligrama. Zero-shot learning via semantic similarity embedding. In Proceedings of the IEEE international conference on computer vision, pp. 4166-41744 2015.
Ziming Zhang and Venkatesh Saligrama. Zero-shot recognition via structured prediction. In European conference on computer vision, pp. 533-548. Springer, 2016.

---

Under review as a conference paper at ICLR 2020
airpiane
automabile e
bird
cat
deer
dlog
trog
horse
ship
truck
Figure 4: Visualization of the DBI Metric with t-Distributed Stochastic Neighbour Embedding (t-SNE) which focuses the neighbour distances. Each row shows the t-SNE projections in two dimensional space for one architecture. From top to bottom: CapsNet, AllConv, ResNet, NIN and LeNet.
SUPPLEMENTARY WORK
A THE POSSIBILITY OF A REPRESENTATION BIAS
The experiments show the possibility of representations to not work well for unknown classes. Many of these classes, however, share similar representations such as the dog and cat, truck and car. Thus, here we formulate a possible interpretation for the results.
The objective of supervised learning algorithm is perhaps to map the input to output in such way that the decision boundary reflects the real decision boundary. To achieve this, it known that when dealing with complex problems, learning algorithms need first learn set of invariant features that are present throughout classes such that their recognition becomes robust against variations in the dataset.
Human beings, however, learn set of invariant features that is not only able to solve current tasks or recognize current classes. We learn set of features that can describe most if not all unseen classes and unknown tasks. Thus, we define representation bias as the bias towards invariant features that describe current seen classes tasks but fail to describe unknown classes and tasks.
B EXTENDED ANALYSIS OF DBI METRIC
Figure shows visualization of DBI's results with t-Distributed Stochastic Neighbour Embedding
(t-SNE) MMaten & Hinton 2008). DBI results are visualized using projection into two dimensions while focusing neighbour distances. The idea for having this visualisation is to investigate whether
12

---

Under review as a conference paper at ICLR 2020
555
10g
Minil
Figure 5: Histograms from which the AM is calculated. Each row shows the histograms of one architecture. Dark shaded thinner and light shaded wider bins are respectively the soft-labels from the ground-truth from the classifier trained all classes and the soft-labels of the classifier trained 1 classes. From top to bottom: WideResNet, DenseNet, ResNet, NIN, AllConv, CapsNet, LeNet.
the cluster can be split into one or more different classes. be noted that IsoMap and t-SNE two different visualisations for the same feature space.
The figure shows that CapsNet forms pretty dense projections. Even if form clusters have different classes, the gaps between the classes will be too low relatively to other architectures. It also shows that in the high-dimensional space all the logits are moderately close to each other. While for the other architectures, there exists some points which can form their own cluster and be termed as different label. Hence for these architectures it can have one or more different classes for the same label which is contradicting to our hypothesis that there should exist only single class for single label. This inference can also be linked to adversarial attacks and supports the argument that CapsNet is relatively more robust than the other architectures.
C EXTENDED ANALYSIS OF AMALGAM METRIC
To enable visualisation of the metric, the computed histograms (h'; and h; from Equation plotted in 5 is interesting to note that in Figure the histograms from CapsNet are different from the other ones, by the entirely different architecture employed by CapsNet. This reveals that this metric can capture such representation differences. Ta that Amalgam metric is less reliable than DBI. p-values are also high enough to render its relationship less obvious. Recall, however, that the ranking based on Amalgam is very similar to the raking obtained with DBI metric.
D hi;
h where
ne
h'
Σe
e' and
Σ
ej
13

---

Under review as a conference paper at ICLR 2020
airpleme
sutomobile
dog
frog
Figure 6: Histograms of D for each soft label. Each shows the difference in histograms for one architecture. From top to bottom: CapsNet, AllConv, ResNet, NIN and LeNet.
14

---

Under review as a conference paper at ICLR 2020
Model
Newton Fool
PGD
Virtual
FGM
Deep Fool
BIM
Carlini
Pearson Correlation between Amalgam and Accuracy (p-value)
WidcResNet
0.29 (0.41)
-0.05 (0.88)
0.36 (0.31)
-0.11 (0.77)
0.32 (0.37)
-0.22 (0.53)
0.44 (0.20)
DenseNet
0.49 (0.15)
0.22 (0.55)
0.49 (0.16)
0.52 (0.13)
0.49 (0.15)
-0.26 (0.46)
0.51 (0.14)
ResNet
0.09 (0.80)
0.24 (0.51)
0.09 (0.80)
(0.788
0.09 (0.81)
0.24 (0.51)
0.04 (0.91)
NetInNet
-0.02 (0.95)
0.70 (0.02)
-0.03 (0.93)
(0.75)
0.02 (0.96)
0.71 (0.02)
0.12 (0.75)
AllConv
-0.04 (0.91)
-0.32 (0.37)
-0.04 (0.91)
(0.78)
-0.03 (0.93)
-0.27 (0.45)
0.04 (0.91)
CapsNet
(0.04)
-0.11 (0.76)
(0.03)
0.45 (U.20)
(0.02)
0.15 (0.69)
0.71 (0.02)
LeNet
(0.06)
0.73 (0.02)
(0.11)
0.81 (0.00)
(0.08)
0.72 (0.02)
0.61 (0.06)
Pearson Comelation between Amalgam and 1.2 Score (p-value)
WideResNet
0.66 (0.04)
0.59 (0.07)
0.59 (0.07)
0.66 (0.04)
0.64 (0.05)
0.71 (0.02)
0.69 (0.03)
DenseNet
0.59 (0.07)
0.70 (0.02)
0.59 (0.07)
0.72 (0.02)
0.62 (0.05)
-0.75 (0.01)
-0.71 (0.02)
ResNet
0.52 (0.12)
0.43 (0.22)
(0.18)
0.39 (0.26)
(0.15)
0.49 (0.15)
(0.14)
NetlnNet
0.35 (0.32)
0.35 (0.32)
0.39 (0.26)
0.29 (0.42)
0.37 (0.29)
0.33
0.39 (0.27)
AllConv
-0.08 (0.82)
-0.12 (0.74)
-0.10 (0.79)
-0.15 (0.69)
-0.12 (0.73)
-0.16 (0.66)
-0.15 (0.69)
CapsNet
(0.14)
0.80 (0.01)
0.41 (0.24)
0.81 (0.00)
0.44 (0.20)
0.67 (0.03)
0.50 (0.14)
LeNet
0.31 (0.38)
0.47 (0.17)
0.33 (0.35)
0.41 (0.24)
0.31 (0.38)
0.45 (0.20)
0.46 (0.18)
Table 5: Pearson correlation between Amalgam Metric and both L.2 score well as accuracy of attacks
The Amalgam Metric showed that both CapsNet and AllConv have the best scores which is in accordance with their top robustness score. Figure 66 shows a visualization of equation which is part of the main equation of Amalgam Metric. On analysing further the phenomena about absolute difference between h' and h in the Amalgam Metric or D in equation 6 It can be noted from the figure that for most labels of CapsNet and AllConv the difference relatively low than the other architectures. This contributes to have CapsNet and AllConv to have best scores. Further investigations can be carried out to analyse the effect of a label in adversarial attack based on this figure. This can also provide insight on the labels which are robust to adversarial attacks. A further study can also be carried out to analyse the characteristics of the neural network's 's representation which makes label more robust than other labels.
15


Reviews:
[{"review_id": "rkgtA6nRFB", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Summary:\nThis paper aims at revealing the relationship between the quality of deep representations and the attack susceptibility of deep classification models. To this end, they propose the zero-shot test to investigate the \"quality\" of learned representations for unknown classes. Specifically, they leverage two kinds of quality metrics on data of unknown classes. The first one is based on clustering named Davies-Bouldin Index which measures the compactness of intra-cluster. The second one is based on the difference of soft-label histogram distributions with/without unknown classes during training, which may describe the generalization for unknown classes or bias towards known classes of learned features. Finally, with these two metrics, they rank the quality of different models and compare such ranking results with the attack robustness obtained by different attack techniques on CIFAR-10 dataset.\n\n+Strengths:\n1. This paper shows satisfactory related works about recent advances in adversarial attacks and defenses. \n\n-Weaknesses:\n1. The writing of this paper contains many mistakes, especially for the issue of using singular and plural. I strongly recommend the authors to polish the paper carefully. Take the first two pages as an example, “results suggests” in Abstract, “Networks’s” in Introduction, “a DNNs”, “perturbations causes” in Sec1.1, “methods… which uses” in Sec.1.2, etc.\n2. The idea of representations quality on unknown classes determining attack robustness is not reasonable. Since the adversarial attack is defined on the known classes, the reasons for that are in two aspects. The first is the training dataset. If the collected training data cannot represent/fulfil the whole continuous distribution/manifold of the categories or even biased, the models are of course easily fooled by unseen modes during test. The second is the mapping for classification is from high to low, which is naturally many to one. In a word, I think the quality of models on unknown classes is not directly related to the problem of adversarial attack.\n3. This paper only conducts experiments on CIFAR-10 dataset, which is not convincing enough. It would be better to evaluate their method on more challenging benchmarks and also give more validation about their idea. E.g. on ImageNet, if enough number of categories has been seen for models, whether they would become more robust to adversarial attack. Evaluating features of more layers rather than the softmax outputs is also needed. \n4. What is amalgam proportion? Please explain it in detail or give a reference paper. Otherwise the readers cannot understand the motivation of the second metric in Sec3.2. Besides, Fig.2 contains little information and few captions for readers to understand their method. \n5. In Tab.1 and 2, the meanings of Attack Accuracy and Average Amount of Perturbations (typo in caption for Tab.2) are not introduced. Tab.5 shows 7 methods but its caption says “five”. Fig.4 is also confused, for AM, which histogram is with unknown classes and which one is without?\n6. The title says \"representation quality explain adversarial attacks\". After reading this paper, I haven’t found the mechanism leading to the adversarial attacks of DNNs.", "extra_scores": {"experience_assessment": "I have published one or two papers in this area.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper thoroughly.", "review_assessment:_checking_correctness_of_experiments": "I carefully checked the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I carefully checked the derivations and theory."}, "extra_text": {}}, {"review_id": "SkgpIc7wcB", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 1, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper proposes to evaluate the robustness of the neural networks by extrapolating to the unseen classes. However, the authors only include evaluation for non-robust trained models, without considering the robust trained model, such as Madry et al. [1]. The conclusion is not convincing that the authors studied the robustness using only non-robust models, because it is well known that the accuracy for attacking non-robust model can be 100% (for CIFAR-10). It is useful if the authors study whether their method can be used to measure the robustness of the robust trained models.\n\n1. The paper only evaluate robust accuracy on models without robust training.\n\n2. The evidence in the paper does not support their point. The paper shows that there's link between adversarial robustness and the generalization ability of neural network to extrapolate to unseen classes. But could not support the claim in the abstract, that \n\n\"The main idea lies in the fact that some features are present on unknown classes and that unknown classes can be defined as a combination of previous learned features without representation bias (a bias towards representation that maps only current set of input-outputs and their boundary)\"\n\nThe extrapolation score does not prove this.\n\n3. The attack accuracy in Table 1 is not convincing. In [1], the PGD for epsilon=0.3 is 100% for non-robust models and around 55% for robust models, the number in the Table matches non of these.\n\n4. The writing of this paper is not clear and has gramma issues. For example, Table 1 and 2 miss information in Caption.\n\n5. The author should cite \"Adversarial Examples Are a Natural Consequence of Test Error in Noise\", which also studied the generalization and adversarial robustness.\n\n6. This evaluation measurement is not practical, it costs more computation than one evaluates a model directly using attacks. The measurement is also fragile under adversarial attacks, that one can feed in adversarial attacks to fool the score metrics, which is not convincing.", "extra_scores": {"experience_assessment": "I have published one or two papers in this area.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_experiments": "I carefully checked the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}, {"review_id": "H1lyxtDatB", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 1, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "=== A. Summary ===\n\nThis paper proposes to evaluate the representation (here, the output of 9 probabilities for 9 classes at the last softmax output layer) of an image taken from an unseen category by two metrics:\n- The first metric (DBI) essentially measures how compact the learned manifold is (i.e. how nearby are the softmax probability vectors of images from an unseen category). \n- The second metric basically measures how close a 9-D probability output vector (\"soft labels\" in the paper) of a classifier trained on the 9 classes is to a \"ground-truth\" which is the same 9-D vector but obtained when the classifier is trained on the full 10 CIFAR-10 classes.  \n\nThe authors computed these \"representation measures\" for a set of 7 networks (6 convnets + CapsuleNet) trained on CIFAR-10.\nThen, a set of white-box adversarial attacks were performed on these 7 networks and the results (both how accurate the classifier is, and the amount of L2 perturbations required to change a label) were used as a \"robustness measures\".\nThe authors then claim that the ranking of the 7 classifiers based on the representation measures match the rankings derived from the robustness measures.\n\nThe paper attempts to find the correlation between the representation quality of a classifier with its adversarial robustness. This general direction is important and worth pursuing!\nHowever, the claimed correlation is very weakly supported by the evidence in the paper.\n\n=== B. Decision ===\n\nReject.\n\nThe aim of connecting the representation quality with adversarial robustness is interesting!\nHowever, this paper has several major issues:\n\n0. The paper uses the wrong citation format of ICLR (changing from LastName et al. 2019 --> (12) ). The paper maybe 8.5 to 9 pages long if using the original format. If this format violation was accepted, it would be unfair to other submissions and be a bad example this citation violation would be OK.\n1. The paper has many claims (e.g. \"DBI metric matches extremely well\") that were not supported by evidence or clarified.\n2. The paper claims to study classifiers' \"representation quality\". However, all the experiments were conducted on only the 10-class CIFAR-10 dataset and the \"representations\" are taken at the softmax layer as opposed to some mid-CNN feature layer (e.g. the popular fc7 in AlexNet trained on the 1000-class ImageNet). \n3. The key Definition in Sec. 2 that the paper hinges on has 0 references and is a debatable definition.\n\n=== C. Suggestions for Improvement ===\n\n- I'd advise the authors to perform careful literature review to identify where the gap to fill is before conducting the research. There is a ton of work that has been done in the intersection of zero-shot, adversarial examples, deep features, and classifier caliberation. \n\nFor example, [1] has looked at how out-of-distribution samples can be represented by the deep features of a classifier:\n[1] Bendale, A., & Boult, T. E. (2016). Towards open set deep networks. In Proceedings of the IEEE conference on computer vision and pattern recognition (pp. 1563-1572).\n- This work [2] has shown adding a \"center loss\" to force the deep features to be closer to the feature centroid (i.e. related to your DBI measure) helps improve the adversarial robustness. Indeed, the visualization in Fig. 3 is very interesting and is consistent with the result in [2].\n[2] Agarwal, C., Nguyen, A., & Schonfeld, D. (2019, September). Improving Robustness to Adversarial Examples by Encouraging Discriminative Features. In 2019 IEEE International Conference on Image Processing (ICIP) (pp. 3801-3505). IEEE.\n\n- The word \"representation quality\" here is confusing when it refers to the soft labels, which is more relevant to *caliberation* than quality of deep features as the title implies. And the paper misses this body of work in the related work. See here: https://nicholas.carlini.com/writing/2019/all-adversarial-example-papers.html\n- Try to avoid \"extremely\" or \"clearly\" in claims (and in general scientific writing) because they tend to be overclaims and it is hard for readers to interpret those intensifiers. \n- I don't see how the rankings by DBI or AM are similar to the original rankings by adversarial measures at all (except for CapsNet). At least, reporting Pearson correlation (or any statistical similarity measure) would be more convincing.\n- Section 2 is highly debatable and has no references for the \"Definition\".\n- I'd perform this work on ImageNet, at least to follow the intuition in Sec. 2. With only 10 CIFAR-10 classes, it is unrealistic how to semantically represent a \"frog\" using the other 9 classes (truck, airplane, etc).\n", "extra_scores": {"experience_assessment": "I have published in this field for several years.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper thoroughly.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}]

Author rebuttal:

Meta-review: The reviewers found the aim of the paper interesting (to connect representation quality with adversarial examples). However, the reviewers consistently pointed out writing issues, such as inaccurate or unsubstantiated claims, which are not appropriate for a scientific venue. The reviewers also found the experiments, which are on simple datasets, unconvincing.
Decision: Reject