Title: REPRESENTATION QUALITY EXPLAINS ADVERSARIAL ATTACKS

Abstract: Neural networks have been shown vulnerable to adversarial samples. Slightly perturbed input images are able to change the classification of accurate models, showing that the representation learned is not as good as previously thought. To aid the development of better neural networks, it would be important to evaluate to what extent are current neural networks' representations capturing the existing features. Here we propose a way to evaluate the representation quality of neural networks using a novel type of zero-shot test, entitled Raw Zero-Shot. The main idea lies in the fact that some features are present on unknown classes and that unknown classes can be defined as a combination of previous learned features. To evaluate the soft-labels of unknown classes, two metrics are proposed. One is based on clustering validation techniques (Davies-Bouldin Index) and the other is based on soft-label distance of a given correct soft-label. Experiments show that such metrics are in accordance with the robustness to adversarial attacks and might serve as a guidance to build better models as well as be used in loss functions to create new types of neural networks. Interestingly, the results suggests that dynamic routing networks such as CapsNet have better representation while current deeper DNNs are trading off representation quality for accuracy.

1. INTRODUCTION:

Adversarial samples are slightly perturbed inputs that can make neural networks misclassify. They are carefully crafted by searching for variations in the input that, for example, could decrease the soft-labels of the correct class. Since they were discovered some years ago (Szegedy, 2014) , the number of adversarial samples have grown in both number and types. Random noise was shown to be recognized with high confidence by neural networks (Nguyen et al., 2015) , universal perturbations, that can be added to almost any image to generate an adversarial sample, were shown to exist (Moosavi-Dezfooli et al., 2017) , and the addition of crafted patches was shown to cause networks to misclassify (Brown et al., 2017) . Only one pixel is enough to make networks misclassify (Su et al., 2017) . Such attacks can also be easily transferred to real-world scenarios (Kurakin et al., 2016) , (Athalye & Sutskever, 2018) , which confers a big issue as well as a security risk for current deep neural networks' applications.

Albeit the existence of many defences, there is not any known learning algorithm or procedure that can defend against adversarial attacks consistently. Many works have tried to defend by hiding or modifying the gradients to make neural networks harder to attack. However, a recent paper shows that most of these defences fall into the class of obfuscated gradients which have their shortcomings (e.g., they can be easily bypassed by transferable attacks) (Athalye et al., 2018) . Additionally, the use of an augmented dataset with adversarial samples (named adversarial training) is perhaps one of the most successful approaches to construct robust neural networks (Goodfellow et al., 2014) , (Huang et al., 2015) , (Madry et al., 2018) . However, it is still vulnerable to attacks and has a strong bias in the type of adversarial samples used in training (Tramèr et al., 2018) . This shows that a deeper understanding of the issues is needed to enable more consistent defences to be created. Few works focused on understanding the reason behind such lack of robustness. In (Goodfellow et al., 2014) , it is argued that Deep Neural Networks's (DNN) linearity are one of the main reasons. Recent investigations reveal that attacks are changing where the algorithm is paying attention (Vargas & Su, 2019) , other experiments show that deep learning neural networks learn false structures that are easier to learn rather than the ones expected (Thesing et al., 2019) and an accuracy and robustness trade-off for models were shown to exist (Tsipras et al., 2019) .

In this paper, we propose a methodology of how to evaluate the representation of machine learning methods. Based on these metrics, we reveal a link between deep representations' quality and attack susceptibility. Specifically, we propose a test called Raw Zero-Shot and two metrics to evaluate DNN's representations.

1.1. RECENT ADVANCES IN ATTACKS AND DEFENSES:

DNNs were shown vulnerable to many types of attacks. For example, the output high confidence results to noise images (Nguyen et al., 2015) , universal perturbations in which a single perturbation can be added to almost any input to create an adversarial sample are possible (Moosavi-Dezfooli et al., 2017) , the addition of image patches can also make them misclassify (Brown et al., 2017) . Moreover, the vulnerability can be exploited even with a single pixel, i.e., changing a single pixel is often enough to make a DNNs misclassify (Su et al., 2017) . Most of these attacks can be transformed into real-world attacks by merely printing the adversarial samples (Kurakin et al., 2016) . Moreover, crafted glasses (Sharif et al., 2016) or even general 3d adversarial objects (Athalye & Sutskever, 2018) can be used as attacks.

Although many defensive systems were proposed to tackle the current problems, there is still no consistent solution available. Defensive distillation in which a smaller neural network squeezes the content learned by the original DNN was proposed (Papernot et al., 2016) . However, it was shown not to be robust enough (Carlini & Wagner, 2017) . Adversarial training was also proposed as a defence, in which adversarial samples are used to augment the training dataset (Goodfellow et al., 2014) , (Huang et al., 2015) , (Madry et al., 2018) . With adversarial training, DNNs increase slightly in robustness but not without a bias towards the adversarial samples used and while still being vulnerable to attacks in general (Tramèr et al., 2018) . There are many recent variations of defenses in which the objective is to hide the gradients (obfuscated gradients) (Ma et al., 2018) , (Guo et al., 2018) (Song et al., 2018) . However, they can be bypassed by various types of attacks (such as attacks not using gradients, transfer of adversarial samples, etc.) (Athalye et al., 2018) , (Uesato et al., 2018) .

There are a couple of works which are trying to understand the reason behind such lack of robustness. To citep some, in (Goodfellow et al., 2014) , it is argued that the main reason may lie in DNNs' lack of non-linearity. Another work argues that the perturbations cause a change in the saliency of images which makes the model switch the attention to another part of it (Vargas & Su, 2019) . False structures that are easier to learn were also shown related to the problem (Thesing et al., 2019) . Moreover, in (Tsipras et al., 2019) , the accuracy and robustness trade-off was shown to exist.

1.2. ZERO-SHOT LEARNING:

Zero-Shot learning is a method used to estimate unknown classes which do not appear in the training data. The motivation of Zero-Shot learning is to transfer knowledge from training classes to unknown classes. Existing methods approach the problem by estimating unknown classes from an attribute vector defined manually. Attribute vectors are annotated to both known and unknown classes, and for each class, whether an attribute, such as "colour" and "shape", belongs to the class or not is represented by 1 or 0. For example, in (Lampert et al., 2009) the authors proposed Direct Attribute Prediction (DAP) model, which learns each parameter for estimating the attributes from the target data. It estimates an unknown class of the source data which is estimated from the target data by using these parameters. Based on this research, other zero-shot learning methods have been proposed which uses an embedded representation generated using a natural language processing algorithm instead of a manually created attribute vector (Zhang & Saligrama, 2016; Fu et al., 2015; Norouzi et al., 2013; Akata et al., 2015; Bucher et al., 2016) .

In (Zhang & Saligrama, 2015) , a different approach to estimate unknown classes is proposed. This method constructs the histogram of known classes distribution for an unknown class. In this approach, it is assumed that the unknown classes are the same if these histograms generated in the target domain and the source domain are similar. This perspective is similar to our approach because our method approach to represent an unknown class as the distribution of known classes. However, our objective is not estimating the unknown class, and we do not use the source domain. Our objective here is to analyze DNNs' representation by using this distribution.

2. ON THE LINKS BETWEEN ROBUSTNESS EVALUATION AND REPRESENTATION:

In the canonical classification setting, the goal of a classifier is to achieve low expected loss:

EQUATION

Robustness against adversarial attacks is a slightly different setting. To achieve a high robustness in this setting, a classifier should have a lower adversarial loss in noise δ ∈ ∆1 :

EQUATION

Considering Mean Squared Error (MSE), we have:

E (x,y)∼D [L(x + δ, y; θ)] = E (x,y)∼D [(f (x + δ) -h(x + δ)) 2 ] + E (x,y)∼D [(h(x + δ) -ŷ(x + δ)) 2 ],

where

h(x) = E[h D (x)]

is the expected behavior of the prediction when averaged over many datasets, f (x) is the ground-truth and ŷ(x) = h D (x) is the output after learned on a given dataset D. For robustness to increase, adversarial training requires that datasets should have many noisy samples, i.e., x + δ ∈ D. However, the more noise is added to images the more D becomes close to all possible images R M * N :

lim ∆→∞ D = R M * N (3) However, f (x) is undefined 2 for D ∈ R M * N in which y / ∈ C

for the set of known classes C. Even a small amount of noise may be enough to cause y / ∈ C and thus f (x) undefined. Would it be possible to evaluate the robustness and/or the quality of a model without a well defined y?

To answer this question we take into account an ideal representation z and the representation learned by the model ẑ:

E[(f (x + δ; z) -h(x + δ; ẑ)) 2 ] + E[(h(x + δ; ẑ) -ŷ(x + δ; ẑ)) 2 ]

Interestingly, although y is undefined, z represents the features learned and is well defined for any input. Moreover, by considering learned classes to be clustered in z space, unsupervised learning evaluation can be used to evaluate z even without a well defined y. We use here a famous clustering analysis index to evaluate clusters in z by their intracluster distance. In z, it is also possible to evaluate the representation of known and unknown classes which should share some features. Moreover, we hypothesize here that unknown classes should evaluate z with less bias because a direct map from input to output is inexistent. Any projection of the input in any of the feature maps or the output layer could be used as z. To take the entire projection into account, we use here z as the final projection of the input to the classes, i.e. z is the soft label array e.

3. RAW ZERO-SHOT:

In this paper, we propose to evaluate the representation learned by conducting experiments over the soft-labels of the image in unknown classes. This is based on the hypothesis that if a model is capable of learning useful features, an unknown class would also trigger some of these features inside the model. We call this type of test over unknown classes and without any other information, Raw Zero-Shot (Figure 1 ). To evaluate the representation quality, metrics computed over the soft-labels are used. These metrics are based on a different hypothesis of what defines a feature or a class. In the same way that there are different types of robustness, there are also different types of representation quality. Therefore, metrics are somewhat complementary, each highlighting a different aspect of the whole. The following subsections define two of them.

3.1. DAVIES-BOULDIN METRIC -CLUSTERING HYPOTHESIS:

Soft labels of a classifier compose a space in which a given image would be classified as a weighted vector concerning the previous classes learned. Considering that a cluster in this space would constitute a class, we can use clustering validation techniques to evaluate the representation (Figure 2 ). Here we choose for simplicity one of the most used metric in internal cluster validation, Davies-Bouldin Index (DBI). DBI is defined as follows:

EQUATION

)

in which cn is the centroid of the cluster, e is one soft-label and n e is the number of samples.

3.2. AMALGAM METRIC -AMALGAM HYPOTHESIS:

If DNNs can learn the features present in the classes, it would be reasonable to consider that the sof-labels also describe a given image as a combination of the previously learned classes. This is also true when an image contains an unknown class. Similar to a vector space in linear algebra, the soft-labels can be combined to describe unknown objects in this space. This is analogous to how children describe previously unseen objects as a combination of previously seen objects. Differently from the previous metric, here we are interested in the exact values of the soft-labels. However, what would constitute the correct soft-labels for a given unknown class needs to be determined.

To calculate the correct soft-label of a given unknown class (amalgam proportion) automatically, we use here the assumption that accurate classifiers should output a good approximation of the amalgam proportion already. Therefore, if a classifier is trained in the n classes, the soft-labels of the remaining n -1 classes is the amalgam proportion (Figure 2 illustrates the concept). Consequently, the Amalgam Metric (AM) is defined as:

EQUATION

in which, e' is the normalized (such that they sum to one) soft-label from the classifier trained over n classes and e is the soft-labels from the classifier trained over n -1 classes.

4. RAW ZERO-SHOT EXPERIMENTS:

Here, we conduct Raw Zero-Shot experiments to evaluate the representation of DNNs. To obtain results over a wide range of architectures, we chose to evaluate CapsNet (a recently proposed completely different architecture based on dynamic routing and capsules) (Sabour et al., 2017) , ResNet (a state-of-the-art architecture based on skip connections) (He et al., 2016) , Network in Network (NIN) (an architecture which uses micro neural networks instead of linear filters) (Lin et al., 2013) , All Convolutional Network (AllConv) (an architecture without max pooling and fully connected layers) (Springenberg et al., 2015) and LeNet (a simpler architecture which is also a historical mark) (LeCun et al., 1998) . All the experiments are run over the CIFAR dataset by using a training dataset with all the samples of one specific class removed. This process is repeated for all classes, removing the samples of a different class each time.

To analyze the correlation between representation metrics and robustness against adversarial attacks, we conducted adversarial attacks on all the architectures tested using the most well known algorithms such as Carlini (Carlini & Wagner, 2017) , Fast Gradient Method (FGM) (Goodfellow et al., 2014) , Basic Iterative Method (BIM ) (Kurakin et al., 2016 ), DeepFool (Moosavi-Dezfooli et al., 2016) , Projected Gradient Descent Method (PGDM) (Madry et al., 2018) (Table 1 ).

For all tests, ε is fixed to the corresponding value given in the table. However, different methods have different meanings for ε: (a) for pixel attacks, ε is the maximum number of pixels to be changed, (b) for threshold attacks, ε is the maximum amount of change allowed per pixel, (c) for FGM, BIM and PGDM, ε is the attack step size (input variation) and (d) for DeepFool, ε is the overshoot parameter.

Table 2 shows the ranking based on the attack accuracy and their required perturbation ( placement while the remaining networks vary depending on the perspective used to analyze (i.e., higher accuracy or lower L 2 ). These robustness rankings will be used in the next sections to verify the relationship of robustness against adversarial samples and metrics to evaluate the representation quality.

4.1. EXPERIMENTS ON DBI METRIC:

Table 3 shows the results with the DBI metric (the smallest the better) and the respective ranking of each neural network. According to this metric, CapsNet possesses the best representation of all networks tested. LeNet is considered the second-best neural network regarding representation, followed by AllConv, NIN and then deeper neural networks. DBI metric matches exceptionally well with both accuracy and L 2 ranking based on robustness against adversarial samples. Most differences in Hamming distance lies in the exact places in which both accuracy and L 2 rankings differ. Notice that DBI metric does not use anything related to attacks and still arrives at similar rankings. To further demonstrate the correlation between DBI and adversarial attacks, a Pearson correlation of the DBI for each network is shown in Table 4 . This table suggests that DBI and adversarial attacks have a statistical significant correlation.

The fact that LeNet and other relatively more uncomplicated networks achieve a high representation quality which is at odds with accuracy may seem extremely unlikely. However, as discussed in (Tsipras et al., 2019) simple networks are also easier to attack (low-rank accuracy) but need more perturbation to achieve the same accuracy (high rank L 2 ). Therefore, LeNet and other simple networks might be easier to attack because the search space is less complicated (less obfuscation (Athalye et al., 2018) ). However, this does not mean they are less robust. Alternatively, as DBI suggests, LeNet and other simple networks might have achieved relatively good representations but without high accuracy.

To enable visualization of this metric, we plotted in Figure 3 a projection into two dimensions of all the points in the decision space of unknown classes. All this is done while preserving the high-dimensional distance between the points. Here we use the Isomap (Tenenbaum et al., 2000) to achieve this effect. It can be easily observed that CapsNet's results for unknown classes are more clustered and thus form a better-defined cluster than other architectures.

4.2. EXPERIMENTS ON AMALGAM METRIC:

In this section, the AM for all the networks is evaluated, which is based on the similarity of soft-labels for networks that were trained in all classes. The results shown in Table 3 reveal almost the same representation ranking as the robustness rankings related to L 2 . Interestingly, although both DBI and AM differ widely in concept and calculation procedure, the rankings are both similar and close to the L 2 . This further suggests that both metrics agree on what would be a good representation quality and can be used to evaluate representation in newer methods. A visualization as well as the Pearson correlation of the metric is included in the supplementary works. 

4.3. PARTS OF A WHOLE:

In (Agarwal et al., 2019) , adding a loss to force features to be close to the feature centroid was shown to be beneficial against adversarial attacks. This is consistent with the proposed DBI metric, demonstrating that both soft-label space as well as other feature spaces benefit from projections to nearby positions. At the same time, we further support the existence of a trade-off between accuracy and robustness for deeper DNNs first point out in (Tsipras et al., 2019) . Other types of networks, such as CapsNet seem to avoid it to some extent. Therefore, the trade-off is shown to vary with the architecture and computing dynamics of a model. Lastly, we hypothesize that a representation bias may hold in which features learned are only invariant to the dataset and not unseen classes.

5. CONCLUSIONS:

Here we proposed the Raw Zero-Shot method to evaluate the representation of classifiers. In order to score the soft-labels, two metrics were formally defined based on a different hypothesis of representation quality. Results suggest that the evaluation of the representation of both metrics (DBI and AM) are linked with the robustness of neural networks. In other words, quickly attacked neural networks have a lower representation score. Interestingly, LeNet scores well in both metrics, albeit being the least accurate. LeNet is followed up by AllConv and NIN, which are less complicated/profound than other models which suggest that deeper architectures might be trading-off representation quality for accuracy. These results shown here further support the claim that there is a trade-off between accuracy and robustness in current deep learning (Tsipras et al., 2019) .

Thus, the proposed Raw Zero-Shot was able to evaluate the representation quality of state-of-the-art DNNs and show their shortcomings concerning adversarial attacks, explaining many of the current problems. It also opens up new possibilities for both the evaluation (i.e. as a quality assessment) and the development (e.g., as a loss function) of neural networks. 

SUPPLEMENTARY WORK A THE POSSIBILITY OF A REPRESENTATION BIAS:

The experiments show the possibility of representations to not work well for unknown classes. Many of these classes, however, share similar representations such as the dog and cat, truck and car. Thus, here we formulate a possible interpretation for the results.

The objective of a supervised learning algorithm is perhaps to map the input to output in such a way that the decision boundary reflects the real decision boundary. To achieve this, it is known that when dealing with complex problems, learning algorithms need first to learn a set of invariant features that are present throughout classes such that their recognition becomes robust against variations in the dataset.

Human beings, however, learn a set of invariant features that is not only able to solve current tasks or recognize current classes. We learn a set of features that can describe most if not all unseen classes and unknown tasks. Thus, we define representation bias as the bias towards invariant features that describe current seen classes or tasks but fail to describe unknown classes and tasks.

B EXTENDED ANALYSIS OF DBI METRIC:

Figure 4 shows a visualization of DBI's results with t-Distributed Stochastic Neighbour Embedding (t-SNE) (Maaten & Hinton, 2008) . DBI results are visualized using a projection into two dimensions while focusing on neighbour distances. The idea for having this visualisation is to investigate whether The figure 4 shows that CapsNet forms pretty dense projections. Even if we form clusters to have different classes, the gaps between the classes will be too low relatively to other architectures. It also shows that in the high-dimensional space all the logits are moderately close to each other. While for the other architectures, there exists some points which can form their own cluster and be termed as different label. Hence for these architectures it can have one or more different classes for the same label which is contradicting to our hypothesis that there should exist only a single class for a single label. This inference can also be linked to adversarial attacks and supports the argument that CapsNet is relatively more robust than the other architectures.

C EXTENDED ANALYSIS OF AMALGAM METRIC:

To enable a visualisation of the metric, the computed histograms (h' i and h i from Equation 5) are plotted in Figure 5 . It is interesting to note that in Figure 5 , the histograms from CapsNet are different from the other ones, by the entirely different architecture employed by CapsNet. This reveals that this metric can capture such representation differences. Table 5 reveals that Amalgam metric is less reliable than DBI. Its p-values are also high enough to render its relationship less obvious. Recall, however, that the ranking based on Amalgam is very similar to the raking obtained with DBI metric. The Amalgam Metric showed that both CapsNet and AllConv have the best scores which is in accordance with their top robustness score. Figure 6 shows a visualization of equation 6 which is part of the main equation of Amalgam Metric. On analysing further the phenomena about absolute difference between h' and h in the Amalgam Metric or D in equation 6. It can be noted from the figure that for most labels of CapsNet and AllConv the difference is relatively low than the other architectures. This contributes to have CapsNet and AllConv to have best scores. Further investigations can be carried out to analyse the effect of a label in adversarial attack based on this figure. This can also provide insight on the labels which are robust to adversarial attacks. A further study can also be carried out to analyse the characteristics of the neural network's representation which makes a label more robust than other labels.

D = | h' i -h i |,

:

Here we use the error in noise to be the adversarial loss instead of the worst-case error, for a discussion of the relationship between error in noise and adversarial samples please refer to(Gilmer et al.,

2019)2 Alternatively, f (x) could be defined for any noise if an additional unknown class is defined, such as with an OpenMax layer(Bendale & Boult, 2016).

Figures and Tables:

Figure 1: Raw Zero-Shot Illustration.

Figure 2: Illustration of both DBI and Amalgam metrics.

Figure 3: Visualization of the results in Table 3 using a topology preserving two-dimensional projection with Isomap. Each row shows the Isomap of one architecture. From top to bottom: WideResNet, DenseNet, ResNet, NIN, AllConv, CapsNet, LeNet.

Figure 4: Visualization of the DBI Metric with t-Distributed Stochastic Neighbour Embedding (t-SNE) which focuses on the neighbour distances. Each row shows the t-SNE projections in two dimensional space for one architecture. From top to bottom: CapsNet, AllConv, ResNet, NIN and LeNet.

Figure 5: Histograms from which the AM is calculated. Each row shows the histograms of one architecture. Dark shaded thinner and light shaded wider bins are respectively the soft-labels from the ground-truth from the classifier trained on all classes and the soft-labels of the classifier trained on n -1 classes. From top to bottom: WideResNet, DenseNet, ResNet, NIN, AllConv, CapsNet, LeNet.

Figure 6: Histograms of D for each soft label. Each row shows the difference in histograms for one architecture. From top to bottom: CapsNet, AllConv, ResNet, NIN and LeNet.

Attack Accuracy (percentage of successful attacks on correct classified samples) and Mean L2 score (L2 difference between original sample and adversarial one) for the CIFAR-10 test dataset for the different architectures.

Table content: <table><tr><td>Model</td><td>WideResNet</td><td>DenseNet</td><td>ResNet</td><td>NetInNet</td><td>AllConv</td><td>CapsNet</td><td>LeNet</td></tr><tr><td/><td/><td/><td colspan="2">Accuracy (in %)</td><td/><td/><td/></tr><tr><td>NewtonFool</td><td>51.00</td><td>50.17</td><td>51.80</td><td>53.45</td><td>62.33</td><td>75.00</td><td>66.24</td></tr><tr><td>PGD</td><td>94.90</td><td>94.10</td><td>94.34</td><td>91.84</td><td>89.24</td><td>79.31</td><td>88.78</td></tr><tr><td>Virtual</td><td>39.14</td><td>43.68</td><td>46.05</td><td>47.07</td><td>56.72</td><td>52.27</td><td>58.87</td></tr><tr><td>FGM</td><td>78.69</td><td>78.19</td><td>81.91</td><td>83.11</td><td>81.32</td><td>87.54</td><td>85.28</td></tr><tr><td>Deep Fool</td><td>44.47</td><td>48.00</td><td>49.32</td><td>51.91</td><td>62.76</td><td>68.75</td><td>61.91</td></tr><tr><td>BIM</td><td>98.18</td><td>97.98</td><td>97.65</td><td>97.26</td><td>96.95</td><td>96.70</td><td>94.12</td></tr><tr><td>Carlini</td><td>94.39</td><td>65.50</td><td>95.99</td><td>86.44</td><td>88.68</td><td>57.64</td><td>77.30</td></tr><tr><td>Average</td><td>71.54</td><td>68.23</td><td>73.87</td><td>73.01</td><td>76.86</td><td>73.89</td><td>76.07</td></tr><tr><td/><td/><td/><td colspan="2">L2 Score</td><td/><td/><td/></tr><tr><td>NewtonFool</td><td colspan="7">2564.0167 2675.4007 2657.9091 2556.8375 2555.9087 2451.6315 2518.5794</td></tr><tr><td>PGD</td><td>935.1888</td><td>934.5662</td><td>934.9485</td><td>942.2018</td><td>946.7355</td><td>962.7172</td><td>952.4294</td></tr><tr><td>Virtual</td><td colspan="7">2656.4656 2673.4656 2629.6419 2631.2242 2620.5943 2595.9619 2579.7693</td></tr><tr><td>FGM</td><td colspan="6">2497.5955 2534.9078 2504.6836 2515.7904 2523.1862 2506.7086</td><td>2509.134</td></tr><tr><td>Deep Fool</td><td colspan="7">2592.1814 2627.0271 2602.9897 2583.3581 2582.5377 2490.3162 2568.3051</td></tr><tr><td>BIM</td><td colspan="6">3328.8821 3296.0967 3295.5864 3324.4652 3404.1802 3312.3621</td><td>3566.806</td></tr><tr><td>Carlini</td><td colspan="3">2422.023 2549.4311 2430.7893</td><td colspan="4">2412.361 2411.7259 2345.2674 2377.3322</td></tr><tr><td>Average</td><td colspan="7">2428.0504 2470.1279 2436.6498 2423.7483 2434.9812 2380.7093 2438.9079</td></tr><tr><td/><td/><td>Model</td><td colspan="3">Rank (Acc) Rank (L 2 ) Diff</td><td/><td/></tr><tr><td/><td/><td>WideResNet</td><td>3</td><td>7</td><td>4</td><td/><td/></tr><tr><td/><td/><td>DenseNet</td><td>2</td><td>4</td><td>2</td><td/><td/></tr><tr><td/><td/><td>ResNet</td><td>7</td><td>6</td><td>1</td><td/><td/></tr><tr><td/><td/><td>NIN</td><td>4</td><td>5</td><td>1</td><td/><td/></tr><tr><td/><td/><td>AllConv</td><td>5</td><td>2</td><td>3</td><td/><td/></tr><tr><td/><td/><td>CapsNet</td><td>1</td><td>1</td><td>0</td><td/><td/></tr><tr><td/><td/><td>LeNet</td><td>6</td><td>3</td><td>3</td><td/><td/></tr></table>

Overall ranking for both accuracy (Acc) and L 2 attacks. The rankings are obtained by ordering the average accuracy and L 2 for all attacks.

Table content: <table/>

Mean DBI, the rank regarding this representation metric and the Hamming distance to the robustness rankings against adversarial samples for each neural network. AM value for each of the different architectures and their respective ranking. The Hamming distance of the DBI's and AM's rankings for both the accuracy and L 2 robustness ranking against adversarial samples are also shown.

Table content: <table><tr><td>Model</td><td>DBI</td><td>Ranking</td><td>Diff</td><td>Diff</td><td/><td>AM</td><td>Ranking</td><td>Diff</td><td>Diff</td></tr><tr><td/><td/><td/><td colspan="2">(Acc) (L 2 )</td><td/><td/><td/><td colspan="2">(Acc) (L 2 )</td></tr><tr><td colspan="2">WideResNet 0.58±0.14</td><td>7</td><td>4</td><td>0</td><td colspan="2">249.48±135.60</td><td>7</td><td>4</td><td>0</td></tr><tr><td>DenseNet</td><td>0.60±0.14</td><td>6</td><td>4</td><td>2</td><td colspan="2">296.76±100.63</td><td>5</td><td>3</td><td>1</td></tr><tr><td>ResNet</td><td>0.63±0.13</td><td>5</td><td>2</td><td>1</td><td colspan="2">281.29±107.82</td><td>6</td><td>1</td><td>0</td></tr><tr><td>NIN</td><td>0.62±0.09</td><td>3</td><td>1</td><td>2</td><td colspan="2">203.15±93.08</td><td>4</td><td>0</td><td>1</td></tr><tr><td>AllConv</td><td>0.64±0.10</td><td>4</td><td>1</td><td>2</td><td colspan="2">101.12±54.98</td><td>1</td><td>4</td><td>1</td></tr><tr><td>CapsNet</td><td>0.23±0.01</td><td>1</td><td>0</td><td>0</td><td colspan="2">124.48±62.23</td><td>2</td><td>1</td><td>1</td></tr><tr><td>LeNet</td><td>0.51±0.02</td><td>2</td><td>4</td><td>1</td><td colspan="2">144.10±75.18</td><td>3</td><td>3</td><td>0</td></tr><tr><td>Model</td><td>Newton Fool</td><td>PGD</td><td>Virtual</td><td/><td>FGM</td><td>Deep Fool</td><td>BIM</td><td/><td>Carlini</td></tr><tr><td/><td colspan="6">Pearson Correlation between DBI and Accuracy (p-value)</td><td/><td/></tr><tr><td>WideResNet</td><td>-0.48 (0.16)</td><td colspan="8">0.01 (0.98) -0.43 (0.22) -0.35 (0.32) -0.42 (0.23) -0.00 (0.99) -0.21 (0.57)</td></tr><tr><td>DenseNet</td><td colspan="9">-0.51 (0.13) -0.19 (0.60) -0.51 (0.13) -0.54 (0.11) -0.51 (0.13) -0.28 (0.43) -0.51 (0.13)</td></tr><tr><td>ResNet</td><td colspan="9">-0.29 (0.41) -0.30 (0.40) -0.29 (0.42) -0.24 (0.50) -0.28 (0.44) -0.26 (0.46) -0.18 (0.62)</td></tr><tr><td>NetInNet</td><td colspan="9">-0.49 (0.15) -0.41 (0.24) -0.51 (0.14) -0.38 (0.27) -0.54 (0.11) -0.68 (0.03) -0.50 (0.14)</td></tr><tr><td>AllConv</td><td colspan="9">-0.61 (0.06) -0.27 (0.45) -0.64 (0.05) -0.45 (0.20) -0.68 (0.03) -0.60 (0.07) -0.70 (0.02)</td></tr><tr><td>CapsNet</td><td>-0.71 (0.02)</td><td colspan="8">0.12 (0.75) -0.80 (0.01) -0.45 (0.20) -0.70 (0.02) -0.12 (0.75) -0.70 (0.02)</td></tr><tr><td>LeNet</td><td colspan="9">-0.60 (0.06) -0.00 (0.99) -0.54 (0.10) -0.28 (0.43) -0.58 (0.08) -0.13 (0.72) -0.61 (0.06)</td></tr><tr><td/><td colspan="6">Pearson Correlation between DBI and L2 Score (p-value)</td><td/><td/></tr><tr><td>WideResNet</td><td colspan="9">-0.58 (0.08) -0.61 (0.06) -0.62 (0.06) -0.64 (0.05) -0.62 (0.06) -0.68 (0.03) -0.69 (0.03)</td></tr><tr><td>DenseNet</td><td colspan="9">-0.51 (0.13) -0.68 (0.03) -0.58 (0.08) -0.71 (0.02) -0.61 (0.06) -0.76 (0.01) -0.74 (0.02)</td></tr><tr><td>ResNet</td><td colspan="9">-0.55 (0.10) -0.60 (0.06) -0.56 (0.09) -0.55 (0.10) -0.55 (0.10) -0.65 (0.04) -0.66 (0.04)</td></tr><tr><td>NetInNet</td><td colspan="9">-0.64 (0.05) -0.71 (0.02) -0.59 (0.07) -0.66 (0.04) -0.61 (0.06) -0.74 (0.01) -0.76 (0.01)</td></tr><tr><td>AllConv</td><td colspan="9">-0.70 (0.02) -0.65 (0.04) -0.67 (0.03) -0.74 (0.01) -0.71 (0.02) -0.75 (0.01) -0.75 (0.01)</td></tr><tr><td>CapsNet</td><td colspan="9">-0.80 (0.01) -0.78 (0.01) -0.81 (0.00) -0.81 (0.00) -0.76 (0.01) -0.74 (0.01) -0.73 (0.02)</td></tr><tr><td>LeNet</td><td colspan="9">-0.57 (0.08) -0.59 (0.07) -0.57 (0.09) -0.60 (0.07) -0.56 (0.09) -0.56 (0.09) -0.60 (0.07)</td></tr></table>

Pearson correlation between DBI and both L2 score as well as accuracy of attacks

Table content: <table/>

Pearson correlation between Amalgam Metric and both L2 score as well as accuracy of attacks

Table content: <table/>

Peer review (iclr2020):
=== A. Summary === This paper proposes to evaluate the representation (here, the output of 9 probabilities for 9 classes at the last softmax output layer) of an image taken from an unseen category by two metrics: - The first metric (DBI) essentially measures how compact the learned manifold is (i.e. how nearby are the softmax probability vectors of images from an unseen category). - The second metric basically measures how close a 9-D probability output vector (*soft labels* in the paper) of a classifier trained on the 9 classes is to a *ground-truth* which is the same 9-D vector but obtained when the classifier is trained on the full 10 CIFAR-10 classes. The authors computed these *representation measures* for a set of 7 networks (6 convnets + CapsuleNet) trained on CIFAR-10. Then, a set of white-box adversarial attacks were performed on these 7 networks and the results (both how accurate the classifier is, and the amount of L2 perturbations required to change a label) were used as a *robustness measures*. The authors then claim that the ranking of the 7 classifiers based on the representation measures match the rankings derived from the robustness measures. The paper attempts to find the correlation between the representation quality of a classifier with its adversarial robustness. This general direction is important and worth pursuing! However, the claimed correlation is very weakly supported by the evidence in the paper. === B. Decision === Reject. The aim of connecting the representation quality with adversarial robustness is interesting! However, this paper has several major issues: 0. The paper uses the wrong citation format of ICLR (changing from LastName et al. 2019 --> (12) ). The paper maybe 8.5 to 9 pages long if using the original format. If this format violation was accepted, it would be unfair to other submissions and be a bad example this citation violation would be OK. 1. The paper has many claims (e.g. *DBI metric matches extremely well*) that were not supported by evidence or clarified. 2. The paper claims to study classifiers* *representation quality*. However, all the experiments were conducted on only the 10-class CIFAR-10 dataset and the *representations* are taken at the softmax layer as opposed to some mid-CNN feature layer (e.g. the popular fc7 in AlexNet trained on the 1000-class ImageNet). 3. The key Definition in Sec. 2 that the paper hinges on has 0 references and is a debatable definition. === C. Suggestions for Improvement === - I*d advise the authors to perform careful literature review to identify where the gap to fill is before conducting the research. There is a ton of work that has been done in the intersection of zero-shot, adversarial examples, deep features, and classifier caliberation. For example, [1] has looked at how out-of-distribution samples can be represented by the deep features of a classifier: [1] Bendale, A., & Boult, T. E. (2016). Towards open set deep networks. In Proceedings of the IEEE conference on computer vision and pattern recognition (pp. 1563-1572). - This work [2] has shown adding a *center loss* to force the deep features to be closer to the feature centroid (i.e. related to your DBI measure) helps improve the adversarial robustness. Indeed, the visualization in Fig. 3 is very interesting and is consistent with the result in [2]. [2] Agarwal, C., Nguyen, A., & Schonfeld, D. (2019, September). Improving Robustness to Adversarial Examples by Encouraging Discriminative Features. In 2019 IEEE International Conference on Image Processing (ICIP) (pp. 3801-3505). IEEE. - The word *representation quality* here is confusing when it refers to the soft labels, which is more relevant to *caliberation* than quality of deep features as the title implies. And the paper misses this body of work in the related work. See here: https://nicholas.carlini.com/writing/2019/all-adversarial-example-papers.html - Try to avoid *extremely* or *clearly* in claims (and in general scientific writing) because they tend to be overclaims and it is hard for readers to interpret those intensifiers. - I don*t see how the rankings by DBI or AM are similar to the original rankings by adversarial measures at all (except for CapsNet). At least, reporting Pearson correlation (or any statistical similarity measure) would be more convincing. - Section 2 is highly debatable and has no references for the *Definition*. - I*d perform this work on ImageNet, at least to follow the intuition in Sec. 2. With only 10 CIFAR-10 classes, it is unrealistic how to semantically represent a *frog* using the other 9 classes (truck, airplane, etc).