Year: 2019

Abstract:
Neural networks have been shown vulnerable to adversarial samples. Slightly perturbed input images are able to change the classification of accurate models, showing that the representation learned is not as good as previously thought. To aid the development of better neural networks, it would be important to evaluate to what extent are current neural networks’ representations capturing the existing features. Here we propose a way to evaluate the representation quality of neural networks using a novel type of zero-shot test, entitled Raw Zero-Shot. The main idea lies in the fact that some features are present on unknown classes and that unknown classes can be defined as a combination of previous learned features. To evaluate the soft-labels of unknown classes, two metrics are proposed. One is based on clustering validation techniques (Davies-Bouldin Index) and the other is based on soft-label distance of a given correct soft-label. Experiments show that such metrics are in accordance with the robustness to adversarial attacks and might serve as a guidance to build better models as well as be used in loss functions to create new types of neural networks. Interestingly, the results suggests that dynamic routing networks such as CapsNet have better representation while current deeper DNNs are trading off representation quality for accuracy.

heading: 1 INTRODUCTION
text: Adversarial samples are slightly perturbed inputs that can make neural networks misclassify. They are carefully crafted by searching for variations in the input that, for example, could decrease the soft-labels of the correct class. Since they were discovered some years ago (Szegedy, 2014), the number of adversarial samples have grown in both number and types. Random noise was shown to be recognized with high confidence by neural networks (Nguyen et al., 2015), universal perturbations, that can be added to almost any image to generate an adversarial sample, were shown to exist (Moosavi-Dezfooli et al., 2017), and the addition of crafted patches was shown to cause networks to misclassify (Brown et al., 2017). Only one pixel is enough to make networks misclassify (Su et al., 2017). Such attacks can also be easily transferred to real-world scenarios (Kurakin et al., 2016),(Athalye & Sutskever, 2018), which confers a big issue as well as a security risk for current deep neural networks’ applications.
Albeit the existence of many defences, there is not any known learning algorithm or procedure that can defend against adversarial attacks consistently. Many works have tried to defend by hiding or modifying the gradients to make neural networks harder to attack. However, a recent paper shows that most of these defences fall into the class of obfuscated gradients which have their shortcomings (e.g., they can be easily bypassed by transferable attacks) (Athalye et al., 2018). Additionally, the use of an augmented dataset with adversarial samples (named adversarial training) is perhaps one of the most successful approaches to construct robust neural networks (Goodfellow et al., 2014),(Huang et al., 2015), (Madry et al., 2018). However, it is still vulnerable to attacks and has a strong bias in the type of adversarial samples used in training (Tramèr et al., 2018).
This shows that a deeper understanding of the issues is needed to enable more consistent defences to be created. Few works focused on understanding the reason behind such lack of robustness. In (Goodfellow et al., 2014), it is argued that Deep Neural Networks’s (DNN) linearity are one of the main reasons. Recent investigations reveal that attacks are changing where the algorithm is paying attention (Vargas & Su, 2019), other experiments show that deep learning neural networks learn false
structures that are easier to learn rather than the ones expected (Thesing et al., 2019) and an accuracy and robustness trade-off for models were shown to exist (Tsipras et al., 2019).
In this paper, we propose a methodology of how to evaluate the representation of machine learning methods. Based on these metrics, we reveal a link between deep representations’ quality and attack susceptibility. Specifically, we propose a test called Raw Zero-Shot and two metrics to evaluate DNN’s representations.

heading: 1.1 RECENT ADVANCES IN ATTACKS AND DEFENSES
text: DNNs were shown vulnerable to many types of attacks. For example, the output high confidence results to noise images (Nguyen et al., 2015), universal perturbations in which a single perturbation can be added to almost any input to create an adversarial sample are possible (Moosavi-Dezfooli et al., 2017), the addition of image patches can also make them misclassify (Brown et al., 2017). Moreover, the vulnerability can be exploited even with a single pixel, i.e., changing a single pixel is often enough to make a DNNs misclassify (Su et al., 2017). Most of these attacks can be transformed into real-world attacks by merely printing the adversarial samples (Kurakin et al., 2016). Moreover, crafted glasses (Sharif et al., 2016) or even general 3d adversarial objects (Athalye & Sutskever, 2018) can be used as attacks.
Although many defensive systems were proposed to tackle the current problems, there is still no consistent solution available. Defensive distillation in which a smaller neural network squeezes the content learned by the original DNN was proposed (Papernot et al., 2016). However, it was shown not to be robust enough (Carlini & Wagner, 2017). Adversarial training was also proposed as a defence, in which adversarial samples are used to augment the training dataset (Goodfellow et al., 2014),(Huang et al., 2015), (Madry et al., 2018). With adversarial training, DNNs increase slightly in robustness but not without a bias towards the adversarial samples used and while still being vulnerable to attacks in general (Tramèr et al., 2018). There are many recent variations of defenses in which the objective is to hide the gradients (obfuscated gradients) (Ma et al., 2018), (Guo et al., 2018) (Song et al., 2018). However, they can be bypassed by various types of attacks (such as attacks not using gradients, transfer of adversarial samples, etc.) (Athalye et al., 2018),(Uesato et al., 2018).
There are a couple of works which are trying to understand the reason behind such lack of robustness. To citep some, in (Goodfellow et al., 2014), it is argued that the main reason may lie in DNNs’ lack of non-linearity. Another work argues that the perturbations cause a change in the saliency of images which makes the model switch the attention to another part of it (Vargas & Su, 2019). False structures that are easier to learn were also shown related to the problem (Thesing et al., 2019). Moreover, in (Tsipras et al., 2019), the accuracy and robustness trade-off was shown to exist.

heading: 1.2 ZERO-SHOT LEARNING
text: Zero-Shot learning is a method used to estimate unknown classes which do not appear in the training data. The motivation of Zero-Shot learning is to transfer knowledge from training classes to unknown classes. Existing methods approach the problem by estimating unknown classes from an attribute vector defined manually. Attribute vectors are annotated to both known and unknown classes, and for each class, whether an attribute, such as “colour” and “shape”, belongs to the class or not is represented by 1 or 0. For example, in (Lampert et al., 2009) the authors proposed Direct Attribute Prediction (DAP) model, which learns each parameter for estimating the attributes from the target data. It estimates an unknown class of the source data which is estimated from the target data by using these parameters. Based on this research, other zero-shot learning methods have been proposed which uses an embedded representation generated using a natural language processing algorithm instead of a manually created attribute vector (Zhang & Saligrama, 2016; Fu et al., 2015; Norouzi et al., 2013; Akata et al., 2015; Bucher et al., 2016).
In (Zhang & Saligrama, 2015), a different approach to estimate unknown classes is proposed. This method constructs the histogram of known classes distribution for an unknown class. In this approach, it is assumed that the unknown classes are the same if these histograms generated in the target domain and the source domain are similar. This perspective is similar to our approach because our method approach to represent an unknown class as the distribution of known classes. However, our objective
is not estimating the unknown class, and we do not use the source domain. Our objective here is to analyze DNNs’ representation by using this distribution.

heading: 2 ON THE LINKS BETWEEN ROBUSTNESS EVALUATION AND REPRESENTATION
text: In the canonical classification setting, the goal of a classifier is to achieve low expected loss:
E (x,y)∼D [L(x, y; θ)] (1)
Robustness against adversarial attacks is a slightly different setting. To achieve a high robustness in this setting, a classifier should have a lower adversarial loss in noise δ ∈ ∆ 1:
E (x,y)∼D [L(x+ δ, y; θ)] . (2)
Considering Mean Squared Error (MSE), we have:
E (x,y)∼D [L(x+ δ, y; θ)] = E (x,y)∼D [(f(x+ δ)− h(x+ δ))2] + E (x,y)∼D [(h(x+ δ)− ŷ(x+ δ))2],
where h(x) = E[hD(x)] is the expected behavior of the prediction when averaged over many datasets, f(x) is the ground-truth and ŷ(x) = hD(x) is the output after learned on a given dataset D. For robustness to increase, adversarial training requires that datasets should have many noisy samples, i.e., x+δ ∈ D. However, the more noise is added to images the moreD becomes close to all possible images RM∗N :
lim ∆→∞
D = RM∗N (3)
However, f(x) is undefined 2 for D ∈ RM∗N in which y /∈ C for the set of known classes C. Even a small amount of noise may be enough to cause y /∈ C and thus f(x) undefined. Would it be possible to evaluate the robustness and/or the quality of a model without a well defined y?
To answer this question we take into account an ideal representation z and the representation learned by the model ẑ:
E[(f(x+ δ; z)− h(x+ δ; ẑ))2] + E[(h(x+ δ; ẑ)− ŷ(x+ δ; ẑ))2]
Interestingly, although y is undefined, z represents the features learned and is well defined for any input. Moreover, by considering learned classes to be clustered in z space, unsupervised learning evaluation can be used to evaluate z even without a well defined y. We use here a famous clustering analysis index to evaluate clusters in z by their intracluster distance. In z, it is also possible to evaluate the representation of known and unknown classes which should share some features. Moreover, we hypothesize here that unknown classes should evaluate z with less bias because a direct map from input to output is inexistent. Any projection of the input in any of the feature maps or the output layer could be used as z. To take the entire projection into account, we use here z as the final projection of the input to the classes, i.e. z is the soft label array e.

heading: 3 RAW ZERO-SHOT
text: In this paper, we propose to evaluate the representation learned by conducting experiments over the soft-labels of the image in unknown classes. This is based on the hypothesis that if a model is
1Here we use the error in noise to be the adversarial loss instead of the worst-case error, for a discussion of the relationship between error in noise and adversarial samples please refer to (Gilmer et al., 2019)
2Alternatively, f(x) could be defined for any noise if an additional unknown class is defined, such as with an OpenMax layer (Bendale & Boult, 2016).
capable of learning useful features, an unknown class would also trigger some of these features inside the model. We call this type of test over unknown classes and without any other information, Raw Zero-Shot (Figure 1).
The Raw Zero-Shot is a supervised learning test in which only n− 1 of the n classes are shown to the classifier during training. The classifier also has only n− 1 possible outputs. During testing, only unknown classes are presented to the classifier. The soft-labels outputted for the given unknown class is recorded, and the process is repeated for all possible n classes, removing a different class each time.
To evaluate the representation quality, metrics computed over the soft-labels are used. These metrics are based on a different hypothesis of what defines a feature or a class. In the same way that there are different types of robustness, there are also different types of representation quality. Therefore, metrics are somewhat complementary, each highlighting a different aspect of the whole. The following subsections define two of them.

heading: 3.1 DAVIES-BOULDIN METRIC - CLUSTERING HYPOTHESIS
text: Soft labels of a classifier compose a space in which a given image would be classified as a weighted vector concerning the previous classes learned. Considering that a cluster in this space would constitute a class, we can use clustering validation techniques to evaluate the representation (Figure 2).
Here we choose for simplicity one of the most used metric in internal cluster validation, DaviesBouldin Index (DBI). DBI is defined as follows:
DBI =  1 ne ne∑ j=1 |ej − cn|2 1/2 , (4)
in which cn is the centroid of the cluster, e is one soft-label and ne is the number of samples.

heading: 3.2 AMALGAM METRIC - AMALGAM HYPOTHESIS
text: If DNNs can learn the features present in the classes, it would be reasonable to consider that the sof-labels also describe a given image as a combination of the previously learned classes. This is also true when an image contains an unknown class. Similar to a vector space in linear algebra, the soft-labels can be combined to describe unknown objects in this space. This is analogous to how children describe previously unseen objects as a combination of previously seen objects. Differently from the previous metric, here we are interested in the exact values of the soft-labels. However, what would constitute the correct soft-labels for a given unknown class needs to be determined.
To calculate the correct soft-label of a given unknown class (amalgam proportion) automatically, we use here the assumption that accurate classifiers should output a good approximation of the amalgam proportion already. Therefore, if a classifier is trained in the n classes, the soft-labels of the remaining n− 1 classes is the amalgam proportion (Figure 2 illustrates the concept). Consequently, the Amalgam Metric (AM) is defined as:
h’i = ne∑ j=1 e’j ,hi = ne∑ j=1 ej
AM =
( 1
n n∑ i=1 ‖h’i − hi‖1 n− 1
) ,
(5)
in which, e’ is the normalized (such that they sum to one) soft-label from the classifier trained over n classes and e is the soft-labels from the classifier trained over n− 1 classes.

heading: 4 RAW ZERO-SHOT EXPERIMENTS
text: Here, we conduct Raw Zero-Shot experiments to evaluate the representation of DNNs. To obtain results over a wide range of architectures, we chose to evaluate CapsNet (a recently proposed completely different architecture based on dynamic routing and capsules) (Sabour et al., 2017), ResNet (a state-of-the-art architecture based on skip connections)(He et al., 2016), Network in Network (NIN) (an architecture which uses micro neural networks instead of linear filters) (Lin et al., 2013), All Convolutional Network (AllConv) (an architecture without max pooling and fully connected layers)(Springenberg et al., 2015) and LeNet (a simpler architecture which is also a historical mark) (LeCun et al., 1998). All the experiments are run over the CIFAR dataset by using a training dataset with all the samples of one specific class removed. This process is repeated for all classes, removing the samples of a different class each time.
To analyze the correlation between representation metrics and robustness against adversarial attacks, we conducted adversarial attacks on all the architectures tested using the most well known algorithms such as Carlini (Carlini & Wagner, 2017), Fast Gradient Method (FGM) (Goodfellow et al., 2014), Basic Iterative Method (BIM )(Kurakin et al., 2016), DeepFool (Moosavi-Dezfooli et al., 2016), Projected Gradient Descent Method (PGDM) (Madry et al., 2018) (Table 1).
For all tests, ε is fixed to the corresponding value given in the table. However, different methods have different meanings for ε: (a) for pixel attacks, ε is the maximum number of pixels to be changed, (b) for threshold attacks, ε is the maximum amount of change allowed per pixel, (c) for FGM, BIM and PGDM, ε is the attack step size (input variation) and (d) for DeepFool, ε is the overshoot parameter.
Table 2 shows the ranking based on the attack accuracy and their required perturbation (Table 1). In general, CapsNet is shown to be the most robust, AllConv and DenseNet follow with a proper
placement while the remaining networks vary depending on the perspective used to analyze (i.e., higher accuracy or lower L2). These robustness rankings will be used in the next sections to verify the relationship of robustness against adversarial samples and metrics to evaluate the representation quality.

heading: 4.1 EXPERIMENTS ON DBI METRIC
text: Table 3 shows the results with the DBI metric (the smallest the better) and the respective ranking of each neural network. According to this metric, CapsNet possesses the best representation of all networks tested. LeNet is considered the second-best neural network regarding representation, followed by AllConv, NIN and then deeper neural networks. DBI metric matches exceptionally well with both accuracy and L2 ranking based on robustness against adversarial samples. Most differences in Hamming distance lies in the exact places in which both accuracy and L2 rankings differ. Notice that DBI metric does not use anything related to attacks and still arrives at similar rankings. To further demonstrate the correlation between DBI and adversarial attacks, a Pearson correlation of the DBI for each network is shown in Table 4. This table suggests that DBI and adversarial attacks have a statistical significant correlation.
The fact that LeNet and other relatively more uncomplicated networks achieve a high representation quality which is at odds with accuracy may seem extremely unlikely. However, as discussed in (Tsipras et al., 2019), accurate models can trade-off robustness for accuracy. DBI suggests that this trade-off happens because the representation quality has worsened. Interestingly, LeNet and other
simple networks are also easier to attack (low-rank accuracy) but need more perturbation to achieve the same accuracy (high rank L2). Therefore, LeNet and other simple networks might be easier to attack because the search space is less complicated (less obfuscation (Athalye et al., 2018)). However, this does not mean they are less robust. Alternatively, as DBI suggests, LeNet and other simple networks might have achieved relatively good representations but without high accuracy.
To enable visualization of this metric, we plotted in Figure 3 a projection into two dimensions of all the points in the decision space of unknown classes. All this is done while preserving the high-dimensional distance between the points. Here we use the Isomap (Tenenbaum et al., 2000) to achieve this effect. It can be easily observed that CapsNet’s results for unknown classes are more clustered and thus form a better-defined cluster than other architectures.

heading: 4.2 EXPERIMENTS ON AMALGAM METRIC
text: In this section, the AM for all the networks is evaluated, which is based on the similarity of soft-labels for networks that were trained in all classes. The results shown in Table 3 reveal almost the same representation ranking as the robustness rankings related to L2. Interestingly, although both DBI and AM differ widely in concept and calculation procedure, the rankings are both similar and close to the L2. This further suggests that both metrics agree on what would be a good representation quality and can be used to evaluate representation in newer methods. A visualization as well as the Pearson correlation of the metric is included in the supplementary works.

heading: 4.3 PARTS OF A WHOLE
text: In (Agarwal et al., 2019), adding a loss to force features to be close to the feature centroid was shown to be beneficial against adversarial attacks. This is consistent with the proposed DBI metric, demonstrating that both soft-label space as well as other feature spaces benefit from projections to nearby positions. At the same time, we further support the existence of a trade-off between accuracy and robustness for deeper DNNs first point out in (Tsipras et al., 2019). Other types of networks, such as CapsNet seem to avoid it to some extent. Therefore, the trade-off is shown to vary with the architecture and computing dynamics of a model. Lastly, we hypothesize that a representation bias may hold in which features learned are only invariant to the dataset and not unseen classes.

heading: 5 CONCLUSIONS
text: Here we proposed the Raw Zero-Shot method to evaluate the representation of classifiers. In order to score the soft-labels, two metrics were formally defined based on a different hypothesis of representation quality. Results suggest that the evaluation of the representation of both metrics (DBI and AM) are linked with the robustness of neural networks. In other words, quickly attacked neural networks have a lower representation score. Interestingly, LeNet scores well in both metrics, albeit being the least accurate. LeNet is followed up by AllConv and NIN, which are less complicated/profound than other models which suggest that deeper architectures might be trading-off representation quality for accuracy. These results shown here further support the claim that there is a trade-off between accuracy and robustness in current deep learning (Tsipras et al., 2019).
Thus, the proposed Raw Zero-Shot was able to evaluate the representation quality of state-of-the-art DNNs and show their shortcomings concerning adversarial attacks, explaining many of the current problems. It also opens up new possibilities for both the evaluation (i.e. as a quality assessment) and the development (e.g., as a loss function) of neural networks.

heading: A THE POSSIBILITY OF A REPRESENTATION BIAS
text: The experiments show the possibility of representations to not work well for unknown classes. Many of these classes, however, share similar representations such as the dog and cat, truck and car. Thus, here we formulate a possible interpretation for the results.
The objective of a supervised learning algorithm is perhaps to map the input to output in such a way that the decision boundary reflects the real decision boundary. To achieve this, it is known that when dealing with complex problems, learning algorithms need first to learn a set of invariant features that are present throughout classes such that their recognition becomes robust against variations in the dataset.
Human beings, however, learn a set of invariant features that is not only able to solve current tasks or recognize current classes. We learn a set of features that can describe most if not all unseen classes and unknown tasks. Thus, we define representation bias as the bias towards invariant features that describe current seen classes or tasks but fail to describe unknown classes and tasks.

heading: B EXTENDED ANALYSIS OF DBI METRIC
text: Figure 4 shows a visualization of DBI’s results with t-Distributed Stochastic Neighbour Embedding (t-SNE) (Maaten & Hinton, 2008). DBI results are visualized using a projection into two dimensions while focusing on neighbour distances. The idea for having this visualisation is to investigate whether
the cluster can be split into one or more different classes. It is to be noted that IsoMap and t-SNE are two different visualisations for the same feature space.
The figure 4 shows that CapsNet forms pretty dense projections. Even if we form clusters to have different classes, the gaps between the classes will be too low relatively to other architectures. It also shows that in the high-dimensional space all the logits are moderately close to each other. While for the other architectures, there exists some points which can form their own cluster and be termed as different label. Hence for these architectures it can have one or more different classes for the same label which is contradicting to our hypothesis that there should exist only a single class for a single label. This inference can also be linked to adversarial attacks and supports the argument that CapsNet is relatively more robust than the other architectures.

heading: C EXTENDED ANALYSIS OF AMALGAM METRIC
text: To enable a visualisation of the metric, the computed histograms (h’i and hi from Equation 5) are plotted in Figure 5. It is interesting to note that in Figure 5, the histograms from CapsNet are different from the other ones, by the entirely different architecture employed by CapsNet. This reveals that this metric can capture such representation differences. Table 5 reveals that Amalgam metric is less reliable than DBI. Its p-values are also high enough to render its relationship less obvious. Recall, however, that the ranking based on Amalgam is very similar to the raking obtained with DBI metric.
D = | h’i − hi |, where
h’i = ne∑ j=1 e’j and hi = ne∑ j=1 ej (6)
The Amalgam Metric showed that both CapsNet and AllConv have the best scores which is in accordance with their top robustness score. Figure 6 shows a visualization of equation 6 which is part of the main equation of Amalgam Metric. On analysing further the phenomena about absolute difference between h’ and h in the Amalgam Metric or D in equation 6. It can be noted from the figure that for most labels of CapsNet and AllConv the difference is relatively low than the other architectures. This contributes to have CapsNet and AllConv to have best scores. Further investigations can be carried out to analyse the effect of a label in adversarial attack based on this figure. This can also provide insight on the labels which are robust to adversarial attacks. A further study can also be carried out to analyse the characteristics of the neural network’s representation which makes a label more robust than other labels.

Review summaries:
This paper aims at revealing the relationship between the quality of deep representations and the attack susceptibility of deep classification models. To this end, they propose the zero-shot test to investigate the "quality" of learned representations for unknown classes. Specifically, they leverage two kinds of quality metrics on data of unknown classes. The first one is based on clustering named Davies-Bouldin Index which measures the compactness of intra-cluster. The second one is based on the difference of soft-label histogram distributions with/without unknown classes during training, which may describe the generalization for unknown classes or bias towards known classes of learned features. Finally, with these two metrics, they rank the quality of different models and compare such ranking results with the attack robustness obtained by different attack techniques on CIFAR-10 dataset.

This paper proposes to evaluate the robustness of the neural networks by extrapolating to the unseen classes. However, the authors only include evaluation for non-robust trained models, without considering the robust trained model, such as Madry et al. [1]. The conclusion is not convincing that the authors studied the robustness using only non-robust models, because it is well known that the accuracy for attacking non-robust model can be 100% (for CIFAR-10). It is useful if the authors study whether their method can be used to measure the robustness of the robust trained models.