Title:  A Theoretical Framework for Robustness of (Deep) Classifiers against Adversarial Samples
Authors: Beilun Wang
Ji Gao
Yanjun Qi
Venue: ICLR 2017
Keywords: Deep learning

Abstract: Most machine learning classifiers, including deep neural networks, are vulnerable to adversarial examples. Such inputs are typically generated by adding small but purposeful modifications that lead to incorrect outputs while imperceptible to human eyes. The goal of this paper is not to introduce a single method, but to make theoretical steps toward fully understanding adversarial examples. By using concepts from topology, our theoretical analysis brings forth the key reasons why an adversarial example can fool a classifier ($f_1$) and adds its oracle ($f_2$, like human eyes) in such analysis. 
By investigating the topological relationship between two (pseudo)metric spaces corresponding to predictor $f_1$ and oracle $f_2$, we develop necessary and sufficient conditions that can determine if $f_1$ is always robust (strong-robust) against adversarial examples according to $f_2$. Interestingly our theorems indicate that just one unnecessary feature can make $f_1$ not strong-robust, and the right feature representation learning is the key to getting a classifier that is both accurate and strong robust.


TL;DR: We propose a theoretical framework to explain and measure model robustness and harden DNN model against adversarial attacks.

Review discussion:
The theory provides some interesting perspectives for analyzing the adversarial phenomenon. The important thing is to link the theory with novel concrete algorithms.
I have following questions for bettering understanding the paper:
1.	In Corollary 3.4 and Section 3.6, what are features and what are classifiers?
2.	A similar question is that, in Section 4.2, what do middle layer outputs specifically refer to as? Is it for the penultimate layer or many intermediate layers?
3.	In Definition 4.1, f2(x) = f2(t) is finally determined by the inf norm.
Then, what is the actual difference between the proposed ARC with existing methods?
4.	In Equation (4.6), is L1 cross-entropy loss for the softmax classifier or is it also a Euclidean-based Siamese loss?
Similarly, in Figure 5, how is the “original method” trained? With softmax classifier?
5.	The Siamese training basically try to match features, what is the essential different with this paper ( https://arxiv.org/abs/1511.06381 )?
Answer: Thank you for pointing out this writing issue.
- We have added two paragraphs at the beginning of Section-3. We have copied those contents in our answers to Q1 from AnonReviewer2 (therefore not copying here);
- We have revised the whole subsection 4.2 and Figure-5 accordingly.
- We have revised the third paragraph in Section 3.6 accordingly. The contents are copied as follows:
•  f1(·): f1(·) is a DNN classifier with multiple layers, including linear perceptron layers, activation layers, convolutional layers and softmax decision layer.
• (X1 , d1 ): X1 denotes the feature space discovered by the layer right before the last fully connected layer. This feature space is automatically extracted from the original image space (e.g., RGB representation) by the DNN. (X, d′1) is defined by d1 using Eq. (3.3).
• (X2 , d2 ): X2 denotes the feature space that oracle (e.g., human annotators) used to decide ground- truth labels of training images.
For example, a human annotator needs to recognize a hand-written digit “0”. X2 includes what patterns he/she needs for such a decision. (X, d′2) is defined by d2 using Eq. (3.4)
Q3. In Definition 4.1, f2(x) = f2(t) is finally determined by the inf norm.
Then, what is the actual difference between the proposed ARC with existing methods?
Answer: Thank you for raising this concern. We have added two paragraphs right after Definition 4.1 to explain the connections. They are copied as follows:
Two recent studies (Moosavi-Dezfooli et al., 2015; Papernot et al., 2015b) propose two similar measures both assuming d2 as norm functions. They do not consider the importance of oracle. More importantly, (Papernot et al., 2015b) does not provide any computable way to calculate the measure. In (Moosavi-Dezfooli et al., 2015), the measure is normalized by the size of the test samples, while no evidence exists to show that the size of perturbation is related to the size of test samples.
The fact that previous measures neglect oracle f2 leads to a severe problem: the generated adversarial samples are not necessarily valid. This is because if the size of perturbation is too large, oracle f2 may classify the perturbed sample into a different class (different from the class of the seed sample).
Q4. In Equation (4.6), is L1 cross-entropy loss for the softmax classifier or is it also a Euclidean-based Siamese loss?
Similarly, in Figure 5, how is the “original method” trained? With softmax classifier?
Answer: Thank you for inquiring these missed experimental details.
- L1 is cross-entropy loss for the softmax classifier
- The original method is trained by optimizing cross-entropy loss for the softmax classifier
Q5. The Siamese training basically try to match features, what is the essential different with this paper ( https://arxiv.org/abs/1511.06381 )?
Answer: We want to sincerely thank reviewer for pointing out this missed reference. This has helped us completely rewrite the whole subsection 4.2 and Figure-5 accordingly.
- Siamese training is closely related to paper ( https://arxiv.org/abs/1511.06381 ). Differently we use randomly perturbed sample as $x’$, while this paper use adversarially perturbed sample as $x’$, though both trying to match the middle layer outputs.
- We have removed the novelty claim of Siamese training (across the whole paper)
- More importantly, our revised writing focuses more on connecting Siamese training to the theorems we have proposed. This is exactly the reason why we put Siamese training in the main content: to provide an empirical study showing how these theorems can be useful for understanding and analyzing a specific learning strategy. We argue that the reason why Siamese training can improve weak-robustness is because it intuitively helps a DNN classifier near topological equivalence with its oracle (see revised Figure-5).
The robustness definition is not self-contained
- How does X1 and X2 relate to f1 and f2? It can be guessed based on the previous paragraph, but it is not specified formally in the definition.
In Theorem 3.7 what do d_1 and d_2 refer to?
In Figure 3, it is suggested: "This figure shows one situation that (X1, d1) and (X2, d2) are not topologically equivalent.
Therefore, in this case, the DNN is vulnerable to adversarial attacks." This does not look like a valid implication.
The purely topological conclusions of Section 3 seem to be very disconnected to the quantitative measurements of Section 4. A more thorough exposure on the connections between the two sections would be helpful.
Q1. The robustness definition is not self-contained - How does X1 and X2 relate to f1 and f2? It can be guessed based on the previous paragraph, but it is not specified formally in the definition. In Theorem 3.7 what do d_1 and d_2 refer to?
Answer: Thank you for pointing out this writing issue. We have added two paragraphs at the beginning of Section-3. They are copied as follows:
As shown in Figure~1 we assume a common machine learning classifier $f_1 = g_1 \circ c_1 $, where $g_1 : X \to X_1$ represents the feature extraction and $c_1: X_1 \to Y$ performs the operation of classification. In Section-2, we decompose $f_2$ as $f_2 = g_2 \circ c_2$ where $g_2 : X \to X_2$ represents the feature extraction and $c_2: X_2 \to Y$ performs the operation of classification. Here, $X_2$ describes the feature space in which oracle makes the decision.
The reason why we need the above decomposition of $f_1$ and $f_2$ is to introduce two metric spaces defined in $X_1$ and $X_2$. In this section, we theoretically prove a special relationship between two metric spaces is a sufficient condition in determining whether $f_1$ is strong-robust at test time, or not.
Then we extend the discussion from metric spaces into two pseudometric spaces defined in $X$ and prove that a special relationship between two pseudometric spaces is sufficient and necessary in determining the strong-robustness of $f_1$.
The second paragraph of Section 3.1 also helps in understanding this decomposition:
As shown in Figure1, the two metric spaces in our formulation are $(X_1,d_1)$ and $(X_2,d_2)$. For a pair of samples $(x,x')$ ($x,x' \in X$), the feature extraction of $f_1$ and $f_2$ transfer $(x,x')$ to $(x_1, x_1')$ ($x_1,x_1' \in X_1$) and $(x_2, x_2')$ ($x_2,x_2' \in X_2$).
Then the adversarial problem defined by Definition 2.2 is to find a pair of samples $(x,x')$ satisfying that $d_2(x_2,x_2')$ is small while $d_1(x_1,x_1')$ is large.
Q2. In Figure 3, it is suggested: "This figure shows one situation that (X1, d1) and (X2, d2) are not topologically equivalent.
Therefore, in this case, the DNN is vulnerable to adversarial attacks." This does not look like a valid implication.
Answer: Thank you for pointing out this error.
- We have revised Figure-3 into new Figure-3 in the updated draft.
- The correct way to explain the figure should be: This figure shows one situation that (X, d’_1) is not a finer topology of (X, d’_2). According to Theorem 3.7, in this case, the DNN is vulnerable to adversarial attacks at test time.
- Theorem 3.7 in Section 3.4 proves that:
A machine-learning classifier $f_1$ is strong-robust against adversarial test samples if and only if $(X,d_1')$ is a finer topology than $(X,d_2')$.
Q3. The purely topological conclusions of Section 3 seem to be very disconnected to the quantitative measurements of Section 4. A more thorough exposure on the connections between the two sections would be helpful.
Answer:  We want to sincerely thank reviewer for point out that we missed such an important discussion. We have added three paragraphs at the beginning of Section-4. They are copied as follows:
Section 3's analysis indicates that strong-robustness is a strong condition of machine learning classifiers and requires thorough understanding of oracle. Since many state-of-the-art learning models, including DNNs, are not strong-robust, it is important to understand and quantify how far they are away from strong-robust. We name such situations as "weak-robustness" and propose a quantitative measure to describe how robust a classification model is against adversarial attacks. The proposed measure is named as Adversarial Robustness for Classifiers (ARC) and uses the expectation of how difficult it is to generate adversarial samples on a classifier.
Comparing to relevant studies in the literature, our analysis of weak-robustness has the following contributions:
• (1) We propose a measure ARC to quantify a classifier’ weak-robustness by considering both the predictor f1 and the oracle f2 (introduced in Section 2). Measures proposed by previous studies (Papernot et al., 2015b; Moosavi-Dezfooli et al., 2015) simply calculate the average of model accuracy under adversarial samples, which ignores the importance of oracle.
• (2) Similar to the definition of strong-robustness, we connect the notion of weak-robustness to Definition (2.2) (the general definition of Adversarial test sample for a classification task).
• (3) We further prove that a classifier f1 is strong-robust against adversarial samples if and only if its ARC achieves the maximum (1 since ARC is rescaled to [0, 1]). This clearly shows that the weak-robustness is quantitatively related to the strong-robustness.
This paper theoretically analyzes the adversarial phenomenon by modeling the topological relationship between the feature space of the trained and the oracle discriminate function.
In particular, the (complicated) discriminant function (f) is decomposed into a feature extractor (g) and a classifier (c), where the feature extractor (g) defines the feature space.
The main contribution of this paper is to propose abstract understanding and analysis for adversarial phenomenon, which is interesting and important.
However, this paper also has the following problems.
1)	It is not clear how the classifier c can affect the overall robustness to adversarial noises. The classifier c seems absent from the analysis, which somehow indicates that the classifier does not matter. (Please correct me if it is not true) This is counter-intuitive.
For example, if we always take the input space as the feature space and the entire f as the classifier c, the strong robustness can always hold.
I am also wondering if the metric d has anything to do with the classifier c.
2)	A very relevant problem is how to decompose f into g and c. For examples, one can take any intermediate layer or the input space as the feature space for a neural network. Will this affect the analysis of the adversarial robustness?
3)	The oracle is a good concept.
However, it is hard to explicitly define it. In this paper, the feature space of the oracle is just the input image space, and the inf-norm is used as the metric. This implementation makes the algorithm in Section 4 quite similar to existing methods (though there are some detailed differences as mentioned in the discussion).
Due to the above problems, I feel that some aspects of the paper are not ready. If the problems are resolved or better clarified, I believe a higher rating can be assigned to this paper.
In addition, the main text of this paper is somehow too long, the arguments can be more focused if the main paper become more concise.
Thank you for raising this important question. We added two figures in the appendix to illustrate classification c function does not directly influence robustness.
1. In Figure 5, we show one case of $n_1 = n_2 = 2, X_1 = X_2 = \RR^2$.
In terms of classification, $f_1$ (green boundary line) is not accurate according to $f_2$ (red boundary line).
However, $f_1$ is strong-robust to the adversarial samples.
2. In Figure 6, we show one case of $1 = n_1 < n_2 = 2, X_1 \subset X_2$. Similar to Figure 5, in terms of classification, $f_1$ (green boundary line) is not accurate according to $f_2$ (red boundary line).
However, $f_1$ is strong-robust to the adversarial samples.
3. For both figures Test-sample Case (c), we want to point out for the case of $X$ being an infinite space, $\P((x,x') | d_2(x,x') <\epsilon \& \text{$(x,x')$ are across the boundary of $f_1$}) = 0$. The adversarial sample problem this paper focuses on is different from ``boundary based adversarial attacks''. ``Boundary based adversarial attacks'' can only attack the points whose distance to the boundary of $f_1$ is smaller than $\epsilon$.
In fact, based on probability theory, the probability of this set is $0$ if $X$ is infinite.
Thanks for pointing this out. We have added the following paragraph in Section 7.2.
It is difficult to decompose an arbitrary $f_1$ into $g_1 \circ c_1$.
However, since in our context, $f_1$ is a machine learning classifier, we can enumerate many possible $g_1$ functions to cover classic machine learning classifiers.
1. Various feature selection methods are potential $g_1$.
2. For DNN, $g_1$ includes all the layers from input layer to the layer before the classification layer.
3. In SVM, $X_1,d_1$ is decided by the chosen reproducing Hilbert kernel space.
4. Regularization is another popular implicit feature extraction method.
For example, $\ell_1$ regularization can automatically do the feature extraction by pushing some parameters to be $0$.
Dear Reviewer:
Thank you for pointing out this important issue.
Even though being difficult, we want to argue that it is possible to theoretically model "oracles" for some state-of-the-art applications.
For instance, illustrated by the seminal cognitive neuroscience paper "untangling invariant object recognition" ~\cite{dicarlo2007untangling} and its follow-up study \cite{dicarlo2012does}, the authors show that one can view the information processing of visual object recognition by  human brains as the process of finding operations that progressively transform retinal representations into a new form of representation ($X_2$ in this paper), followed by the application of relatively simple decision functions (e.g., linear classifiers~\cite{duda2012pattern}).
More specifically, in human and other primates, such visual recognition takes place along the ventral visual stream and this stream is considered to be a progressive series of visual re-representations, from V1 to V2 to V4 to IT cortex ~\cite{dicarlo2007untangling}.
Multiple relevant studies (e.g., ~\cite{dicarlo2007untangling,johnson1980sensory,hung2005fast}) have argued that this viewpoint of representation learning plus simple decision is more productive than hypothesizing that brains directly learn very complex decision functions (highly non-linear) that operate on the retinal image representation.
This is because the experimental evidence suggests that this view takes the problem apart in a way that is consistent with the architecture and response properties of the ventral visual stream.
Besides simple decision functions can be easily implemented in a single step of biologically plausible neuronal processing (i.e., a thresholded sum over weighted synapses).
Dear Reviewer:
Here is another example in which modeling oracle is possible.
As another example, the authors of ~\cite{xu_automatically_2016} used genetic programming to find "adversarial samples (by solving ~\eref{eq:multi-class}) for a learning-based malicious-PDF classifier. This search needs an oracle to determine if a variant $x'$ preserves the malicious behavior of a seed PDF $x$ (i.e., $f_2(x) = f_2(x')$).
The authors of \cite{xu_automatically_2016} therefore used the Cuckoo sandbox (a malware analysis system through actual execution) to run a variant PDF sample in a virtual machine installed with a PDF reader and reports the behavior of the sample including network APIs calls. By comparing the behavioral signature of the original PDF malware and the manipulated variant, this oracle successfully determines if the malicious behavior is preserved from $x$ to $x'$.
One may argue that "since Cuckoo sandbox works well for PDF-malware identification, why a machine-learning based detection system is even necessary?". This is because Cuckoo sandbox is computationally expensive and runs slow. For many machine-centered security-sensitive applications, oracles do exist, but machine-learning classifiers are used popularly due to speed or efficiency.
~\cite{xu_automatically_2016}: Weilin Xu, Yanjun Qi, and David Evans. Automatically evading classifiers. In Proceedings of the Network and Distributed Systems Symposium, 2016.
This paper aims at making three contributions:
- Charecterizing robustness to adversarials in a topological manner.
- Connecting the topological characterization to more quantitative measurements and evaluating deep networks.
- Using Siamese network training to create models robust to adversarial in a practical manner and evaluate their properties.
In my opinion the paper would improve greatly if the first, topological analysis attempt would be removed from the paper altogether.
A central notion of the paper is the abstract characterization of robustness. The main weakness is the notion of strong robustness itself, which is an extremely rigid notion. It requires the partitioning of the predictor function by class to match the exact partitioning of the oracle. This robustness is almost never the case in real life: it requires that the predictor is almost perfect.
The main flaw however is that the output space is assumed to have discrete topology and continuity is assumed for the classifier. Continuity of the classifier wrt. a discrete output is also never really satisfied.
However, if the output space is assumed to be continues values with an interesting topology (like probabilities), then the notion of strong robustness becomes so constrained and strict, that it has even less practical sense and relevance. Based on those definition, several uninteresting, trivial consequences follow. They seem to be true, with inelegant proofs, but that matters little as they seem irrelevant for any practical purposes.
The second part is a well executed experiment by training a Siamese architecture with an explicit additional robustness constraint. The approach seems to be working very well, but is compared only to a baseline (stability training) which performs worse than the original model without any trainings for adversarials. This is strange as adversarial examples have been studied extensively in the past year and several methods claimed improvements over the original model not trained for robustness.
The experimental section and approach would look interesting, if it were compared with a stronger baseline, however the empty theoretical definitions and analysis attempts make the paper in its current form unappealing.
Q1: The main weakness is the notion of strong robustness itself, which is an extremely rigid notion. It requires the partitioning of the predictor function by class to match the exact partitioning of the oracle. This robustness is almost never the case in real life: it requires that the predictor is almost perfect.
Answer: We want to point out that this comment indicates some incorrect understanding of our paper:
1. Strong-robustness “does NOT require the partitioning of the predictor function by class to match the exact partitioning of the oracle”.
2. A robust classifier does not need to be perfect. A trivial example for strong-robust models is $f_1(x) = 1, \forall x \in X$.
However, it is not a perfect predictor at all.
3. Our theoretical analysis provides a much better understanding of the relationship between robustness and accuracy. Two typical examples can be derived from Table-3:
- If f_1 misses discovering some related features and does not extract unrelated features, f_1 is strong-robust (even tough its accuracy may be not satisfactory).
- If f_1 uses some extra and unrelated features, it will not be strong-robust to adversarial attacks (though it may be a very accurate predictor).
We are updating the paper to explain the theoretical conclusions using a more accessible language.
4. In the adversarial setting, we should aim to get a classifier that is both strong-robust and accurate. A better feature learning is exactly the solution that may improve both goals!
Q2:  The main flaw however is that the output space is assumed to have discrete topology and continuity is assumed for the classifier. Continuity of the classifier wrt. a discrete output is also never really satisfied. … , several uninteresting, trivial consequences follow. … matters little as they seem irrelevant for any practical purposes.
Answer: We want to point out that this comment indicates some incorrect understanding of our paper:
1. Our topology is not about the output space. They are about the input space and the extract features. More specifically, the equations are as follows:
-	$d_1'(x, x') = d_1(g_1(x),g_1(x'))$
-	$d_2'(x, x') = d_2(g_2(x),g_2(x'))$
2. We do NOT assume continuity for the classifier. The assumption we made on f_1 is “almost everywhere (a.e.) continuity”. (Explained in Section 2.3.1 and Appendix Section 6.5 of the 2016-Dec-14 version).
3. Lemma (6.3) shows that if a classifer f_1 is not continuous almost everywhere (a.e.), f_1 is not robust (even) against random noise.
4. Popular machine learning classifiers all satisfy the almost everywhere (a.e.) continuity assumption. Explanation of DNN is a.e. continuous is included in Section 2.3.1.  Explanations of logistic regression and SVM being continuous almost everywhere are included in Section 6.6. (of the 2016-Dec-14 version).
5. The a.e. Continuity essentially indicates,
- It does not mean that the function $f_1$ is continuous in every point in its feature space X;
- As we all know (https://en.wikipedia.org/wiki/Probability_density_function): If a probability distribution admits a density, then the probability of every one-point set {a} is zero; the same holds for finite and countable sets. The same conclusion holds for zero measure sets (Null_set: https://en.wikipedia.org/wiki/Null_set ), for instance, straight lines  in  $R^n$.
- The a.e. Continuity follows the same property as density function: the probability of picking one-point set {x} from the whole feature space is zero; the same holds for zero measure sets. This means: the probability of picking the discontinuous points (e.g., points on the decision boundary) is zero, because they are null_set.
We have updated the paper with the more detailed explanations about a.e. continuity and its indications,
6. As we have answered in the third point of Q1- AnonReviewer2, our theoretical analysis provides a much better understanding of the relationship between robustness and accuracy. A very interesting finding from our analysis is about the importance of feature learning. In the adversarial setting, we should aim to get a classifier that is both strong-robust and accurate. A better feature learning is exactly the solution that may improve both goals!
I was aware that there are construct robust, but incorrect functions.
However, I find it extremely rigid that if the predictor predicts label l correctly for an instance, then it needs to predict l correctly for each instances that have ground-truth l. This is still a very strong requirement, unlikely to be useful.
1.
However a.e. continuity and the conclusions and theorems of the paper depend on the topology of the output space.
2., 4. a.e. continuity is fine for most classifier, still it makes the robustness property a very strong one.
3. Strong robustness is still an overly strong notion (that restricts the space of functions to which it applies to a very limited subset)
5. This was clear.
6. I still can't see how one could ever generate a strong-robust (and accurate) classifier in practice: whenever a label is predicted correctly for one element of the class, it is required to be correct for each instances of that class. This is an extremely strong correctness assumption.
1.	One of the most important insights from our theorem is “ Generating a strong-robust (and accurate) classifier in practice is extremely difficult.” A very simple illustration is given in Figure 2. It clearly shows that just one extra irrelevant feature, which does not hurt accuracy, totally makes classifier not robust to adversarial noise at all (For sample point a.e. in the sample space X, it is easy to find its adversarial samples.).
2.	We want to argue that the probability of the adversarial samples phenomena happening is either 1 or 0. The probability 1 case has been clearly illustrated in Figure 2. The probability 0 case is all the cases satisfying strong-robustness.
3.      It is worth to point out this conclusion assumes $X$ is not a finite space. If $X$ is finite, the ratio of cross-boundary point pairs (with the pairwise distance smaller than \epsilon) is not 0, which makes the point 2 doesn't hold. We want to argue that most problems that machine learning classifiers are useful for have/assume infinite $X$ space.
4.	Based on this comments we find an error in our old Equation (3.1) (version Dec 14).
Therefore we have reformulated it into a new probabilistic form:
\forall x,x' \in X
\P(f_1(x) = f_1(x') | d_2(x,x') < \epsilon) = 1
Please see our following response with title One of the most important insights from our theorem is “ Generating a strong-robust (and accurate) classifier in practice is extremely difficult.”
I don't think that you need any theorem or deep considerations to see that strong robustness (combined even with partial correctness) is an overly restrictive notion.
As mentioned: if you take any predictor that predicts one single instance correctly for each output label, then strong robustness implies perfect accuracy of the predictor.
Given that assuming one single correctly classified instance per class is a reasonable assumption for a classifier, it makes it clear that strong robustness is a notion that implies f_1 = f_2 and therefore it is a way too rigid notion to be useful.
Dear reviewer,
Thank you for your prompt responses. We sincerely appreciate all the comments that have helped us improve the paper.
However, we want to point out that the comment " strong robustness implies perfect accuracy of the predictor" is incorrect. This is closely related to AnonReviewer3 question1.
Essentially, this is another important insight from our theorem -- $g_1$ matters for the strong-robustness and $c_1$ does not. The related discussion is included in Section 3.5.4 and Section 7.3 in the latest version.
Your concerns: "if you take any predictor that predicts one single instance correctly for each output label, then strong robustness implies perfect accuracy of the predictor" have been explained by Figure 5 and Figure 6. In both figures $f_1$ is strong-robust according to $f_2$ and $f_1$ predicts one single instance correctly for each output label.
However, $f_1$ is not accurate according to $f_2$.
More specifically, Figure 5 is about $n_1 = n_2 = 2, X_1 = X_2 = \RR^2$ and Figure 6 is about $1 = n_1 < n_2 = 2, X_1 \subset X_2$.
Our definition of strong-robustness is $\P(f_1(x) = f_1(x') | d_2(x,x') < \epsilon) = 1, \forall x,x' \in X$.
All pairs of test samples $(x,x')$ can be categorized into the three cases as follows:
Test-case (a) in both figures is about when $x$ and $x'$ are predicted as the same class by both and $f_1$ gets correct predictions according to $f_2$. There exist no adversarial samples.
Test-case (b) in both figures is about when $x$ and $x'$ are predicted as the same class by both and $f_1$ gets incorrect predictions according to $f_2$. There exist no adversarial samples.
Test-case (c) in both figures shows the case of $f_1(x) \ne f_1(x') | d_2(x,x') < \epsilon$. This case has been explained in Section 3.5.4. Essentially, this is about ``Boundary based adversarial attacks'' not adversarial sample problem we focus in this paper. This type of attacks can only attack the points whose distance to the boundary of $f_1$ is smaller than $\epsilon$.
In fact, based on probability theory, the probability of this set is $0$ if $X$ is assumed infinite.
Under my point of view, the robustness of a classifier against adversarial noise it is interesting if we find any relationship between that robustness and generalization to new unseen test samples. I guess that this relationship is direct in most of the problems but perhaps classifier C1 could be more robust than C2 against adv. noise but not better for new unseen samples from the task in consideration. Best results on new unseen samples are normally related to robustness against the common distortions of the data, e.g. invariance to scale, rotation… than robustness to adv. noise.
I can not see any direct conclusion from table 5 results.
Essentially i am not convinced about the necessity to measure the robustness against adversarial noise.
Q1: …. the robustness of a classifier against adversarial noise is interesting if we find any relationship between that robustness and generalization to new unseen test samples. I guess that this relationship is direct in most of the problems …
Answer:
If we understand this question correctly, we think this question is about the relationship between adversarial robustness and test accuracy.
Our theoretical analysis in Table 3 provides four cases to understand the relationship between robustness and accuracy. Two typical examples can be derived from Table 3:
- If $f_1$ misses discovering some related features and does not extract unrelated features, $f_1$ is strong-robust (even tough its accuracy may be not satisfactory).
This is illustrated by a simple example in Figure 6 showing 1 = n_1 < n_2 = 2, X_1 \subset X_2.
In terms of classification, $f_1$ (green boundary line) is not accurate according to $f_2$ (red boundary line).
However, $f_1$ is strong-robust to the adversarial samples. For Test-sample Case (c) see the discussions of boundary points in Section 3.5.4.
- If $f_1$ uses some extra and unrelated features, it will not be strong-robust to adversarial attacks (though it may be an accurate predictor).
This is illustrated by a simple example in Figure 2.
In the adversarial setting, we should aim to get a classifier that is both strong-robust and accurate. A better feature learning is exactly the solution that may improve both goals.
This question partially relates to the discussion of "how g function and c function influence the adverial robustness". For concrete discussion see Section 3.5.3 -- "g_1 MATTERS FOR THE STRONG - ROBUSTNESS AND c_1 DOES NOT".
Q2: Incorrect comment: “... classifier C1 could be more robust than C2 against adv. noise but not better for new unseen samples from the task in consideration. ”
Answer:
Dear reviewer,
1). Do you mean "classifier C1" the f_1 or its classification step c_1 in our paper?
As shown in Figure 1, we assume a common machine learning classifier $f_1 = c_1 \circ g_1 $, where $g_1 : X \to X_1$ represents the feature extraction and function $c_1: X_1 \to Y$ performs the operation of classification.
Similarly, we decompose an oracle $f_2$ as $f_2 = c_2 \circ g_2$ where $g_2 : X \to X_2$ represents the feature extraction and $c_2: X_2 \to Y$ performs the operation of classification. Here, $X_2$ describes the feature space of the oracle.
2). In our formulation, it does not make sense to compare c_1 and c_2 in terms of their strong-robustness. Because it only makes sense to say f_1 is strong-robust against adversarial samples according to f_2.
Q3: "Essentially i am not convinced about the necessity to measure the robustness against adversarial noise."
Answer:
Dear reviewer,
We are not totally clear about this question. Do you mind providing more details?
If this question is about importance of machine learning under adversarial noise, please check out the following references:
1. Intriguing properties of neural networks
C Szegedy, W Zaremba, I Sutskever, J Bruna, D Erhan, I Goodfellow, ...
arXiv preprint arXiv:1312.6199	 citation: 286  year: 2013
https://scholar.google.com/citations?view_op=view_citation&hl=en&user=iYN86KEAAAAJ&citation_for_view=iYN86KEAAAAJ:MXK_kJrjxJIC
2. Explaining and harnessing adversarial examples
IJ Goodfellow, J Shlens, C Szegedy
arXiv preprint arXiv:1412.6572	 citation: 148	year: 2014
https://scholar.google.com/citations?view_op=view_citation&hl=en&user=iYN86KEAAAAJ&citation_for_view=iYN86KEAAAAJ:_kc_bZDykSQC
3.Distributional Smoothing with Virtual Adversarial Training
Takeru Miyato, Shin-ichi Maeda, Masanori Koyama, Ken Nakae, Shin Ishii
iclr 2016  https://arxiv.org/abs/1507.00677
Dear Reviewers:
With your valuable comments, we realize that the structure of the paper is not clear enough.
Therefore, we are making several major revisions on the paper right now. A revised version with major revision has been submitted.
Dear Reviewers:
We have updated the paper with a major revision. The current version is listed as “14 Jan 2017ICLR 2017 conference paper485 Add Revisionreaders: everyone” in the system.
Due to many parts of the paper have been changed largely, we sincerely request reviewers to re-read the current draft.
The revision includes:
- A revised abstract and introduction that capture the essence of the paper much better;
- Revising many math notations to make all equations more consistent and coherent;
- Revising and adding many section/subsection/subsubsection headers;
- Reordering many parts of the manuscript to get a better logical flow.
For instance,
+ we move the discussion of modeling and decomposing f_1 to Section 2.2
+ we reorder all contents in  “Section 3.4 why theorems are important” to get more rational discussions about theoretical  insights obtained from the proposed theorems, e.g,  moving two corollaries into this section;
+ we reorder all contents in Section 4.2 about using Siamese architecture to motivate this strategy from a finer topology angle.
- Our older draft (versions before Jan5th) only covered theoretical analyses about when f_1 and f_2 are both continuous a.e.. In this major revision,
+ We add discussions of boundary points, boundary-based adversarial attacks, and  discussion/figures connecting  f_i not continuous a.e. to boundary points of f_i;
+ We add a more general definition of adversarial sample (eq 2.3) in Section 2.5 and a more general definition of strong-robustness (eq 3.1) in Section 3.1.
+ We revise the definition of “strong-robustness” into a claim with a high probability;
+ We add two new theorems (Theorem 3.3 and Theorem 3.5) covering cases when f_1 is not continuous a.e.. For each theorem, we also consider two cases (one for f_2 being continuous a.e. and the other for f_2 being not continuous a.e.) and add the corresponding proofs of each case.
+ We add more discussions of modeling the oracle f_2 in Section 2.3 and Section 6.3.
+ We add Figure 6 for illustrating three types of boundary points.
+ We add Figure 9 for illustrating a case when f_1 is not continuous a.e.
+ We add section 3.5 to connect our theoretical results to studies in the literature, and to provide a list of possible hardening solutions for improving a classifier's adversarial robustness.
- We have moved many parts of the older main draft into the appendix.
+ we move detailed definitions of metric space, pseudometric space, topological equivalence, finer topology to the Appendix: Section 7
+ We move the whole section of ARC to the Appendix: Section 8.2
+ We move the summary of previous algorithms generating “adversarial samples” to the Appendix: Section 6.2
+ We also move the details of experiments/experiment settings to Appendix: Section 8.1
Thank you !
Dear Reviewers:
We add two paragraphs in Section 8.1 about details of the baseline "stability training" from (Zheng et al., 2016)
- Stephan Zheng, Yang Song, Thomas Leung, and Ian Goodfellow. Improving the robustness of deep neural networks via stability training. arXiv preprint arXiv:1604.04326, 2016.
Reasons why we pick this baseline:
- A modified distillation strategy (Papernot et al., 2015b) was proposed to improve the robustness of DNNs against adversarial samples, though it has been shown to be unsuccessful recently (Carlini & Wagner, 2016a).
- Multiple techniques incorporate a smoothness penalty (Miyato et al., 2016; Zheng et al., 2016) or a layer-wise penalty (Carlini & Wagner, 2016b) as a regularization term in the loss function to promote the smoothness of the DNN model distributions.  Among them, Stability training is the closest to the "Siamese training".
Again, the main purpose of covering Siamese training in this paper is to provide an empirical study showing how our theorems can be useful for analyzing a specific hardening strategy. We do not intend to claim it as a novel hardening strategy.
ATT: Stability training was shown to improve the model robustness against Gaussian noise in (Zheng et al., 2016). Differently, our experiments focus on testing a learning model’s robustness against "adversarial noise". The sole purpose of including this baseline is to show where state-of-art hardening strategies are in our experimental setting.
We have revised the appendix with a newly added section 8.2. This section connects and compares "Siamese Training" to relevant hardening solutions, including "stability training" by Zheng16.
We have updated Definition 2.2 "ALMOST EVERYWHERE (A.E.) CONTINUITY"  and customized it for when f_i are classification functions.

Meta-review:
role: PC
data: comment: The authors propose a framework to analyze "robustness" to adversarial perturbations using topological concepts. The authors conduct an empirical study using a siamese networks. 
 
 The paper generated extensive discussions. The authors implemented many changes following the reviewers' suggestions. The resulting version of the paper is rather dense. It is unclear whether a conference is appropriate for reviewing such material in limited time. 
 
 We invite the authors to present their main results in the workshop track. Revising the material of the paper will generate a stronger submission to a future venue such as a journal.

Decision: Invite to Workshop Track