Venue: ICLR 2017 conference

Reviews:

final_score: rating: null
review_content: The robustness definition is not self-contained
- How does X1 and X2 relate to f1 and f2? It can be guessed based on the previous paragraph, but it is not specified formally in the definition.

In Theorem 3.7 what do d_1 and d_2 refer to?

In Figure 3, it is suggested: "This figure shows one situation that (X1, d1) and (X2, d2) are not topologically equivalent. Therefore, in this case, the DNN is vulnerable to adversarial attacks." This does not look like a valid implication.

The purely topological conclusions of Section 3 seem to be very disconnected to the quantitative measurements of Section 4. A more thorough exposure on the connections between the two sections would be helpful.
review_title: Questions on theoretical definitions

final_score: rating: 5: Marginally below acceptance threshold
review_content: Under my point of view, the robustness of a classifier against adversarial noise it is interesting if we find any relationship between that robustness and generalization to new unseen test samples. I guess that this relationship is direct in most of the problems but perhaps classifier C1 could be more robust than C2 against adv. noise but not better for new unseen samples from the task in consideration. Best results on new unseen samples are normally related to robustness against the common distortions of the data, e.g. invariance to scale, rotation… than robustness to adv. noise.

I can not see any direct conclusion from table 5 results. 

Essentially i am not convinced about the necessity to measure the robustness against adversarial noise.




review_title: Connection with generalization capabilities

final_score: rating: 5: Marginally below acceptance threshold
review_content: This paper theoretically analyzes the adversarial phenomenon by modeling the topological relationship between the feature space of the trained and the oracle discriminate function. In particular, the (complicated) discriminant function (f) is decomposed into a feature extractor (g) and a classifier (c), where the feature extractor (g) defines the feature space. 

The main contribution of this paper is to propose abstract understanding and analysis for adversarial phenomenon, which is interesting and important. 

However, this paper also has the following problems. 

1)	It is not clear how the classifier c can affect the overall robustness to adversarial noises. The classifier c seems absent from the analysis, which somehow indicates that the classifier does not matter. (Please correct me if it is not true) This is counter-intuitive. For example, if we always take the input space as the feature space and the entire f as the classifier c, the strong robustness can always hold. 
I am also wondering if the metric d has anything to do with the classifier c.
2)	A very relevant problem is how to decompose f into g and c. For examples, one can take any intermediate layer or the input space as the feature space for a neural network. Will this affect the analysis of the adversarial robustness?
3)	The oracle is a good concept. However, it is hard to explicitly define it. In this paper, the feature space of the oracle is just the input image space, and the inf-norm is used as the metric. This implementation makes the algorithm in Section 4 quite similar to existing methods (though there are some detailed differences as mentioned in the discussion). 

Due to the above problems, I feel that some aspects of the paper are not ready. If the problems are resolved or better clarified, I believe a higher rating can be assigned to this paper. 

In addition, the main text of this paper is somehow too long, the arguments can be more focused if the main paper become more concise.
 

review_title: Interesting idea, good insights, but have some flaws

final_score: rating: null
review_content: The theory provides some interesting perspectives for analyzing the adversarial phenomenon. The important thing is to link the theory with novel concrete algorithms. 

I have following questions for bettering understanding the paper:
1.	In Corollary 3.4 and Section 3.6, what are features and what are classifiers? 
2.	A similar question is that, in Section 4.2, what do middle layer outputs specifically refer to as? Is it for the penultimate layer or many intermediate layers?
3.	In Definition 4.1, f2(x) = f2(t) is finally determined by the inf norm. Then, what is the actual difference between the proposed ARC with existing methods? 
4.	In Equation (4.6), is L1 cross-entropy loss for the softmax classifier or is it also a Euclidean-based Siamese loss? Similarly, in Figure 5, how is the “original method” trained? With softmax classifier? 
5.	The Siamese training basically try to match features, what is the essential different with this paper ( https://arxiv.org/abs/1511.06381 )?

review_title: Some clarification

final_score: rating: 3: Clear rejection
review_content: This paper aims at making three contributions:
- Charecterizing robustness to adversarials in a topological manner.
- Connecting the topological characterization to more quantitative measurements and evaluating deep networks.
- Using Siamese network training to create models robust to adversarial in a practical manner and evaluate their properties.

In my opinion the paper would improve greatly if the first, topological analysis attempt would be removed from the paper altogether.

A central notion of the paper is the abstract characterization of robustness. The main weakness is the notion of strong robustness itself, which is an extremely rigid notion. It requires the partitioning of the predictor function by class to match the exact partitioning of the oracle. This robustness is almost never the case in real life: it requires that the predictor is almost perfect.

The main flaw however is that the output space is assumed to have discrete topology and continuity is assumed for the classifier. Continuity of the classifier wrt. a discrete output is also never really satisfied. However, if the output space is assumed to be continues values with an interesting topology (like probabilities), then the notion of strong robustness becomes so constrained and strict, that it has even less practical sense and relevance. Based on those definition, several uninteresting, trivial consequences follow. They seem to be true, with inelegant proofs, but that matters little as they seem irrelevant for any practical purposes.

The second part is a well executed experiment by training a Siamese architecture with an explicit additional robustness constraint. The approach seems to be working very well, but is compared only to a baseline (stability training) which performs worse than the original model without any trainings for adversarials. This is strange as adversarial examples have been studied extensively in the past year and several methods claimed improvements over the original model not trained for robustness.

The experimental section and approach would look interesting, if it were compared with a stronger baseline, however the empty theoretical definitions and analysis attempts make the paper in its current form unappealing.

review_title: Flawed topological considerations, interesting practical results

final_score: rating: 10: Top 5% of accepted papers, seminal paper
review_content: A promising attempt to quantify the difference between human annotator and DNN. The contribution of defining topological equivalence of two metric spaces is significant to understand the origin of adversarial examples. Appreciate the examples showed in Figure2 and Figure 3, straightforward and intuitively helpful. More interestingly, a useful measure is provided for quantify the robustness of DNN model. Researchers can use this measure to compare different DNN models.
 
Some critiques:
It might be nicer to think more about the detailed mechanics of human vision. In the early state of primary visual cortex, feature space might be linear, L2 norm might be sufficient to capture the key difference between DNN and human metrics. However, when the information flow goes deep, single one metric might not be sufficient when the Oracle feature space is highly curved. Tracking and reducing the metric distance in cognitive level would be extremely hard. 

This paper offers a pioneering advance to tackle this problem. Hope more work can be done to further our understanding towards the limitation of DNN.
review_title: Intuitively helpful illustration

Meta-review: The authors propose a framework to analyze "robustness" to adversarial perturbations using topological concepts. The authors conduct an empirical study using a siamese networks. 
 
 The paper generated extensive discussions. The authors implemented many changes following the reviewers' suggestions. The resulting version of the paper is rather dense. It is unclear whether a conference is appropriate for reviewing such material in limited time. 
 
 We invite the authors to present their main results in the workshop track. Revising the material of the paper will generate a stronger submission to a future venue such as a journal.
Decision: Invite to Workshop Track