Venue: ICLR 2018 conference

Reviews:

final_score: rating: 4: Ok but not good enough - rejection
review_content: The manuscript proposes two objective functions based on the manifold assumption as defense mechanisms against adversarial examples. The two objective functions are based on assigning low confidence values to points that are near or off the underlying (learned) data manifold while assigning high confidence values to points lying on the data manifold. In particular, for an adversarial example that is distinguishable from the points on the manifold and assigned a low confidence by the model, is projected back onto the designated manifold such that the model assigns it a high confidence value. The authors claim that the two objective functions proposed in this manuscript provide such a projection onto the desired manifold and assign high confidence for these adversarial points. These mechanisms, together with the so-called shell wrapper around the model (a deep learning model in this case) will provide the desired defense mechanism against adversarial examples.

The manuscript at the current stage seems to be a preliminary work that is not well matured yet. The manuscript is overly verbose and the arguments seem to be weak and not fully developed yet. More importantly, the experiments are very preliminary and there is much more room to deliver more comprehensive and compelling experiments.
review_title: Interesting ideas but at the current stage, this seems to be a preliminary work that is not well matured yet!

final_score: rating: 5: Marginally below acceptance threshold
review_content: 
1) Summary
This paper proposes a new approach to defending against adversarial attacks based on the manifold assumption of natural data. Specifically, this method takes inputs (possibly coming from an adversarial attack), project their semantic representation into the closest data class manifold. The authors show that adversarial attack techniques can be with their algorithm for attack prevention. In experiments, they show that using their method on top of a base model achieves perfect success rate on attacks that the base model is vulnerable to while retaining generalizability.


2) Pros:
+ Novel/interesting way of defending against adversarial attacks by taking advantage of the manifold assumption.
+ Well stated formulation and intuition.
+ Experiments validate the claim, and insightful discussion about the limitations and advantages of the proposed method.

3) Cons:
Number of test examples used too small:
As mentioned in the paper, the number of testing points is a weakness. There needs to be more test examples to make a strong conclusion about the method’s performance in the experiments.

Comparison against other baselines:
Even though the method proposes a new approach for dealing with adversarial attacks using Madry et al. as base model, it would be useful to the community to see how this method works with other base models.

Algorithm generalizability:
As mentioned by the authors, their method depends on assumptions of the learned embeddings by the model being used. This makes the method less attractive for people that may be interested in dealing with adversarial examples in, for example, reinforcement learning problems. Can the authors comment on this?

Additional comments:
The writing needs to be polished.


4) Conclusion:
Overall, this is a very interesting work on how to deal with adversarial attacks in deep learning, while at the same time, it shows encouraging results of the application of the proposed method. The experimental section could improve a little bit in terms of baselines and test examples as previously mentioned, and also the authors may give some comments on if there is a simple way to make their algorithm not depend on assumptions of the learned embeddings.

review_title: Review

final_score: rating: 3: Clear rejection
review_content: The authors argue that "good" classifiers naturally represent the classes in a classification as well-separated manifolds, and that adversarial examples are low-confidence examples lying near to one of these manifolds. The authors suggest "fixing" adversarial examples by projecting them back to the manifold, essentially by finding a point near the adversarial example that has high confidence.

There are numerous issues here, which taken together, make the whole story pretty unconvincing.

The term "manifold" is used very sloppily. To be fair, this is unfortunately common in modern machine learning. An actual manifold is a specific mathematical structure with specific properties. In ML, what is generally hypothesized is that the data (often per class) lives "near" to some "low-dimensional" structure. In this paper, even the low-dimensionality isn't used --- the "manifold assumption" is used as a stand-in for "the regions associated with different classes are well-separated." (This is partially discussed in Section 6, where the authors point out correctly that the same defense as used here could be used with a 1-nn model.) This is fine as far as it goes, but the paper refs Basri & Jacobs 2016 multiple times as if it says anything relevant about this paper: Basri & Jacobs is specifically about the ability of deep nets to fit data that falls on (actual, mathematical) manifolds. This reference doesn't add much to the present story.

The essential argument of the paper rests on the "Postulate: (A good model) F is confident on natural points drawn from the manifolds, but has low confidence on points outside of the manifolds." 

This postulate is sloppy and speculative. For instance, taken in its strong form, if believe the postulate, then a good model:
1. Can classify all "natural points" from all classes with 100% accuracy.
2. Can detect adversarial points with 100% accuracy because all high-confidence points are correct classifications and all low-confidence points are adversarial.
3. All adversarial examples will be low-confidence.

Point 1 makes it clear that no good model F fully satisfying the postulate exists --- models never achieve 100% accuracy on difficult real-world distributions. But the method for dealing with adversarial examples seems to require Points 2 and 3 being true.

To be fair, the paper more-or-less admits that how true these points are is not known and is important. Nevertheless, I think this paper comes pretty close to arguing something that I *think* is not true, and doesn't do much to back up its argument. Because of the quality of the writing (generally sloppy), it's hard to tell, but I believe the authors are basically arguing that:
a. You can generally easily detect adversarial points because they are low confidence.
b. If you go through a procedure to find a point near your adversarial point that is high-confidence, you'll get the "correct" (or perhaps "original") class back.

I think b follows from a, but a is extremely suspect. I do not personally work in adversarial examples, and briefly looking at the literature, it seems that most authors *do* focus on how something is classified and not its confidence, but I don't think it's *that* hard to generate high-confidence adversarial examples. Early work by Goodfellow et al. ("Explaining and Harnessing Adversarial Examples", Figure 1, shows an example where the incorrect classification has very high confidence. The present paper only uses Carlini-Wagner attacks. From a read of Carlini-Wagner, it seems they are heavily concerned with finding *minimal* perturbations to achieve a given misclassification; this will of course produce low-confidence adversaries, but I see no reason why this is a general property of all adversarial examples.

The experiments are weak. I applaud the authors for mentioning the experiments are very preliminary, but that doesn't make them any less weak. 

What are we to make of the one image discussed at the end of Section 5 and shown in Figure 1? The authors note that the original image gives low-confidence for the correct class. (Does this mean that the classifier isn't "good"? Is it evidence against some kind of manifold assumption?) The authors note the adversarial category has significantly higher confidence, and say "in this case, it seems that it is the vagueness of the signals/data that lead to a natural difficulty." But the signals and data are ALWAYS vague. If they weren't, machine learning would be easy. This paper proposes something, looks at a tiny number of examples, and already finds a counterexample to the theory. What's the evidence *for* the theory? 

A lot of writing is given over to how this method is "semantic", and I just don't buy it. The connection to manifolds is weak. The basic argument here is really "(1) If our classifiers produce smooth well-separated high-confidence regions, (2) then we can detect adversaries because they're low-confidence, and (3) we can correct adversaries by projecting them back to high-confidence." (1) seems vastly unlikely to me based on all my experience: neural nets often get things wrong, they often get things wrong with high confidence, and when they're right, the confidence is at least sometimes low. The authors use a sloppy postulate about good models and so could perhaps argue I've never seen a good model, but the methods of this paper require a good model. (2) seems to follow logically from (1). (3) is also suspect --- perturbations which are *minimal* can be corrected as this paper does (and Carlini-Wagner attacks are minimal by design), but there's no reason to expect general perturbations to be minimal.

The writing is poor throughout. It's generally readable, but the wordings are often odd, and sometimes so odd it's hard to tell what was meant. For instance, I spent awhile trying to decide whether the authors assumed common classifiers are "good" (according to the postulate) or whether this paper was about a way to *make* classifiers good (I eventually decided the former).
review_title: Not convincing.

Meta-review: The original paper was sloppy in its use of mathematical constructs such as manifolds, made assumptions that are poorly motivated (see review #2 for details), and presented an empirical evaluation is preliminary. Based on the reviews, the authors have substantially revised the paper to try and address those issues by adding new theory, etc.

Unfortunately, it is difficult to assess whether these revisions are sufficient to address the aforementioned issues without going through a second round of "full" review. I encourage the authors to use the reviewer comments to further improve the paper, and re-submit to a different venue.
Decision: Reject