Venue: ICLR 2018 Workshop

Reviews:

final_score: rating: 3: Clear rejection
review_content: This paper shows that the "feature squeezing" defense against adversarial examples can be defeated using standard attacks, as long as the budget for the adversary is increased. Previous limitations on adversary strength may have been overly conservative, and there are visually indistinguishable adversarial examples that violate commonly-used constraints.

Analyzing the weaknesses of proposed defenses is a valuable contribution. I disagree with the comment thread about making the adversary stronger -- it's fine to evaluate against a stronger adversary (with looser constraints on the examples), as long as you can show that the resulting examples are sufficiently visually similar to the original images.

However, I thought that this paper fell short in two critical ways:

1. When the main innovation is a stronger or less-constrained adversary, it's very important to demonstrate that this adversary still generates good images. This was shown qualitatively with a small number of images, but since this claim is central, it needs a more thorough evaluation. This could be done with a user study, for example. (I was also confused by the epsilon=0.5 case -- shouldn't the adversary be able to create an all-gray image, which would always confuse a classifier? But I may have missed something, so I'll give it the benefit of the doubt.)

2. In the paper "Adversarial Example Defenses: Ensembles of Weak Defenses are not Strong" (He et al., 2017), the authors showed that feature squeezing is indeed vulnerable to adversarial attacks. This workshop paper needs to distinguish its contribution from previously published work. Feature squeezing was claimed to be effective, but that claim has already been disproven.
review_title: Feature squeezing is vulnerable, but this was already known

final_score: rating: 6: Marginally above acceptance threshold
review_content: By increasing the strength of adversarial attacks by tuning their parameters out of the standard range, the provided experiments on MIST and CIFAR-10 show that feature squeezing defense can be by passed.

The issue of this claim is that if there are no limit range for the adversarial attacks parameters, it is obviously possible to change the input image in a way that the image class changes.
The author uses an implicit limit range that corresponds to a soft criteria corresponding to visual similarity. 
Could you quantify "visual similarity" ?
What means L_2, L_2, or L_\infty norms in terms of visual similarity ?

When a classifier is built to discriminate images between 10 classes on an input space that contains much more that the 10 classes, is it surprising that some "small" perturbations can change the predicted classes ?
For most of input images the true class should be I don't know.

Overall I think that paper could be interesting to feed the discussions at the workshop.

review_title: This paper provides experiments that show the limits of feature squeezing defense 

Meta-review: Based on the reviews, this paper has not been accepted for presentation at the ICLR workshop. However, the conversation and updates can continue to appear here on OpenReview.
Decision: Reject