Venue: ICLR 2018 conference

Reviews:

final_score: rating: 6: Marginally above acceptance threshold
review_content: This paper studies input discretization and white-box attacks on it to make deep networks robust to adversarial examples. They propose one-hot and thermometer encodings as input discretization and  
also propose DGA and LS-PGA as white-box attacks on it.
Robustness to adversarial examples for thermometer encoding is demonstrated through experiments.

The empirical fact that thermometer encoding is more robust to adversarial examples than one-hot encoding,
is interesting. The reason why thermometer performs better than one-hot should be pursued more.

[Strong points]
* Propose a new type of input discretization called thermometer encodings.
* Propose new white-box attacks on discretized inputs.
* Deep networks with thermometer encoded inputs empirically have higher accuracy on adversarial examples.

[Weak points]
* No theoretical guarantee for thermometer encoding inputs.
* The reason why thermometer performs better than one-hot has not unveiled yet.

[Detailed comments]
Thermometer encodings do not preserve pairwise distance information.
Consider the case with b_1=0.1, b_2=0.2, b_3=0.3, b_4=0.4 and x_i=0.09, x_j=0.21 and x_k=0.39.
Then, 0.12=|x_j-x_i|<|x_k-x_j|=0.18 but ||tau(b(x_i))-tau(n(x_j))||_2=sqrt(2)>1=||tau(b(x_k))-tau(n(x_j))||_2.
review_title: Thermometer encoding is an interesting input discretization that is empirically shown to be robust to adversarial examples.

final_score: rating: 6: Marginally above acceptance threshold
review_content: This is a beautiful work that introduces both (1) a novel way of defending against adversarial examples generated in a black-box or white-box setting, and (2) a principled attack to test the robustness of defenses based on discretized input domains. Using a binary encoding of the input to reduce the attack surface is a brilliant idea. Even though the dimensionality of the input space is increased, the intrinsic dimensionality of the data is drastically reduced. The direct relationship between robustness to adversarial examples and intrinsic dimensionality is well known (paper by Fawzi.). This article exploits this property nicely by designing an encoding that preserves pairwise distances by construction. It is well written overall, and the experiments support the claims of the authors. 

This work has a crucial limitation: scalability.
The proposed method scales the input space dimension linearly with the number of discretization steps. Consequently, it has a significant impact on the number of parameters of the model when the dimensionality of the inputs is large. All the experiments in the paper report use relatively small dimensional datasets. For larger input spaces such as Imagenet, the picture could be entirely different:

	- How would thermometer encoding impact the performance on clean examples for larger dimensionality data (e.g., Imagenet)?
	- Would the proposed method be significantly different from bit depth reduction in such setting? 
	- What would be the impact of the hyper-parameter k in such configuration?
	- Would the proposed method still be robust to white box attack?
	- The DGA and LS-PGA attacks look at all buckets that are
within ε of the actual value, at every step. Would this be feasible in a large dimensional setting? More generally, would the resulting adversarial training technique be practically possible?

While positive results on Imagenet would make this work a home run,  negative results would not affect the beauty of the proposal and would shed critical light on the settings in which thermometer encoding is applicable. I lean on the accept side, and I am willing to increase the score greatly if the above questions are answered.
review_title: Interesting, I want more.

final_score: rating: 6: Marginally above acceptance threshold
review_content: The authors present an in-depth study of discretizing / quantizing the input as a defense against adversarial examples. The idea is that the threshold effects of discretization make it harder to find adversarial examples that only make small alterations of the image, but also that it introduces more non-linearities, which might increase robustness. In addition, discretization has little negative impact on the performance on clean data. The authors also propose a version of single-step or multi-step attacks against models that use discretized inputs, and present extensive experiments on MNIST, CIFAR-10, CIFAR-100 and SVHN, against standard baselines and, on MNIST and CIFAR-10, against a version of quantization in which the values are represented by a small number of bits.

The merits of the paper is that the study is rather comprehensive: a large number of datasets were used, two types of discretization were tried, and the authors propose an attack mechanism better that seems reasonable considering the defense they consider. The two main claims of the paper, namely that discretization doesn't hurt performance on natural test examples and that better robustness (in the author's experimental setup) is achieved through the discretized encoding, are properly backed up by the experiments.

Yet, the applicability of the method in practice is still to be demonstrated. The threshold effects might imply that small perturbations of the input (in the l_infty sense) will not have a large effect on their discritized version, but it may also go the other way: an opponent might be able to greatly change the discretized input without drastically changing the input. Figure 8 in the appendix is a bit worrysome on that point, as the performance of the discretized version drops rapidly to 0 when the opponents gets a bit stronger. Did the authors observe the same kind of bahavior on other datasets? What would the authors propose to mitigate this issue? To what extend the good results that are exhibited in the paper are valid over the wide range of opponent's strengths?

minor comment:
- the experiments on CIFAR-100 in Appendix E are carried out by mixing adversarial / clean examples while training, whereas those on SVHN in Appendix F use adversarial examples only.

review_title: an interesting study, but the validity of the approach is still to be demonstrated

Meta-review: This paper is borderline.  The reviewers agree that the method is novel and interesting, but have concerns about scalability and weakness to attacks with larger epsilon.  I will recommend accepting; but I think the paper would be well served by imagenet experiments, and hope the authors are able to include these for the final version
Decision: Accept (Poster)