Venue: ICLR 2018 conference

Reviews:

final_score: rating: 6: Marginally above acceptance threshold
review_content: The authors propose a simple defense against adversarial attacks, which is to add randomization in the input of the CNNs. They experiment with different CNNs and published adversarial training techniques and show that randomized inputs mitigate adversarial attacks. 

Pros:
(+) The idea introduced is simple and flexible to be used for any CNN architecture
(+) Experiments on ImageNet1k prove demonstrate its effectiveness
Cons:
(-) Experiments are not thorougly explained
(-) Novelty is extremely limited
(-) Some baselines missing


The experimental section of the paper was rather confusing. The authors should explain the experiments and the settings in the table, as those are not very clear. In particular, it was not clear whether the defense model was trained with the input randomization layers? Also, in Tables 1-6, how was the target model trained? How do the training procedures of target vs. defense model differ? In those tables, what is the testing procedure for the target model and how does it compare to the defense model? 

The gap between the target and defense model in Table 4 (ensemble pattern attack scenario) shrinks for single step attack methods. This means that when the attacker is aware of the randomization parameters, the effect of randomization might diminish. A baseline that reports the performance when the attacker is fully aware of the randomization of the defender (parameters, patterns etc.) is missing but is very useful.

While the experiments show that the randomization layers mitigate the effect of randomization attacks, it's not clear whether the effectiveness of this very simple approach is heavily biased towards the published ways of generating adversarial attacks and the particular problem (i.e. classification). The form of attacks studied in the paper is that of additive noise. But there is many types of attacks that could be closely related to the randomization procedure of the input and that could lead to very different results.
review_title: unclear effects of randomization

final_score: rating: 7: Good paper, accept
review_content: This paper proposes an extremely simple methodology to improve the network's performance by adding extra random perturbations (resizing/padding) at evaluation time.

Although the paper is very basic, it creates a good baseline for defending about various types of attacks and got good results in kaggle competition.

The main merit of the paper is to study this simple but efficient baseline method extensively and shows how adversarial attacks can be mitigated by some extent.

Cons of the paper: there is not much novel insight or really exciting new ideas presented.

Pros: It gives a convincing very simple baseline and the evaluation of all subsequent results on defending against adversaries will need to incorporate this simple defense method in addition to any future proposed defenses, since it is very easy to implement and evaluate and seems to improve the defense capabilities of the network to a significant degree. So I assume that this paper will be influential in the future just by the virtue of its easy applicability and effectiveness.


review_title: Simple new baseline (/additional evaluation technique) for defenses against adversarial attacks.

final_score: rating: 6: Marginally above acceptance threshold
review_content: The paper basically propose keep using the typical data-augmentation transformations done during training also in evaluation time, to prevent adversarial attacks. In the paper they analyze only 2 random resizing and random padding, but I suppose others like random contrast, random relighting, random colorization, ... could be applicable.

Some of the pros of the proposed tricks is that it doesn't require re-training existing models, although as the authors pointed out re-training for adversarial images is necessary to obtain good results.


Typically images have different sizes, however in the Dataset are described as having 299x299x3 size, are all the test images resized before hand? How would this method work with variable size images?

The proposed defense requires increasing the size of the input images, have you analyzed the impact in performance? Also it would be good to know how robust is the method for smaller sizes.

Section 4.6.2 seems to indicate that 1 pixel padding or just resizing 1 pixel is enough to get most of the benefit, please provide an analysis of how results improve as the padding or size increase. 

In section 5 for the challenge authors used a lot more evaluations per image, could you provide how much extra computation is needed for that model?


review_title: Simple idea but seems work in well in some cases

Meta-review: Paper proposes adding randomization steps during inference time to CNNs in order to defend against adversarial attacks.

Pros:

- Results demonstrate good performance, and the team achieve a high rank (2nd place) on a public benchmark.
- The benefit of the proposed approach is that it does not require any additional training or retraining.

Cons:

- The approach is very simple, common sense would tend to suggest that adding noise to images would make adversarial attempts more difficult. Though perhaps simplicity is a good thing.
- Update: Paper does not cite related and relevant work, which takes a similar approach of requiring no retraining, but rather changing the inference stage: https://arxiv.org/pdf/1709.05583.pdf
  
Grammatical Suggestions:

This paper would benefit from polishing. For example:

- Abstract: sentence 1: replace “their powerful ability” to “high accuracy”
- Abstract: sentence 3: replace “I.e., clean images…” with “For example, imperceptible perturbations added to clean images can cause convolutional neural networks to fail”
- Abstract: sentence 4: replace “utilize randomization” to “implement randomization at inference time” or something similar to make more clear that this procedure is not done during training.
- Abstract: sentence 7: replace “also enjoys” with “provides”

Main Text: Capitalize references to figures (i.e. “figure 1” to “Figure 1”).

Introduction: Paragraph 4: Again, please replace “randomization” with “randomization at inference time” or something similar to better address reviewer concerns.

Decision: Accept (Poster)