Venue: ICLR 2020 Conference

Reviews:

final_score: rating: 3: Weak Reject
review_content: This paper proposes a method to create adversarial perturbations whose target labels are similar to their ground truth. The target labels are selected using an existing perceptual similarity measure for images.  Perturbations are generated using a DeepFool-like algorithm. Human evaluation supports that the pair of the generated images and target labels are more natural to humans than prior attack algorithms.

This paper should be rejected due to the lack of motivation to create adversarial examples less detectable by humans automatically. Attackers can manually select target labels and apply targeted attacks. In the target label selection, attackers can choose less detectable labels if necessary. It is encouraged to provide some applications where attackers want to create less detectable adversarial examples in label space without manually assigning target labels.

==========
Update:

After reading the authors' responses, the motivation of the paper became clearer. I will not get surprised if this paper is accepted. However, all reviewers still share concerns about the importance of the problem tackled. I think the paper needs to suggest more applications and emphasize the value of the goal in the main paper before being published.
review_title: Official Blind Review #4

final_score: rating: 3: Weak Reject
review_content: This paper describes a technique for creating adversarial images where the added perturbations are not only imperceptible to machines, but also to human observers. The authors describe why this might be beneficial. The method works by finding labels that are not too far from the source image's ground-truth labels, and moving the source image in that direction. To find the target label, the authors use a threshold on the confidence of predicted ground-truth labels. The authors test their algorithm using a newly proposed metric of how much a method allows imperceptibility for a human observer. They show that their method creates images whose perturbations are more impercetible to humans, compared to other methods, but are also imperceptible to machines.

My concern is as follows: If the misclassification is between A and B, and they are related classes, is the attack so bad? And what are the scenarios in practice when a user simply wants to create an attack, without regards to the target label chosen? I imagine normally the attacker has a target label in mind, so the part of the paper that chooses a target label is not very useful; and this is the main element of novelty, since the rest of the method is from DeepFool, as the authors explain. Some specific use cases of this methods should be discussed. 

Minor suggestion: It would be useful to see examples like in Fig. 5 but with the classes (true/target) listed.
review_title: Official Blind Review #2

final_score: rating: 1: Reject
review_content: This paper proposes a method for constructing adversarial attacks that are less detectable by humans, by changing the target class to be a class similar to the original class of the image. The resulting attack methodology is then studied in terms of its imperceptibility in label space, and shown to be less perceptible in label space to human observers, while not coming at a cost in image space.

The paper presents compelling evaluation of the method and does seem to succeed in proving that their proposed attack satisfies the stated goal. However, it appears as though this goal is somewhat counter to the main point of adversarial examples---indeed, if the label is reasonable to a human, then what makes the adversarial example adversarial? The main threat in adversarial examples research seems to be that it is possible to induce predictions that are arbitrarily different from humans' on natural-looking in puts. Thus, changing the label to something that a human actually agrees with would actually reduce the impact of the adversarial attack.

In order to improve the paper, I would suggest applying the same (or similar) methodologies to other areas of ML security where imperceptibility in label space is commonly desired---for example, in data poisoning attacks or backdoor attacks. In general, such attacks are much more likely to be "inspected" by humans, and so imperceptibility in both label and image space is very desirable. However, I suspect that this would require significant effort and changes to the paper, and so for now I recommend rejection.
review_title: Official Blind Review #3

Meta-review: Thanks for the discussion with reviewers, which improved our understanding of your paper significantly.
However, we concluded that this paper is still premature to be accepted to ICLR2020. We hope that the detailed comments by the reviewers help improve your paper for potential future submission.
Decision: Reject