Venue: ICLR 2020 Conference

Reviews:

final_score: rating: 3: Weak Reject
review_content: This paper proposes an adversarial pNML scheme for adversarial defence and adversarial example detection. The idea is very intuitive and pNML is adopted from the literature work for the purpose of adversarial defence. 

The authors provided some explanation on why the adversarial pNML should work. The reasoning is quite intuitive, lacking of thorough justification.  The authors may consider using experiments to provide empirical justifications for the explanations. 

The proposed method is heavily dependent on previous works. The section 6 adaptive adversary part is not clear.  How to do the adaptive attack based on Eq.(16)? Maximizing the loss in Eq.(16)?  How to determine the threshold for adversarial example detection?

The experimental results in Table 1 seems to be very good. However, have the state-of-the-art defence methods been compared?

review_title: Official Blind Review #3

final_score: rating: 1: Reject
review_content: Summary:
This paper focuses on the area of adversarial defense -- both to improve robustness and to detect adversarially perturbed images. They approach the problem from the universal prediction / universal learning framework.
Modivation:
I had a hard time understanding the motivation of this work -- specifically the connection to the universal learning framework which to be fair I am unfamiliar with. In the absence of the universal learning framework formalism the method proposed here is quite simple and in my opinion clever -- create a new prediction based on performing an adversarial attack to each target class. What value does universal learning bring to this?
Second, I do not follow the intuition for the chosen hypothesis class -- why work off of refined images in the first place? Is there some reason to believe this will improve robustness?
Finally, the view that adversarial defense and attack are important topics to explore is under some debate. I am not considering this as part of my review but I would encourage the authors to look at [1].
Writing:
The writing was clear and typo free.
Experiments:
Overall the experiments seemed inconclusive.
Section 5 shows robustness against the unmodified / unrefined model (the attacks are done on the base model not the refined model). Given that these attacks are performed against the unmodified model then evaluated on the modified model the results seem a bit unfair / harder to interpret. The authors note this, and in Section 6 explore the "Adaptive Adversary" setting.
The results presented are performed on Mnist and Cifar10. Overall the results were not convincing to me. Table 1 shows mixed performance -- a drop in natural accuracy in all cases, decreases in FGSM. The main increase in performance is in the PGD. This was noted, but understanding in more depth why this method helps here will hopefully lead to improved performance in FGSM as well.
Figure 2a shows very weak correlations. Figure 2b seems promising but also not necessarily a surprise given that the adversarial examples are generated against the base model and not the refined model.
For section 6, one risk is that the BPDA attack doesn't successfully work. Having some more proof that the attacks presented here are strong would greatly improve the work.
Larger scale experiments would of course be nice and strengthen the paper but more importantly it would be great to see some form of toy example or demonstration of the principle improving robustness as well over just results. Something to probe the mechanism of action for example.
Finally, having some comparisons to other defense strategies would improve this paper.
Rating:
Given the gap between the universal learning framework and the method proposed, as well as the inconclusive experiments at this point I would not recommend the paper for acceptance.

[1] https://arxiv.org/abs/1807.06732
review_title: Official Blind Review #2

final_score: rating: 3: Weak Reject
review_content: In this paper, the authors proposed the Adversarial predictive normalize maximum likelihood (pNML) scheme to achieve adversarial defense and detection. 
The proposed method is an application of universal learning, which is compatible with existing adversarial training strategies like FGSM and PGD. 
However, the experimental results indicate that the proposed method is more suitable for the models trained under PGD-based attacks. 
According to the analysis shown in the paper, the proposed method works best when the adversary finds a local maximum of the error function, which makes it more robust to strong attacks. 
It seems that the proposed work is a good attempt that applies universal learning to adversarial training, but more experiments are required to support its usefulness and effectiveness, especially for the weak attack like FGSM. Additionally, I would like to see more discussions about the limitations of the proposed method. 

Minors:
In Figure 2, I would like to see the results related to FGSM.
review_title: Official Blind Review #1

Meta-review: The reviewers attempted to give this paper a fair assessment, but were unanimous in recommending rejection.  The technical quality of motivation was questioned, while the experimental evaluation was not found to be clear or convincing.  Hopefully the feedback provided can help the authors improve their paper.
Decision: Reject