Venue: ICLR 2020 Conference

Reviews:

final_score: rating: 3: Weak Reject
review_content: 
====== AFTER READING THE AUTHOR RESPONSE ======

Many thanks for the extensive response and the respective revision from the author(s).
Mainly, I found the main results are adjusted in the revision to rather demonstrate its good detection performance at 99% TPR, and I feel the message of the manuscript becomes more strengthen. However, at the same time I feel the proposed method would be slightly less motivated without the improved results of the "pure" robust accuracy, as the method itself is anyway a variant of adversarial training. There are several works that specially focus on detecting adversarial examples [1, 2], and comparing the results with them on diverse threat models would more strengthen the paper. 

In overall, I keep my score unchanged to the current version of manuscript.

[1] A Simple Unified Framework for Detecting Out-of-Distribution Samples and Adversarial Attacks, NeurIPS 2018
[2] The Odds are Odd: A Statistical Test for Detecting Adversarial Examples, ICML 2019

===============================================

The paper proposes a new adversarial training scheme to improve generalization of robustness over unseen threat models, e.g. larger perturbations or different noise distributions. The key idea is to impose uniform confidence on the adversarial examples depending on the distance from the original example, based on a pre-determined distance metric, e.g. L-infinity distance. Experimental results shows its effectiveness on detecting adversarial examples and unseen robustness for MNIST, SVHN, and CIFAR-10 datasets, under L-infinity and L-2 adversaries.

In overall, I agree that improving generalization over unforeseen adversaries is a very important problem in adversarial training, and the paper addresses this problem with a novel approach. The manuscript is generally well-presented with clear motivation. In particular, I appreciated the simplicity of the proposed idea, and the thoroughness of experiments as a defense paper, e.g. presenting per-example worst-case results across diverse attacks. However, I am currently on a slightly negative side, due to some unclear points in the experimental results. I would like to increase the score if the issue could be addressed, regarding the importance of the problem and their approach, which seems valuable to be shared in the community.

Mainly, it is still hard for me to interpret the presented experimental results at the positive side, unless the authors could further address on the important points of the results. In general, the results are not that clear as claimed, especially when tau=0, to show that CCAT improves robustness: At the original threat model (L-inf with the smallest epsilon), CCAT shows much inferior results across all the datasets. It does improves the L-2 results, except for CIFAR-10 which should be a bare baseline to show the scalability of the method. Although the paper also point out that CCAT sometimes achieves much lower clean test error, but sometimes this also signals the less robustness. I hope the paper could justify such points in the Table 2 for better presentation.

- Perhaps ResNet-20 is too small for CIFAR-10 tasks, as the training loss would hardly minimized into 0? I think WRN-10 is a more fair standard for CIFAR-10 in AT, and wonder if the result could show more effectiveness (or get more aligned tendency across datasets) of the proposed method if the capacity of network is increased.
- Apart from the thoroughness of the attack methods assumed, considering only L-inf and L-2 adversaries may be not enough to claim the "generalization ability" of a defense. Could the method also improves robustness against other attacks, e.g. general corruption (such as MNIST-C, CIFAR-10-C), or unrestricted adversarial examples?
- Eq 4: Does it mean that the logits other than the 2nd predictions are not considered when generating adversarial examples? Personally, I don't much get the motivation of using this objective, even while it is described in below Eq. 4.
review_title: Official Blind Review #2

final_score: rating: 3: Weak Reject
review_content: Summary:
This paper proposes the "confidence-calibrated adversarial training (CCAT)" to train robust DNNs against adversarial examples. The idea is to perform adversarial training with gradually smoothed network predictions (in an exponential or power rate to perturbation size). This new approach seems can help detection and defense at the same time to some extent. While the idea is quite interesting, the paper is poorly written and many times hard to read. 

Here are some detailed comments:
1. Introduction. It is hard to get how the "robustness-accuracy trade-off" and the "poor generalization to other or stronger attacks" are addressed by CCAT. I can hardly agree that "standard adversarial training strongly depends on the training attack". PGD-AT generalizes pretty well to other types of attacks such as CW. It needs more concrete evidence to make such a claim.
2. Related work can be improved. There are a few places inaccurate descriptions, like "White-box attacks utilizing projected gradient ascent to ...". And since the focus of this paper is adversarial training, related works deserves more detailed discussions.
3. Section 3, the 100%/50% adversarial training is not properly explained. For example,  in (3), I cannot tell how the 50% comes from, as input x in the two components are the same. 
4. The confidence calibration seems related to the reverse cross entropy (RCE) proposed in [*] for detection?
5. Figure 1, missing legends. I cannot find where the PGD-Conf is defined when came across Figures 1/2.
6. Experiments. The settings are not standard, which makes results hard to interpret. The Err, RErr metrics are confusing. Not sure what 99% TPR is for, detection?
7. Hard to tell the real improvement. For example , in Table 2, CIFAR-10, L_{\infty}=0.0.3, CCAT has even lower RErr (\tau =0) and ROC (\tau@99%TPR) than standard AT.

[*] Pang, Tianyu, et al. "Towards robust detection of adversarial examples."

Advances in Neural Information Processing Systems. 2018.
Missing citations  of a few recent papers in adversarial training:
[1] Zhang, Hongyang, et al. "Theoretically principled trade-off between robustness and accuracy." ICML, 2019.
[2] Wang, Yisen, et al. "On the Convergence and Robustness of Adversarial Training." ICML, 2019.
[3] Carmon, Yair, et al. "Unlabeled data improves adversarial robustness." NeurIPS, 2019.
[4] Uesato, et al. "Are Labels Required for Improving Adversarial Robustness?" NeurIPS, 2019.

A few additional high-level suggestions: 
1) Show some concrete examples where standard adversarial training fails while CCAT is not. 
2) Show the formulation of CCAT a little earlier in Section 3. Then link back to standard adversarial training or other defense methods. Not the other way around like it did now (taking too long to get to the idea of CCAT). 
3. Use a standard experiment setting, and compare 1-2 more existing methods fairly. 
4. Show understanding and analysis on CIFAR-10 dataset instead of SVHN.

=============
I appreciate the substantial revisions made during the rebuttal. The paper reads much better now. Unfortunately, my rating remains the same. My major concern goes to the unusually setting adopted in this paper. Detection and defense are two different types of adversarial research. I appreciate the efforts this paper has put to try combining them together into a unified robust defense (eg. detection first, then defense). Specifically, the use of confidence thresholding (equivalent to detection) before defense. This is also the main reason that causes the other reviewers hard to understand the real impact made in this paper. It is not fair to directly compare AT with CCAT, as AT does not have the detection component. I suggest the authors to include an existing detection method (LID (Ma et al. 2018), or Mahalanobis score for AT detection ("A Simple Unified Framework for Detecting Out-of-Distribution Samples and Adversarial Attacks"), before applying AT for prediction. Overall, the authors have done a good rebuttal, but it still requires much improvement to be accepted.

Typo:
1. Table 3, the 3rd column should be Cifar10 not SVHN.

review_title: Official Blind Review #3

final_score: rating: 6: Weak Accept
review_content: *Edit after rebuttal*
See the comment below for a response to the author's rebuttal. I have updated my score to a weak accept.

Summary
========
This paper proposes a more granular adversarial training scheme, where the model is trained to have decaying confidence as the size of the adversarial perturbations increase. 
The idea is quite natural and well motivated and presented. The paper suffers in its theoretical and evaluation sections though, which were both fairly confusing to me. I lean towards rejection for this paper but would consider changing my score if the presentation was improved.


Comments
=========
The observation that adversarial training enforces high confidence in the entirety of the epsilon ball is interesting, and the motivation for the proposed scheme is sound.

Yet, while the intuition of CCAT for addressing the accuracy-robustness trade-off makes sense to me, the purported beneficial effect for perturbations outside the given epsilon ball are not as clear.
The authors note in Section 3.1 that adversarial training gives no indications on how the model should behave outside the given epsilon ball, or on other threat models. While this is true, it is not clear how CCAT supposedly remedies this. By making the confidence decay within the epsilon ball, the training also doesn't explicitly enforce any behavior outside this ball or on other threat models. So if CCAT indeed provides benefits in those settings, it isn't clear why those gains should be expected.

The exposition of the technique in Section 3.2. is very clear, and Figure 1 nicely illustrates the effect of CCAT on the class confidences during an attack.

The theoretical toy example in Section 3.3. is a bit confusing, and I'm not convinced that the proof of Proposition 1 is correct. First, a classification problem parametrized by the class imbalance is a little weird in this setting. I have a few questions about the proof:
- You seem to be assuming that the model's confidence is calculated using a softmax. This assumption should be stated.
- The expressions for \hat{p}(y=2|x=x) and \hat{p}(y=1|x=x) are confusing to me. Why are you taking a log over the softmax? These probabilities are now negative... Also, in the second step, an equality of the form e^x / (e^y + e^x) = 1+e^(y-x) is used. This is incorrect, it should be 1/(1+e^(y-x)). In the end, the values log(1+e^a) and log(1+e^(-b)) are not in [0, 1] so they cannot be probabilities.
- From there on, it isn't clear what the rest of the proof is really saying

The experimental section is quite thorough and the results seem convincing. The considered adaptive attack, that optimizes (4) seems natural given that the defense thresholds on the model's confidence. I wonder how this is optimized though: Do you try a targeted attack for every possible target other than the true class?

I found the conflation of evaluation metrics quite confusing. It seems the authors know this and have done an effort to explain their choices, but I feel like this could still be improved further. For example, some of the metrics (e.g., AUC) should be maximized, while others (Err or RErr) should be minimized. This makes the reading of the result tables quite difficult.I would also suggest dropping one digit of precision in the Tables for better readability.

Overall, I have some mixed feelings about this paper. The idea is simple and well explained. The motivation, theoretical analysis, and reporting of results are somewhat confusing though. Some extra work on the presentation could definitely help this paper.

Some additional references
========================
- In the introduction, when mentioning the trade-off between robustness and accuracy, you should cite "Robustness May Be at Odds with Accuracy" by Tsipras et al. rather than Schmidt et al. (that paper shows that robustness may require more training data).
- Also in the introduction, you could consider referencing the following papers on robustness to different threat models:
    * Engstrom et al., "Exploring the Landscape of Spatial Robustness"
    * Tramer and Boneh, "Adversarial Training and Robustness for Multiple Perturbations"
- In the related work, you reference a number of works for adversarial training but not the original idea in the paper by Szegedy et al.
- In Section 3.1., the observation that the epsilon-ball around training examples can cross class boundaries is further analyzed in "Excessive Invariance Causes Adversarial Vulnerability" and "Exploiting Excessive Invariance caused by Norm-Bounded Adversarial Robustness" by Jacobsen et al.

Typos
=====
p.6 enumerator -> numerator
review_title: Official Blind Review #1

Meta-review: This paper proposes a confidence-calibrated adversarial training (CCAT). The key idea is to enforce that the confidence on adversarial examples decays with their distance to the attacked examples. The authors show that CCAT can achieve better natural accuracy and robustness. After the author response and reviewer discussion, all the reviewers still think more work (e.g., improving the motivation to better position this work, conducting a fair comparison with adversarial training which does not have adversarial example detection component) needs to be done to make it a strong case. Therefore, I recommend reject.
Decision: Reject