Venue: NeurIPS 2021 Conference

Reviews:

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: The paper proposes a new method for training models to achieve better certified L-2 robustness. To this extent, SmoothMix generates adversarial examples by relaxing the perturbation budget and minimizes a mixup loss. The proposed procedure identifies instances classified with high confidence and near-off-class samples as causes of limited robustness in smoothed classifiers and offers an intuitive way to adaptively set a new decision boundary between these samples for better robustness. Finally, the method obtains improved certified accuracy for certain radii ε that each adversarial perturbation must be in.
main_review: Strengths
1. The proposed method is motivated and well-explained.
2. Evaluation results using standard datasets, such as MNIST (Table 1),  Cifar-10 (Table 2), and ImageNet (Table 4), show the effectiveness of SmoothMix across different smoothing factors.
3. Extensive ablation studies aid in understanding the utility of SmoothMix.

Open concerns
1. Average certified radius (ACR) is used to quantify the effectiveness of SmoothMix. However, it may be sensitive to particular adversarial samples, i.e., a few generated adversarial instances could have a high ACR score and thus can mislead the global interpretation of ACR. Analyzing the distribution of ACR scores might shed more light on this point.
2. SmoothMixs' performance on the certified accuracy at different radii and epsilons is not consistent across datasets. In addition, it achieves marginal improvement in most cases. This is observed even for easier datasets like MNIST, where we see the difference gap getting bigger for higher epsilons (this is also arguable as to how practical it is to generate adversarial samples using such larger perturbations).
limitations_and_societal_impact: Yes, the authors have clearly detailed the limitations and the broader societal impact of the work.
review_title: null

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: This paper looks to improve upon SmoothAdv - a methodology that employs adversarial training on a smoothed classifier to further improve certified robustness. The methodology employed looks to tackling overconfident examples rather than adversarial examples by utilising MixUp. They demonstrate that we can combine MixUp with SmoothAdv to improve certified accuracy.
main_review: UPDATE:
The authors have made clearer the motivation of overconfident examples, as a result I am increasing my score to 6. 

Strengths:
- This paper demonstrates we can combine MixUp and adversarial training to improve certified accuracy for smoothed classifiers. 

Weakness and Questions:
- One main concern I have is that the improvements shown in Tables 1 and 2 seems marginal, for example SmoothMix obtains 55.3% accuracy for CIFAR-10 when $\sigma$=0.25 whereas for Consistency it is 55.2%. Given these small margins, the authors should also report error bars on the approximate certified accuracy.

- The main motivation regarding why overconfident examples should be penalised is talked about in Section 3.1. I found this section a bit confusing to read, in particular the two bullet points seems speculative.  I've read bullet point one several times and I'm still unclear what it is trying to say. Further, both bullet points refers to Figure 2 which pinpoints to just two examples. To ensure these are not cherry-picked or anecdotal examples, the authors should show statistically that smoothed classifiers is overconfident towards the direction of adversarial perturbation. 

- It seems throughout when SmoothAdv was used in conjunction, the attack used was only one-step. This has been to known to cause gradient obfuscation / catastrophic overfitting especially when the perturbation radius is large, given that the adversarial example here is unrestricted it would be good to see the results for multi-step attacks as well. Otherwise it is unclear what the one-step attack is doing.

- From Figure 5/6 it would appear that SmoothMix performs worse when the certification radius is smaller compared to SmoothMix and Consistency. This doesn’t seem desirable, can the authors explained why this is? 

limitations_and_societal_impact: Yes
review_title: null

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: The paper observes a connection between the confidence with which a smooth model classifies an input and the robustness radius with which the input's prediction can be certified. In particular, the paper argues that the model's over-confidence on some inputs can be a source for detrimental effects on the certified radius of other (nearby) instances. Inspired by this, the paper proposes to find over-confident near off-class samples (somewhat) near training instances, and conduct mixup training on this pair of instances as a way of improving the certified accuracy of the resulting smoothed classifier. The proposal can be regarded as another way of adversarial training that is better crafted for smoothed classifiers. Experiments conducted on MNIST, CIFAR and ImageNet show marginal but consistent improvements in certified accuracy.
main_review: Strengths:
+ The paper's object of study, i.e. modifying the training of smooth classifiers for enhancing the classifier's certified accuracy, is of interest to the adversarial robustness and certification communities, and in someway of interest for the deep learning community in general.
+ The paper's writing is mostly clear, following a narrative that properly exposes the paper's motivation, relation to previous works and the rationale behind the experiments that are conducted. Figures are also well-built and clear.
+ The experimental protocol is standard and sensible.
+ Experimental results are mostly consistent across factors being studied (datasets, parameters, etc.).
+ The paper's motivation and formulation (although as I expose next I do not entirely understand or agree with) draws on previous works but is for the largest part original.

Weaknesses that affect my rating:
+ In general I find two main issues with the paper: (1) the motivation is still not completely clear to me, and (2) the experimental results are somewhat not significant enough. In particular, I find the motivation rather troubling, as I am not entirely sure about the over-confidence item raised in the paper: is this over-confidence truly a common phenomenon in *smoothed* classifiers? Given the relation between smoothed classifiers and "Lipschitzness" I would have initially thought of the opposite, and so I think this (fundamental) claim in the paper would require experimental validation. Further, while the paper repeats several times that the adversaries are "unconstrained" in practice that does not appear to be the case, precisely because part of the importance of finding adversarial examples is that they are near the original inputs, so perhaps a more precise term for this would simply be adversarial examples with looser constraint. 

Next, I outline some questions that better expose what I find unclear about the paper, and so I would be thankful for further explanation.
+ Eqn. (8) implies that the adversarial examples that the optimization procedure most likely finds are the ones for which the label $y$ has low score. How is this quantity related to "over-confidence", could it not be the case that simply the classifier will be not confident towards $y$ but uniformly confident towards the rest of the classes? Why would over-confidence emerge here?
+ I do not understand the claims made in the bullet point of L138. Fig. 2 compares apparently two cases: (a) when the input suffers some modifications so that the label changes and (b) when the modifications to the input can be simply described as removing characteristic features of the class being described (the deer's antlers, in this case). How are these two cases related to how sensible it is for the classifier to keep its low confidence to the original class? Is the paper stating that the classifier's confidence for the true class when making predictions over the original input should be low because it should be aware of how few changes would be required to change the semantic class? I do not think I would agree with such a premise: even though zebras and horses look very much alike, when perceiving a zebra we can be rather certain it is a zebra and not a horse, precisely because of its *very* distinctive features.
+ In general, while the experimental results are mostly consistent, the improvements are rather marginal, and so I am unsure whether they are significant.
+ My understanding of one of this work's main contributions is observing a relation between the certification of a classifier and its calibration. Because of this observation, the paper proposes to improve calibration through a mixup-inspired procedure so that certification is also improved. Is my understanding correct? If so, would not a valid baseline be other previous calibration techniques? Or is there some issue with this approach? If there is no issue, how do other calibration techniques perform towards improving certification?


Weaknesses that do not affect my rating but should be addressed if possible:
+ L6: of smoothed classifier
+ L8: it trains convex combinations? or *on* convex combinations? 
+ In Fig. 1's caption: "mixup" or "the mixup"?
+ L37: one of important
+ L57: they goes
+ L85: construct
+ L142: "their" or "its"?
+ L144: have
+ L146: "a semantically off-class samples those to be labeled as the uniform confidence"
+ L151: those
+ Fig. 2's caption is not self-contained: what the authors want to express with the figure is unclear from the caption. In particular, what is meant by "in/out-of-class translation"? I think this caption requires more explanation.
+ In L159, I do not understand what the paper means by "mixing the uniform confidence to them". What does "uniform confidence" mean, and to whom is it being "mixed"? Similarly, in L166 the paper says "direct supervision of the uniform confidence on it", which I do not understand. Could the authors please clarify what was meant by these expressions?
+ Table 1's caption: "those"
+ The tables reporting the experiments follow a protocol of adding bold-face and underline that I think is confusing: usually whenever readers see a bold-faced number (I think) they think it is the largest, which is not the case in this paper. I would urge the authors to reconsider this protocol. (For instance the results of other methods are never underlined, which can misguide th reading of results)
+ Table 1's caption: "improves" or "outperforms"?
+ Fig. 4's caption: "plots"
+ L195: trainig
+ Fig. 5's caption: "plots"
+ L230: improve
+ L240: those
+ For the experiment reported in L285, how were the adversarial examples computed? Were both restricted or both unrestricted? Or was there a mismatch in this? If there was a mismatch, I think the results are not interpretable, as the process of finding an adversary for the Gaussian model is constrained and so biased towards small values, while the one for SmoothMix is not.
+ I encourage the authors to rename this method, as there is a paper with the same name: "SmoothMix: a Simple Yet Effective Data Augmentation to Train Robust Classifiers", Lee et al., CVPR20 

limitations_and_societal_impact: Yes, they have addressed it.
review_title: null

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: The paper offers a novel training scheme—called SmoothMix—for fitting deep net classifiers with improved adversarial robustness while maintaining accuracy to the extent possible. The key idea is to combine the mixup loss with smooth classifiers. The former interpolates the input sample with its adversarial perturbation, designed to “mislead” the smoothed classifier. The effectiveness of the proposed idea was studied on MNIST, CIFAR10, and ImageNet, showing competitive performance to that of state-of-the-art adversarial defense strategies (Gaussian smoothing, stability training, adversarial smoothing, MACER, and consistency).
main_review: Originality

The idea of combining the mixup loss with adversarial smoothing is novel. Also, it is nicely motivated. First, it was demonstrated (visually) that adversarial perturbations of a smoothed classifier may have semantic changes that either transform the input to another class or removing relevant content for the current class. Second, the smoothed classifier tends to assign to adversarially-crafted examples higher confidence scores compared to the clean input.

Motivated by the above observations the authors hypothesized that miscalibration (although not defined precisely) degrades the certified robustness of smoothed classifiers. They propose to penalize over-confident predictions by interpolating the original image (and label) with its unrestricted adversarial example, assigning uniform confidence to the latter. This approach encourages the smoothed classifier to produce a uniform confidence score for adversarial samples.

Quality

The paper is well written, the exposition of the idea is clear, the experiments follow existing literature.

Main concern

While the method attains state-of-the-art performance, it is hard to judge whether the new results are significantly better than existing ones, e.g., they are only slightly better than adversarial smoothing. It will be interesting to see a detailed performance analysis, e.g., a report of class-conditional performance metrics could perhaps communicate better the advantages of the proposed method.    

Significance

Certified adversarial robustness is an active area of research that sheds light on the robustness of deep nets. The proposed method achieves state-of-the-art performance on benchmark datasets. It is likely that---at least in the short term---this new method will have a noticeable impact.
limitations_and_societal_impact: It will be a plus to explain that the SmoothMix approach has an additional hyper-parameter to tune (\alpha).
review_title: null

Meta-review: All reviewers praised the simplicity of the idea of combining MixUp and adversarial training to boost certified accuracy and its significance.
At the same time, reviewers questioned the significance of the performance boost in terms of metric used (ACR) and potentially different behaviors across datasets. Furthermore, presentation and writing were deemed improvable. 

Authors were quite responsive during the rebuttal and provided additional explanations and experimental results that answered the major concerns raised by the reviewers.
Fixing the presentation as suggested is expected for the camera-ready version. 


Decision: Accept (Poster)