Venue: NeurIPS 2021 Conference

Reviews:

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: The focus of the paper is to make contrastive pre-training (e.g. SimCLR), which has seen great success in recent years, more robust to adversarial attacks. To this end, the authors propose AdvCL, which makes the following modifications to prior contrastive methods:
1) high-frequency components of images are shown to be important for robustness and are leveraged when building the views.
2) augmenting contrastive learning with "pseudo-supervision" obtained by clustering the features further improves performance.
The authors evaluate test accuracy on CIFAR10, CIFAR100, STL-10 under adversarial perturbations (AutoAttack and standard PGD) using both linear head fine-tuning and full fine-tuning and compare against recent SOTA methods. AdvCL seems to outperform the rest mostly across the board. Ablative studies are conducted.
main_review: Overall, this paper checked all the boxes -- it was reasonably well-written, it had a solid experimental evaluation wherein the proposed method outperformed recent baselines, and ablations were conducted to shed more light on how AdvCL's inner workings. However, I have the following concerns:
1) Only 3 vision datasets were used, and in-domain results were only on 2 (CIFAR10 and CIFAR100). Furthermore, AdvCL loses to Supervised (control) on CIFAR100 standard accuracy (Table 1), yet the AdvCL entry is bolded -- why is this? This is also the case in table 2, CIFAR-100 --> STL-10. If this is indeed true, then improving adversarial robustness using the proposed method can sometimes hurt accuracy in the non-adversarial setting, which is concerning. I would like to see performance on SVHN, MNIST, and maybe a couple others.
2) Furthermore, the method is complex and begs the question -- are the gains reported really worth the complexity? It's quite feasible that many other alternatives of equal complexity and equal opportunity for hyper-parameter tuning can unlock the same gain. However, this may not be reason enough to reject the paper.
Point 1 is why I slightly lean against acceptance. If the authors elaborate on this and add a few more datasets, I would consider leaning in favor of acceptance.

limitations_and_societal_impact: Yes.
review_title: null

final_score: rating: 7: Good paper, accept
review_content: summary: This paper proposes a unified adversarial contrastive learning framework. It proposes to improve the previous methods in the following ways: 1. Besides the previous augmentation methods, it proposes to utilize the high-frequency components as data augmentation to enhance the robustness. 2. It introduces the pseudo label for the pre-training task to mitigate the task difference between pre-training and fine-tuning for improving the transferability of the pre-training network. Many experiment result verifies the effectiveness of the proposed method. The author also offers ablation experiments to demonstrate the improvement brought by each component.
main_review: Originality: This paper employs the pseudo-class prediction task for pre-training and shows it can largely improve the transferability of the pre-training model. This is novel to me. I only saw the literature employing the pre-training task as regularization during fine-tuning.

Quality: The quality is overall good. Many experiments verify the effectiveness of the proposed method. Ablation studies also show the gain comes from each component. It would be better to see its performance on different backbones. There is only resnet18 here.

Clarity: The paper is overall well written and easy to follow.

Significance: The methods proposed in this paper significantly improve the transferability of the adversarial pre-trained model. Contributing to building a universal robust pre-training model.

Other: While the method requires a $\lambda$ to balance two loss terms, the value of $\lambda$ is not discussed.

limitations_and_societal_impact: I don't see other limitations except for the points I mentioned in the main review.

No negative societal impact.
review_title: null

final_score: rating: 7: Good paper, accept
review_content: summary: The submission investigates the adversarial robustness of self-supervised pretraining using contrastive learning with the NT-Xent loss [29]. The goal is to improve robustness of the pretraining step such that a linear classifier suffices to be fine-tuned for state-of-the-art performance, and to increase robustness across tasks. The proposed design, AdvCL, extends the loss formulation to contain four designed views (Eq. 9), as well as a supervision stimulus based on K-means clustering [50]. Experimental results show that the resulting system significantly increases adversarial robustness in the self-supervised setting. The paper also contributes a visualization of qualitative results as well as an ablation study of some of the key parameters.
main_review: The submission builds upon both adversarial training [10, etc.] as well as contrastive learning [29] for its loss and min-max training formulation, and extends these in the ways described above. It is not the first work to tackle the problem of self-supervised adversarial training (the authors cite [26, 27] as relevant related work). Overall originality is medium, since the proposed approach is a novel combination of existing elements -- but this is not a problem, since the resulting system is both well-designed and reasonably well motivated.
Moving adversarial training toward less supervision (self-supervised, unsupervised) as well as making it easier to use (i.e. without full finetuning) are both of high interest for the machine learning community. The paper does not offer any significant insights on adversarial training processes itself, but overall relevancy quite high.

The paper is generally very well written and easy to understand. Its sections are also well structured: introduction and related work lead to a clear formulation of the research questions, then the proposed system design, and finally a detailed evaluation. 
 
The contributions themselves are well motivated and experimental results are overall convincing. However, I would have wished to see an ablation study on parameter \lambda; what happens when the supervision stimulus is not used at all, for example? Or how does the system perform without the proposed view selection? Separating these two contributions clearly in the experimental evaluation would have made the extent of each contribution somewhat clearer.

Overall this submission makes valuable contributions to advancing adversarial robustness in the laid out scenario and is of high enough quality to qualify for NeurIPS acceptance.
A few clarifications would make the submission even stronger:
- I thank the authors for providing footnote 1, which seems prudent. Does this footnote also imply that all presented experimental results were reproduced using either own implementations or code provided by the authors?
- The experimental results are almost all provided on CIFAR-10 or CIFAR-100, whereas experimental results on (additionally) ImageNet would increase confidence in the proposed method. I do see ImageNet results in Table 5. What was the reason for not including ImageNet experiments (or experiments on yet another large-scale, more realistic data set) in the main results? Is this connected to the scalability limitations described on line 377?
- What are the limitations described on line 377 in particular?

Minor notes:
- In lines 147 and 149, both \theta and f_theta are described as "feature encoder". I recommend describing the former as "feature encoder parameters".
- Both Equations 9 and 10 contain a minimization over \theta. I believe this is superfluous in Equation 9; otherwise it would be a double minimization. And the loss term itself should not contain the type of optimization.

There are some typos which could have been avoided by a judicious spell check:
- l. 42: 'fintuning' -> 'finetuning'
- Caption Figure 1: 'pretrainig' -> 'pretraining'
- l. 137: A citation seems to be missing.
- l. 141: 'pretrianing' -> 'pretraining'
- l. 188: 'mutli-view' -> 'multi-view'

limitations_and_societal_impact: Yes, the authors have discussed the limitations of their work briefly in the conclusions, and the societal impact (which, in the eyes of this reviewer is not even applicable to this kind of work and amounts to guesswork) in the supplementary material.
review_title: null

final_score: rating: 5: Marginally below the acceptance threshold
review_content: summary: This work analyzed the fine-tuning phase of unsupervised contrastive learning for adversarial robustness. Furthermore, they proposed AdvCL which is more robust than other unsupervised adversarial contrastive learning methods. They demonstrated that the model trained with this method shows discriminative and interpretable representations, flat loss landscape, and better transferability to other tasks.
main_review: In this work, the objective is to address the issue of forgetting adversarial robustness while training the classifier with supervised learning after training feature encoder with unsupervised learning. They pointed out the different results between standard linear fine-tuning and adversarial full fine-tuning. However, the connection between the issue and the proposed method (AdvCL) is ambiguous. Only clusterfit seems to be related to the issue, but it is intuitive. Additionally, the method showed better performance than other methods when combined with standard linear fine-tuning, but showed similar performance when combined with adversarial full fine-tuning. They focused on the efficiency of standard linear fine-tuning and showed the higher performance of proposed method when considering efficiency, but it seems to be not efficient to calculate the cross-entropy loss perturbation and/or the contrastive loss perturbation respectively when using clusterfit.

The main adversarial contrastive loss (equation (7)) is suitable to make one adversarial perturbation using several augmentations, and HFC is the proper example. However, the connection issue still exists and only clusterfit seems proper. But, if this is included the proposed method (AdvCL + clusterfit), the method should show better performance than other method with clusterfit (RoCL + clutserfit, ACL + clusterfit, …). Additionally, also for the HFC, it should show better performance than other method with HFC (inserting HFC as augmentation or into positive set).

The discussions about the results are inadequate. Specifically, the discussions about the results of qualitative results and the ablation studies are necessary. It is required to discuss at least roughly what each result means and why.

Minor issues

(1) Because the obfuscated gradients issues can be checked using the performance of AutoAttack, there is no need to put the sanity check result on the main paper.

(2) In the method section, equation (4), it is confused that the input pair of similarity function of the denominator and the numerator. The inputs of the numerator may be $(\tau_1(x_1), \tau_2(x_1))$ and $(\tau_2(x_1), \tau_1(x_1))$, and the inputs of the denominator may be $(\tau_1(x_1), \tau_1(x_2))$, $(\tau_1(x_1), \tau_1(x_3))$,…,$(\tau_1(x_1), \tau_2(x_1))$, $(\tau_1(x_1), \tau_2(x_2))$ ,…, but cannot be sure.

(3) There is no the transferability results for cifar10 ↔ cifar100 as in the RoCL paper.

limitations_and_societal_impact: There is no limitation paragraph and exists a societal impact paragraph in Appendix. No negative societal impact. Several potential broader impact such as lightweight and efficient fine-tuning.
review_title: null

Meta-review: With the increasing interest in both adversarial robustness and contrastive self-supervised learning, the paper studies a highly relevant problem at their intersection: how contrastive learning (CL) can be made to learn representations so that robustness is preserved when fine-tuning a linear classification head on top of these representations.

This indeed seems to be a setting that hasn't been explored before in the literature; however, to my knowledge such a setting in which we use standard linear fine-tuning is not commonly used (previous works such as SimCLR only use it to prove a point about the power of the learned representations from CL alone). Thus, I am willing to grant it here as well, as this provides us more insight into the robustness of the CL learned representations, and although one reviewer pointed out concerns about the higher runtime of the proposed procedure, this point may not be so important in this context. Also, as one reviewer pointed out, this may help encourage more light-weight fine-tuning if indeed the robustness can be transferred across tasks making adversarial robustness easier to use.

Moreover, the paper also provides a thorough experimental evaluation not only on the standard linear fine-tuning but also on the adversarially robust fine-tuning, shows that the proposed method (which appears to be an extension of ACL), is superior to previous methods with ablation studies.
Decision: Accept (Poster)