Venue: ICLR 2021 Conference

Reviews:

final_score: rating: 4: Ok but not good enough - rejection
review_content: 
This paper proposes some modifications to BYOL (Bootstrap Your Own Latents) in an attempt to address adversarial robustness in low-label high data regimes. The paper is well written and very easy to follow. Overall, the idea is clear and well presented. 

Although the author's claim their approach to be novel, as is, the main contribution of the paper is adding adversarial training to BYOL, which already does not require labels. 

The authors also claim that their approach is better than pseudo-labeling training for downstream tasks. Yet, this is not always the case. The authors also mention that their approach reaches optimal performance with 5% of labels (and seems to deteriorate with more labelled data). There is no analysis on why this is the case. 

There is also a lack of comparison against contemporary SOTA self-supervised approaches.


review_title: Application based approach, claims not very well supported. 

final_score: rating: 6: Marginally above acceptance threshold
review_content: In this paper, adversarial self-supervised learning is proposed to render robust data representations for down-stream fine-tuning tasks. The core idea is to integrate BYOL with adversarial training. The paper is well written in general. However, I do have several concerns about this submission.  

1. Robust self-supervised pre-training + fine-tuning has been studied in two recent works at least.

[1] Hendrycks, Dan, et al. "Using self-supervised learning can improve model robustness and uncertainty." Advances in Neural Information Processing Systems. 2019.

[2] Chen, Tianlong, et al. "Adversarial Robustness: From Self-Supervised Pre-Training to Fine-Tuning." Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2020.

The comparison with [1-2] is recommended. 

2. Unclear algorithm implementation. 

a) In Figure 2, why is additional prediction head $q(\cdot, \theta)$ in BYOL and BYORL needed? Please clarify it. 

b) It is not clear why a very large eps used in $\ell_2$ robust training, e.g., eps = 128/255. 

c) In Table 2, it is not clear why $\ell_\infty$ robust pre-training results are missing?

3. Not convinced transfer results on unseen tasks. 

In the paper, the authors claimed that "For both representations, we train a robust linear model using adversarial training (see subsection 3.1) with a different label availability on STL-10 and CIFAR-100 against `$\ell_\infty$ and `2 norm-bounded perturbations."

Thus, I supposed that during fine-tuning, the robust representation is frozen, and only the linear classifier is adversarially trained, correct?

If so, in Table 2, the results on Robust (BYORL) + Non-robust on CIFAR-10 seem very strong. It indicated that the standard partial fine-tuning is able to preserve robustness from self-supervised robust representation. However, [2] founds a different conclusion. Thus, it is important to provide additional explanations and comparisons for the achieved results. 

I also suggest the authors check if the proposed defense yields obfuscated gradients, e.g., having a plot of robustness versus different attack strength eps during evaluation. 

Lastly, if the pre-training task is conducted over CIFAR-10, can the results be further improved?


review_title: Review report 

final_score: rating: 7: Good paper, accept
review_content: This paper introduces a new algorithm for learning adversarially robust models in the semi-supervised setting, where a small amount of labeled data is available together with a sizeable unlabeled dataset. The proposed approach BYORL adapts an existing self-supervised learning method BYOL by introducing a new adversarial augmentation technique based on maximizing the cosine similarity between representations. BYORL is evaluated on CIFAR-10 and compared against a recent pseudo-labelling based approach UAT-FT for the semi-supervised setting, and is shown to outperform UAT-FT in terms of robust accuracy under $\ell_2$ and $\ell_\infty$ attacks under the low-labelled data regime. The representations learnt by BYORL are also shown to be better than that of UAT-FT when transferred to other datasets. Finally, robust representations are shown to be more important than learning a robust linear classifier on top.

Strengths:
- Significant improvements in robust accuracy in the low-label regime
- Interesting observations regarding transferability of robust representations and importance of final robust linear classifier
- Paper is generally clear and well-written

Weaknesses:
- RoCL as introduced by "Adversarial Self-Supervised Contrastive Learning", NIPS 2020 can also be applied to the semi-supervised setting, which somewhat compromises novelty. Ideally this method should be included in the experimental evaluations. This paper was cited but not discussed in the context of related work, but should be.
- Label budgets claimed do not include sizeable validation set used for early stopping, which can sometimes exceed the label budget (e.g. result with 500 labeled images also uses validation set of 1024 examples).
- Experimental evaluation could be strengthened in a few ways:
  * Clean accuracy comparison for Fig 3/4 - does UAT-FT do better in terms of clean accuracy?
  * It would be informative to include the robust accuracy of a model trained directly on STL-10 and CIFAR-100 in Table 2 to show how good the transferred representations are.
  * Experimental results seem to be from a single run; multiple runs would give an idea of the variance.

Overall, the robust accuracy improvements achieved in the low-label regime by BYORL are significant and the method is somewhat novel. The paper could be further strengthened by improving the experimental evaluation as described above. The highly related work RoCL should be discussed and ideally compared with.

Other comments:
- The paragraph after Eq 8 was confusing to me. What is meant by symmetrizing the loss? What is the final loss used to train the model after symmetrization? And why does the argument about batch-norm statistics not apply to the online network as well if the loss is symmetrized (which I took to mean that both models would be separately attacked and the losses added up)?
- Some analysis into the representations learnt would be helpful to give some insight into why the method works.
- Why does the method work better under $\ell_2$ attacks as compared to $\ell_\infty$ attacks?
- What is the effect of different transformations on robustness? Are the transformations important for robustness different from those important for clean classification?

*** Post Response Comments ***
I thank the authors for addressing the points raised. I am raising my score accordingly to 7.

Nit: The y-axis labels on Figure 7 and 8 should probably say "Clean accuracy" instead of "Robust test accuracy".
review_title: New method, good improvments in low-label regime

final_score: rating: 7: Good paper, accept
review_content: ##########################################################################
Summary: 

The paper proposes a new self-supervised technique, Bootstrap Your Own Robust Latents (BYORL), based on an existing technique, BYOL. BYORL proposes to provide adversarially robust representations for low-label regimes. The paper claims that BYORL achieves state-of-the-art performance on CIFAR-10 even with data that is labeled as low as 1%. In fact, the authors highlight that the representations resulted from BYORL avoid the explicit training for adversarial robustness, because they are already robust.

##########################################################################

Reasons for score: 
 
Overall, the paper in its current form is above the acceptance threshold. The proposed idea looks very encouraging. Provided the authors address some of the concerns the proposed method has a significant potential to the self-supervised adversarial robustness. If we look at the low-label regime alone, probably the paper has good contributions, that on ImageNet is not entirely convincing. If the authors convince me on that, happy to increase the score to clearly accept.  
##########################################################################


Pros:

1. Overall, the paper is well written except for a few minor grammatical errors, easy to follow and understandable. 

2. The paper discusses the existing literature and positions the proposed approach with respect to the state-of-the-art. The proposed method is definitely a decent contribution towards the field.

3. While robustness evaluations are good, the transfer to unseen and without adversarial training are much more encouraging.


Cons:

1. In abstract “... pioneered by four separate and simultaneous work in 2019,” should be “... pioneered by four separate and simultaneous works in 2019,”

2. “Since Madry et al. (2017), various modification to their ...” should be “Since Madry et al. (2017), various modifications to their ...”

3. As the % of labeled training data increases, there is not a significant improvement in robust test accuracy for CIFAR-10, that increases for CIFAR-100. Why is that happening for CIFAR-10? 

4. On CIFAR-100, for anything over 10% labeled training data the other methods are still the state-of-the-art, the proposed method does not perform well. It again probably is due to the above observation of robust accuracy not increasing with a steep slope like the other methods. 

5. What is robust accuracy, Zhang et al. 2019 or Uesato et al 2019 or others have it defined, nevertheless, please say in a sentence what it is.

6. On the ImageNet, the hypothesis of BYORL to get better as the labeled data increases is hard to buy given what we saw on CIFAR-10 and CIFAR-100. For those two, BYORL starts better and the state-of-the-art methods either reach BORL or outperform it. In ImageNet case, BYORL is outperformed with 1% itself, unless supported with empirical evidence, it is hard to believe the above statement of improvement.

7. Overall, the empirical results are satisfactory but not entirely convincing.

review_title: Review: SELF-SUPERVISED ADVERSARIAL ROBUSTNESS FOR THE LOW-LABEL, HIGH-DATA REGIME

Meta-review: The paper considers the use of adversarial self-supervised learning to render robust data representations for various tasks, in particular to integrate the Bootstrap Your Own Robust Latents (BYOL) with adversarial training, where a small amount of labeled data is available together with a sizable unlabeled dataset.  Especially the low-data regime is of interest.  It extends a previous method with a new adversarial augmentation technique, it is compared against several methods, and the robust representations are shown to be useful more generally.  There were some confusing presentations and questions that were resolved in a detailed discussion with the reviewers.
Decision: Accept (Poster)