Venue: ICLR 2021 Conference

Reviews:

final_score: rating: 7: Good paper, accept
review_content: The authors improve the fine-tuning of a data representation allowing
resistance and recognition of adversarial or corrupt inputs.  Their method,
URRL, uses an auxiliary unsupervised task and robust data augementation to
improve performance for both clean and altered inputs.  A key addition is that,
while fine-tuning on the auxiliary (downstream) task, performance on the
original data is maintained by favoring small differences between original
robust representation and the fine-tuned representation.

The work is well presented, and the main significance is in providing a fast
way to partially gain some of the robustness available in much more difficult
adversarial training.  This might be of significance in applications like
online learning, where only fast fine-tuning is practical.

Their approach limits how much fine-tuning is allowed to deviate from the
original, robust representation.  It is a *fast* way to partially recover some 
of the benefits of a much more expensive adversarial training.  The problem
they address is important.  For example, observations in related works often 
note a slight decrease in performance on the original supervised task during
later fine-tuning.  The issue to resolve is how to nicely limit degradation,
even in face of aggressive or long-term fine-tuning.

To maintain robustness during fine-tuning, one (expensive) option is to
adversarially fine-tune the downstream task.  The authors propose a quicker
approach: a loss function maintaining similarity between fine-tuned and
original representation (e.g. low fine-tuning distortion of dot-product
similarity).

The experiments nicely evaluate robustness with a number of relevant metrics,
using ImageNet with many corruption types.  Results are presented as improvements
to the MoCo v2 codebase.  The use of a linear classifier stage seemed reasonable to me.

Firstly they address what sets of data augmentations are most effective for
images, using a mechanism that probabilistically switches between augmentation
types for supervised and unsupervised tasks.

Secondly, the more interesting results with similarity loss showed very good (usually best)
adversarial performance.  Accuracy on clean data was degraded by ~ 10%, still less than the
degradation caused by adversarial training.  Similarity loss can be applied to other existing
techniques (like their MoCo baseline).

Q: Does the similarity measure loss still make sense if fine-tuning uses
data sampled from a new/changing input distribution?  It might
(see also "Test-Time Training with Self-Supervision
for Generalization under Distribution Shifts", Sun et al.)

---

The authors' comments strengthen the argument for the method.   I (perhaps liberally) conceptualize
their approach as one of tethering "close to a rotation" but still don't have a simple understanding of
why/when this should be better than tethering "close to original parameters". My original rating is unchanged.
review_title: A new similarity loss to fine-tune representations using auxiliary tasks.

final_score: rating: 5: Marginally below acceptance threshold
review_content: ## Paper summary
This paper proposes *Unsupervised Robust Representation Learning* (URRL), a framework that combines several data augmentation schemes and a similarity-based loss. The goal is to improve the robustness of visual representations to image perturbations. A further goal is to maintain the robustness properties of pre-trained representations after fine-tuning the network to downstream tasks.

The proposed method consist of two components: A scheme for sampling combinations of image augmentations, and a loss that is applied during transfer learning and penalizes deviations of the pairwise similarity between datapoint representations from the similarities before fine-tuning.

The method is evaluated on clean data performance and adversarial robustness, as well as calibration error.

## Arguments for acceptance
1. The paper is well written and clear. 
2. The proposed method improves on MoCo-v2 and Augmix in ImageNet accuracy, ImageNet-C corruption error, RMS calibration error and adversarial robustness.

## Arguments against acceptance
3. The proposed augmentation approach appears to be very similar to the methods used in MoCo, AugMix, and related work. Section 2.1 of the paper argues that these methods represent two different families of augmentation ($\mathcal{T}_\text{rep}$ and $\mathcal{T}_\text{rob}$). The paper suggests that augmentations need to be sampled from both of these families in a particular way (Eq. 2) to overcome the issue that one of the families affects the color histogram too much. This approach is only a small and heuristic variant of existing methods. It has been known since long before MoCo and AugMix that image augmentations improve unsupervised representation learning and robustness. Formal arguments for the distinction between $\mathcal{T}_\text{rep}$ and $\mathcal{T}_\text{rob}$ are not given. I am not sure whether the proposed variant is original enough for publication at ICLR. Further, the performance improvement of URRL over MoCov2 is small, less than one percentage point for most metrics.

4. The proposed similarity regularization leads to more significant improvements in fine-tuning performance. However, this method is not compared against baselines. Regularization approaches for improving transfer performance have been proposed before, e.g. see [Li et al., 2018](https://arxiv.org/pdf/1802.01483.pdf). These should be compared to the proposed method. In particular, well-tuned weight decay towards the frozen pre-trained weights ($L^2$-$SP$ in Li et al.) is an essential baseline. I suspect that the proposed similarity regularization may be functionally equivalent to decaying weights towards their initial values.

## Conclusion
While the paper is well organized and clear, in its current form it does not meet the originality and significance standards of ICLR and lacks crucial baselines. Suggestions for improvement: 

5. Provide a more formal argument for the proposed augmentation framework to show that it is a significant conceptual advance over the current knowledge that augmentations are important.
6. Provide a more formal motivation for the similarity regularization, including a formal comparison to $L^2$-$SP$.
7. Compare the similarity regularization to a well-tuned $L^2$-$SP$-decay baseline.

## Update after rebuttal:
The authors provided convincing evidence that the proposed similarity regularization performs better with regards to robustness than $L^2$-$SP$. I adjusted my rating accordingly. I still think the contributions are borderline because the proposed regularization is not backed up by theoretical arguments and the augmentation approach is incremental.
review_title: Minimal and insufficiently motivated changes to existing methods

final_score: rating: 7: Good paper, accept
review_content: This paper uses a different data augmentation (AugMix) scheme to improve self-supervised representation learning. It improves accuracy and (corruption and adversarial) robustness by a sufficiently interesting amount. The paper's presentation is clear, but the paper could be more thorough. Since the technique is simple and general, it could easily be broadly applicable to the burgeoning area of self-supervised learning.

Other pros:
_On the surprising similarities between supervised and self-supervised models_ shows that most techniques for self-supervised learning don't improve robustness at all, so self-supervised learning produces representations that have limited use downstream.
This work counteracts this limitation by also assessing the robustness of self-supervised representations, and the technique they propose improves robustness. They improve robustness with data augmentation, a volatile ingredient in self-supervised learning. SimCLR reminds us that the choice of image modifications is crucial and must be carefully selected, so it was not obvious a priori that their technique should help.

Other cons:
The paper would be stronger if it showed their augmentation scheme helping with more than MoCo. That would demonstrate generality. I'll boost my score if it improves another self-supervised learning technique.
The paper could show results with another robustness benchmark, such as ImageNet-R. This would make a stronger case for robustness. This would be quick to add.
URRL-SR uses embedding distances, but this seems like logit pairing. Logit pairing can lead to an overestimate of adversarial robustness. Please include results with an adversarial attack with restarts. Also state whether you're using l2 or l_infty attacks, rather than having refer to Ilyas et al.
The paper has one only figure, and Table 2's formatting is slightly unintuitive.

Update: I am happy with the changes and am keeping my score.
review_title: A Work on Making Self-Supervised Learning Endow Robustness

Meta-review: The authors propose a framework which combines pretext tasks and data augmentation schemes with the goal of improving robustness of image representations. The authors show that this approach empirically can lead to increased accuracy on both corrupted and uncorrupted data simultaneously. Furthermore, the authors propose a regularization procedure which can be used to maintain a robust representation during transfer to arbitrary downstream tasks. 

The studied problem is significant and highly relevant to the ICLR community. The reviewers agreed that the work is timely, and appreciated the clarity of exposition. At the same time, the reviewers remained in disagreement in terms of novelty and significance of the results -- the proposed method is seen as a clear incremental application of existing techniques in the self-supervised setting. The authors argue that it was not clear that such augmentations would improve the robustness as well as accuracy, but these methods were developed and optimised to improve robustness. In fact, in [1] the authors conclude that “...today's supervised and self-supervised training objectives end up being surprisingly similar” as well as point out that SimCLR is more robust than competing self-supervised methods. Hence, establishing that there is indeed some empirical benefit is a step in the right direction, but not sufficient to meet the bar of acceptance. Furthermore, given the recent trend of scaling-up existing approaches, in particular in terms of the neural architectures and the batch sizes, the computational costs of the proposed regulariser in Eq (4) coupled with the additional hyperparameter to optimize make the approach less practical and general than claimed. In addition, the reviewers pointed out the lack of comparison to a proper baseline, as well as the issue with hyperparameter selection for the baseline after the author response. Finally, given access to additional data augmentations and one more hyperparameter to tune the results should substantially outperform the baselines.

For the reasons outlined above and the incremental nature of the work, I will recommend rejection. That being said, this was a borderline case, and I urge the authors to carefully revise the manuscript with the received feedback.

[1] https://arxiv.org/abs/2010.08377

Decision: Reject