Transferable Unsupervised Robust Representation Learning
Authors: De-An Huang
Zhiding Yu
Anima Anandkumar
Venue:  (2021)
Keywords: unsupervised representation learning
robustness
transfer learning

Abstract: Robustness is an important, and yet, under-explored aspect of unsupervised representation learning, which has seen a lot of recent developments. In this work, we address this gap by developing a novel framework: Unsupervised Robust Representation Learning (URRL), which combines unsupervised representation learning's pretext task and robust supervised learning (e.g., AugMix).  Moreover, it is commonly assumed that there needs to be a trade-off between natural accuracy (on clean data) and robust accuracy (on corrupted data). We upend this view and show that URRL improves both the natural accuracy of unsupervised representation learning and its robustness to corruptions and adversarial noise. A further challenge is that the robustness of a representation might not be preserved in the transfer learning process after fine-tuning on downstream tasks. We develop transferable robustness by proposing a task-agnostic similarity regularization during the fine-tuning process. We show that this improves the robustness of the resulting model without the need for any adversarial training or further data augmentation during fine-tuning.

Under review as a conference paper at ICLR 2021
TRANSFERABLE UNSUPERVISED ROBUST REPRESEN- TATION LEARNING
Anonymous authors Paper under double-blind review
ABSTRACT
Robustness is important, and yet, under-explored aspect of unsupervised repre- sentation learning, which has seen a lot of recent developments. In this work, we address this gap by developing novel framework: Unsupervised Robust Repre- sentation Learning (URRL), which combines unsupervised representation learn- ing's pretext task and robust supervised learning (e.g., AugMix). Moreover, it is commonly assumed that there needs to be trade-off between natural accuracy (on clean data) and robust accuracy (on corrupted data). We upend this view and show that URRL. improves both the natural accuracy of unsupervised represen- tation learning and its robustness to corruptions and adversarial noise. A further challenge is that the robustness of representation might not be preserved in the transfer learning process after fine-tuning on downstream tasks. We develop trans- ferable robustness by proposing a task-agnostic similarity regularization during the fine-tuning process. We show that this improves the robustness of the resulting model without the need for any adversarial training or further data augmentation during fine-tuning.
INTRODUCTION
Recently, there has been an increasing interest in unsupervised visual representation learning. where the goal is to learn effective representations of images without using human annotations (Bachman al. 2019 Chen et al., [2000b] He et al., 2020; Misra & Maaten 2020). In this work, we consider relatively under-explored aspect: the robusiness of these representations. Here, we use robustness to encompass the representation's 's resilience to common corruptions (Hendryeks Dietterich] 2019)) adversarial examples (Madry et al., 20188, and its ability to measure the uncertainty of its output in the face of such perturbations (Kumar et al. 2019).
More specifically, we develop new learning methods for transferable robustness of these representa- tions, which improves and preserves the representation' 's robustness after fine-tuning on downstream tasks. While recent works have only evaluated the virtue of representation by its accuracy clean data after transfer learning and fine-tuning, we argue that transferable robustness should also be an integral part of good representation. Recently, |ezaei & Liu (2020) have shown that downstream task models are vuinerable to adversarial attacks that are based solely on the pre-trained represen- tation before fine-tuning. This highlights the importance of transferable robustness, without which downstream task models are vulnerable and lack generalization.
Moreover, commonly assumed that there needs be trade-off between natural accuracy (on clean data) and robust accuracy (on corrupted data) (Zhang et al., 2019). While there have been works showing that such trade-offis is not necessary for supervised learning (Yang et al.) 2020), is unclear how the principles they leverage is applicable to unsupervixed representation learning.
Summary of Contributions: (1) We show that the accuracy
trade-off is not necessary for unsupervised representation learning. (2) We develop new representation learning framework: Unsupervised Robust Representation Learning (URRL), which uses a novel mix of representation learning pretext task and robust supervised learning (AugMix, Hendrycks et al.) (2020b)). (3) In addition, we propose a task-agnostic similarity regularization that further improves the robustness of downstream tasks without the need for any adversarial training additional data augmentation. (4) We introduce an evaluation framework for the transfer robustness a representation, which includes its resilience to corruptions, adversarial robustness, and uncertainty calibration. (5) We show that

---

Under review as a conference paper at ICLR 2021
%/
f
standard
SLf Si
Original
AugMix
fe
Image
Ltask
(a) Unsupervised Robust Representation Learning
(b) Similarity-bused Regularization
Figure I: (a) We show that by integrating AugMix with pretext tasks of state-of-the oof-the-art art approaches. URRL leads improvement of both clean accuracy and robustness and does not have the accuracy- robustness tradeoff. (b) We propose task-agnostic similarity regularization, where the pairwise similarity ši, of So is regularized by that of the frozen representation I* during fine-tuning by Clask- This encourages fe capture f* robustness without the need to train corrupted data.
URRL improves both the clean accuracy and robustness of state-of-the-art representation learning under linear evaluation, and our full framework, URRL with Similarity Regularization (URRL-SR), further improves its robust accuracy by over 10% on 12 datasets.
The proposed URRL is shown in Figure a). Ja). works, data augmentation has played an important role in both supervised learning robustness and unsupervised representation learning Yun et al. 2019: Chen et al. 2020b). URRL randomly sam- ples a pretext task of unsupervised representation learning and a robust data augmentation (AugMix, Hendryyck al. (2020b)), and then optimizes them both under contrastive learning framework. We show that URRL improves both the clean accuracy (on the original images without corruptions) and the robustness of the learned representation under the standard linear evaluation protocol.
Our approach is motivated by the fact that in previous
Nevertheless, using the representation under linear evaluation limits the performance of transfer learning, and ultimately the usefulness of representation learning downstream tasks. It is thus im- portant to evaluate the representations' transferable robustness after fine-tuning the entire network the downstream task. The biggest challenge to preserve the inherited robustness of the rep- resentations: an aggressive fine-tuning schedule on a small dataset can significantly overwrite the robustness of the learned representations.
We address this challenge by proposing a task-agnostic similarity regularization that is applicable for fine-tuning of the learned representation on any downstream task. show that this improves the robustness of the fine-tuned model. The proposed regularization is shown in Fig fo is initialized by the robust representation f learned in Figure [). and now being fine-tuned by the downstream task loss Ltosk. f is robust because it is trained augmentation of corrupted versions of an image.
We hypothesize that this form of robustness is not just about how f* represents image individ- ually, but also about how f" measures the similarities between images. This implies that we can preserve the robustness of l" by maintaining its similarity measures between images. We achieve this by regularizing the similarity matrix of a representation during fine-tuning to be similar to that of the original representation before fine-tuning. This allows the downstream task model to bet- ter utilize the robustness of the learned representation, while not capping the performance, which would be the case if the representation is fixed. Our full approach, URRL with similarity regular- ization (URRL-SR), further improves URRL's 's robust accuracy by over 10% on 12 datasets without any adversarial training or further data augmentation.
METHOD
In this work, we advocate the transferable robustness of unsupervised representation learning, which measures the robustness of representation after fine-tuning on downstream tasks. The two chal- lenges are: (i) It commonly assumed that there is an accuracy-robustnee trade-off. and the promo- tion of transferable robustness could potentially set back the performance gain from unsupervised
2

---

Under review as a conference paper at ICLR 2021
representation learning. (ii) the robustness of a representation could be changed drastically during the fine-tuning process, and thus make challenging to preserve the robustness after fin-tuning the whole network. In Section we address the first challenge by proposing URRL, which combines the advantages of pretext tasks from unsupervised representation learning with robust augmenta- tions to improve not only the robustness but also the clean accuracy. In Section 2.2 we address the second challenge by proposing a task-agnostic similarity regularization that preserves the robustness of learned representations, while not capping the performance of transfer learning.
2.1 UNSUPERVISED ROBUST REPRESENTATION LEARNING
In this section, we discuss the proposed Unsupervised Robust Representation Learning (URRL), which combines the advantages of representation learning pretext task and robust augmentation. We observe that data augmentation plays a central role in the recent literature of both unsupervised representation learning and supervised learning robustness.
Contrastive learning has become a predominant design for representation learning (Chen et al. 2020b). Given an unlabeled dataset { of images 2;, the goal is to learn the representation of each image 2; So, (az,) by optimizing the parameters 0n of the encoding function. The role of data augmentation is to randomly create two correlated views of the same image. Let Trep be a family of data augmentation. We can then sample two data augmentation functions t t2 ~ Trep, and apply them to an image 2; to obtain a positive pair Tig = t1 (z:) and TK++ = tz(2))) The goal is then to train the encoder fe, such that it still encodes the two augmented images T.g and xz++ similarly despite the random augmentations to the images. In this case, the representation can learn to ignore information that is not task-relevant (introduced by the random augmentation), while still capturing the task-relevant information in the images. One can achieve this through the contrastive loss:
exp(zg 2k+/T)
Lcontra = - log exp(Zy #x+/T)+ + exp(z, -k-/T)²
(1)
where J is the softmax temperature, and žy- are the representations of negative samples. The negative samples are often randomly augmented versions of images in the same batch æj. j * The family of data augmentation Trep plays crucial role here as it would determine what the encoder fe, learns to ignore and preserve in the representation of an image.
On the other hand, data augmentation improves the robustness of supervised learning by effectively expanding the training dataset without human annotation. Given a dataset {z;, y/s of images æ; and the corresponding labels yi. one can apply data augmentation Tras to obtain an augmented dataset (Trro(():) }: where (a;) t(xr),, f ~ Trob- Trob improves the robustness of the model by indicating that the augmented image Tros (a:)) still has the same label y/ as the original image æ;- Recent works have shown that by mixing various types of perturbations, the learned classifier has the potential to be robust to novel perturbations that are not used in training (Hendrycks et al. 2020b). Trab plays a crucial role in which types of perturbation the learned model would be robust to.
Given the importance of data augmentation in both of these aspects, reasonable direction for un- supervised robust representation learning is to design a new family of data augmentation functions that can be used in Eq. (1 to learn robust representation without human annotation. We observe that the goal of Trep and Trob are quite similar at the high-level: In robustness literature, the aim of data augmentation is increase the diversity of training data as much as possible without drifting off the data manifold (Hendrycks et al.] 2020b). On the other hand, recent work hypothesizes that the best data augmentations for unsupervised representation learning are ones that "share the minimal information necessary to perform well at the downstream task" (Tian et al. 2020). If we take the downstream task to be classification and assume that the reason for not drifting off the data manifold is also optimizing for classification performance, then the aims of these two augmentations are actu- ally quite similar: both are trying to preserve information that is enough to do well on classification, while increasing perturbation (and decreasing shared information) as much possible. As we will show in the experiments, this is indeed the case: a representation learned with Trep is already more robust than ones learned by supervised pre-training: At the same time, Trub can already serve as an effective pretext task for unsupervised representation learning. It could seem that using Trab as a pretext would directly lead to unsupervised representation learning with transferable robustness.
However, one missing piece of naively using Trab as a pretext for representation learning is the augmentation that focuses on color distortion. As observed and shown by Chen et al. (2020b).
3

---

Under review as a conference paper at ICLR 2021
color distortion plays critical role in the success of unsupervised representation learning because patches from the same image could share a similar color distribution, and without color distortion augmentation, the encoder could learn to exploit shortcut to minimize Eq. (I) by focusing on the color histogram of an image. Drastic color distortion is often discouraged for Trab because "histogram color augmentation
change the image's class" (Hendrycks al. (020b).
Therefore, we propose a straightforward solution to the problem by incorporating color histogram distortion into Trob. The proposed data augmentation family Turri for URRL does not sample the data augmentation function i directly. Instead, we use a two-stage process:
TTrb with probability p:
T (z;)) t(xx), T~ T
(2)
(Tee with probability 1 1-p,
where we first sample the family of data augmentation T, and then sample the actual augmentation from T. Here, the sampled 'T could be either Trob or Trep. With the inclusion of Trep- Turri prevents the encoder f from exploiting the color histogram. On the other hand, the use of Trob in addition to Trep further strengthens the robustness of the learned representation. We will show that this simple approach is able to outperform more sophisticated approaches to modify Trob- this work, we use the pretext task of Chen et al. (2020d) Trep and AugMix (Hendrycks et al. 2020b as Trub
2.2 SIMILARITY REGULARIZATIO TOWARDS PRESERVED ROBUSTNESS
Now we have discussed the proposed Unsupervised Robust Representation Learning (URRL). The next question is how we use and evaluate this learned representation. Once the representation encoder So, is learned, one either uses it to extract features of images for downstream tasks or perform transfer learning further optimize and fine-tune fo, downstream tasks.
We consider a basic setting of transfer learning, where we have a learned representation fo, and a function goz (-) that can be applied to the extracted feature representation and transform downstream task output. Given a dataset for the downstream task {Ø,, Øj J. where ē, is the input image, and y, is the target output, we assume that the downstream task would perform transfer learning by minimizing the following loss function:
Lrask =
úü = -000,,th (iz)),
(3)
where ((.,) is a loss function between the target ÿJ and the predicted output 인ز- One can choose to minimize Lrask by optimizing only 02 or both 0, and 02. The downside of the first option is that the performance might be limited because fe, could potentially ignore information that is useful for the downstream task. A more common approach is thus to also optimize fe, with Ltask. This has shown to be a successful approach for a variety of downstream tasks, whether So, is learned with supervised pre-training or unsupervised representation learning.
Nevertheless, one missing aspect is the robustness of the resulting model after optimizing fo, and go, with Lrask- As the robustness is not part of the objective, fe, 's robustness could be drastically changed after the optimization, and there is no guarantee that the resulting model would still be robust even if we start with Se, that is robust. One option for the downstream task, for example, by adversarial training (Madry et al. (018)) or even by using robust augmentations again on the downstream task. However, this is more consuming (Shafahi et al. 2019) and might even need to be task-specific (Zhang & Wang 2019).
design robust optimization methods
To this end, we propose a simple yet effective agno-sgi regularization to supplement the trans- fer learning loss in Eq. ( and preserve the robustness of the learned feature representation. The proposed regularization does not significantly increase the training computation and can be eas- ily combined with any form of Lrask. This allows to better preserve the robustness of learned representations during fine-tume tuninunnng and improve the robustness of downstream task models.
The robustness of fe, comes from its ability to represent an image similarly despite the per- turbations. Ccontra in Eq. (I) encourages fe, to represent augmented (and potentially corrupted) versions of image closer than that of other images. This property is beneficial to downstream task robustness because if fe, (ā;) is less affected by a perturbation, then it is more likely that Ú 9a, (fe, (i,)) can be robust to the perturbation.
We hypothesize that this form of robustness is not just about how fe, represents an image individ- ually, but also about how fo, measures the similarities between images. This implies that we can
4

---

Under review as a conference paper at ICLR 2021
Table 1: Linear evaluation results of transferable robustness. URRL combines the advantages of AugMix and MoCo v2 to improve on all of the proposed linear evaluation metrics.
MoCo v2
AugMix
AugMix (AllOp)
AugMix (MoCo)
URRL
ImageNet top-1
67.59
61.45
61.97
64.41
68.47
ImageNet-CC mCE
88.21
94.47
92.34
90.18
84.41
Calibration Error
15.30
16.17
15.44
15.31
14.64
preserve the robustness of fo, by maintaining its similarity measures between images. Therefore, we propose similarity matrix regularization to preserve the robustness of fa, during transfer learning:
Luim || S* S.g so, (a.)) fo, (i)); Si f (
where (*)-) is the frozen version of fo, before fine-tuning. There are two advantages of this reg- ularization: First, we do not significantly increase the computational requirement, since fo, (#:) is already needed, and (0t) is just single forward pass of the dataset. Second, it is task agnostic: only the representation encoder fe, and input data Φ; are needed. The regularization is not affected by any of the task-specific terms in Eq. (3) (ύ), (, go, (-)). Ideally one would want to make sure fo, still preserves the similarities on the augmented data used to pre-train the representation. However, we observe that simply using the downstream task input data is sufficient, while being
much more efficient. practice, we construct S and S using the images the same batch.
3 EXPERIMENTS
We have discussed the proposed URRL, and how learning using similarity regularization. Now we aim to evaluate both its performance and trans- ferable robustness. We evaluate the fine-tuned model from three aspects: (i) resilience to common corruptions, (ii) adversarial examples, and (iii) its ability to measure the uncertainty of its output in the face of such perturbations. This provides holistic view of representation's 's transferable robustness.
preserve its robustness during transfer
3.1 ROBUSTNESS WITH LINEAR EVALUATION PROTOCOL.
Experimental setup. Linear evaluation has been the most popular way evaluate unsupervised representations learning. In this setting, the representation is not changed, and a linear classifier
trained top of the representation with the full ImageNet (Deng et al., 2009). Once trained, this classifier is evaluated on the clean ImageNet without any perturbation. We further evaluate this classifier trained on clean data directly on the perturbed dataset ImageNet-C, in which images from ImageNet are perturbed by 15 types of algorithmically generated corruptions from noise, blur, weather, and digital categories (Hendrycks Dietterich 2019).
Metrics. On the clean ImageNet, we report the top-1 accuracy following previous works. For ImageNet-C, the standard metric is mean Corruption Error (mCE), where we report the error rate normalized by the error rate of AlexNet, and average across the 15 types of corruption. The reason for this normalization that different types of corruption have different difficulties. and we do not want the evaluation to be skewed towards small set of corruptions. In addition to mCE, we further evaluate the RMS Calibration Error to assess the models' uncertainty estimates. The calibration error measures how well the model's confidence in its prediction matches the actual accuracy. For a calibrated model, prediction has 70% confidence, then it should have 70% chance of being correct. We follow previous work and use a discrete approximation of this metric, where each bin has 100 predictions (Hendrycks et al. 2020b).
Baselines. We compare with the following representation learning approaches:
.Mooo v2 (Chen et al.) 2020d). We select MoCo v2 as the baseline and build all of the following approaches top of its publicly available official codebase. We select MoCo v2 due to its state- of-the-art performance and lower requirement for batch size. Our approach and baselines can easily adapt and be built other unsupervised representation learning approaches. In Appendix B
5

---

Under review as a conference paper at ICLR 2021
Table 2: Transferable adversarial robustness results. The highest accuracy is marked with bold. Our full approach URRL-SR has the highest robust accuracy for most of the datasets among represen- tation learning approaches, while losing much less clean accuracy than adversarial training. This is achieved without adversarial training and further data augmentation.
Airerall
Bindsupp
CIFARIO
CIFAR100
Caltech101
Caltech256
Fowar
SUN399
Avarage
Cars
DTD
Food
Pots
robust uccuracy:
ImageNet
2.01
16.18
28.51
19.60
3.07
21.76
26.85
7.60
8.09
2.92
MoCo v2
8.22
5.73
39.37
15.49
$6.86
26.40
24.19
36.76
$1.20
10.44
17.66
9.03
25.11
6.17
39.85
25.75
URRE-L2S0
16.47
8.11
40.68
17.84
59.15
28.42
27.84
37.23
55.08
15.86
21.72
8.59
28.08
MoCo-SR
25.71
11.88
69.48
16.95
55.94
33.55
42.76
40.64
49.59
33.57
40.86
14.59
36.29
URRL-SR
26.55
11.93
70.44
19.55
56.81
33,63
42.94
39.89
49.88
34.28
40.94
13.76
36.72
Adv. Trunning
45.84
42.49
87.93
64.44
39.06
44,80
48.22
25,80
36,25
66.43
19.27
34.46
46.25
clean accuracy:
ImageNet
83.17
72.49
97.27
84.73
93.34
82.10
87.71
74.84
95.15
87.37
93.73
62.41
84.53
MoCo v2
87.69
70.76
97.54
83.04
85.09
70.77
89.64
69.20
94.77
87.69
86.92
57.04
81.68
URRL
87.04
70.91
97.34
83.34
85.60
70.64
68.72
95.09
87.91
57.38
81.77
URRL-L2SP
87.37
71.24
97.37
83.55
85.34
72.48
89.95
70.85
95.76
88.06
88.74
57.75
82.37
MoCo-SR
77.73
60.66
97.24
86.49
67.98
79.31
97.17
88.55 28.83
77.85
URRL-SR
78.55
60.92
97.10
80.60
88.38
67.61
80.66
68.62
95.87
86.74
88.83
46.72
78.38
Adv. Trinning
57.52
56.86
92.94
74.30
42.80
51.97
58.97
31,33
40.66
79.63
23.06
44.31
54.53
show how URRL can be used to improve other unsupervised learning approaches. Results in the main paper would focus on the relative improvement from MoCo v2 for a fair comparison.
AugMix (Hendryeks et al. 2020b). We select AugMix the family of robust data augmentation due to its simplicity and effectiveness. Here, we simply replace the pretext task of MoCo v2 with AugMix for unsupervised representation learning using Eq. (T).
AugMix (AllOp). In the original AugMix paper, augmentation types contrast, color or, excluded as they involve more drastic color distortion and
br brightness, and sharpness, may overlap with ImageNet-C corruptions. However, these types of corruption are already used in the pretext task of MoCo v2. Therefore we add these operations back for fair comparison.
AugMix (MoCo). We can further strengthen the color distortion of AugMix by including the data augmentations used in MoCo v2 within the random convex combinations of AugMix. For example, MoCo v2 randomly makes the cropped patch grayscale as data augmentation. However, in AugMix, this only happens in the extreme case of color augmentation.
URRL. Finally, our URRL in Eq. (2) introduces color distortion to robust augmentation by hierar- chically and randomly selecting the family of data augmentation use. We use the pretext task of MoCo v2 for Trep. and AugMix (AIIOP) for Trob-
Implementation Details. We build on the official repo of MoCo v2, and use the exact same hyper- parameters for all methods (200 epochs with learning rate 0.3). We use 0.5 in Eq. 2
Results. The results are shown in Table Directly using AugMix as a pretext task not ideal, how- ever, the performance is already comparable to competitive approaches in the same setup (He et al.) 2020 Chen et al. 2020b). By gradually increasing the severity of color distortion augmentations, AugMix (AllOp) and AugMix (MoCo) continue to improve both the clean accuracy (ImageNet top- 1) and mCE. URRL performs the best on all three metrics by combining the data augmentation from MoCo v2 and AugMix. The randomly selected family of data augmentation could involve drastic color distortion (when Trep is selected), and thus the model cannot take the shorteut by looking at the color histogram. On the other hand, the corruption and perturbations in AugMix complement well with the pretext used in MoCo v2, and even improves the ImageNet top-1 1 accuracy.
3.2 EVALUATING TRANSFERABLE ADVERSARIAL ROBUSTNESS
While it is helpful to evaluate representations under linear evaluation, the performance is limited when we fine-tune the representations for other datasets. In this section, we evaluate the performance and robustness of representations after transfer learning and fine-tuning the whole model.

---

Under review as a conference paper at ICLR 2021
Table 3: RMS calibration errors on CIFAR-IO-C and CIFAR-100-C. Similarity regularization sig-
nificantly improves models' calibration.
MoCo v2
URRL
MoCo-SR
URRL-SR
CIFAR-IO-C
16.96
17.90
9.92
8,95
CIFAR-100-C
19.05
16.39
12.60
11.25
Experimental Setup. We consider 12 common datasets used for evaluating transfer learning of representation learning (Salman et al. 2020). If not noted otherwise, the representation is trained with the transfer learning loss in Eq. (3). where ge, is linear classifier on top of the representation, and ((.,)) is the cross entropy loss for classification.
Metrics. We report the top-1 clean accuracy on these datasets. In addition, we take the models and report the robust accuracy from adversarial evaluation. We follow the settings of llyas et al. (2019).
Methods for Comparison. We compare MoCo v2 and the proposed URRL with and without sim- versions as MoCo-SR and URRL-SR.
ilarity regularization. We refer to the memnilarity-rgullati We use 1.2-SP as the baseline for transfer learning regularization, which directly regularizes the fine- tuned weights to be similar to that of the pre-trained weights (Li et al. 2018). Our approach with L2-SP is referred to as URRL-L2SP. We also evaluate ImageNet pre-trained weights to compare the transferable robustness of supervised and unsupervised representations. We show results of adver- sarial training on each of the 12 datasets reference for robust accuracy. but it is important to note that it not a fair comparison as it performs much heavier adversarial optimization for downstream tasks.
Implementation Details. We use the ResNet-50 backbone as in Section which has a x convl instead of 3 x 3 convl often modified for CIFAR-IO. We resize the images to 256 x 256 to feed into our network. We select the best learning rate and the weight for Lsim on a validation set, and for the rest of the parameters, we use the same as /Salman et al. (2020). We use the code and parameters of llyas et al. (2019) for adversarial training and evaluation (f2 attack with c = 0.5).
The results for clean accuracy and robust accuracy are shown in Table ImageNet pre- trained weights have the best clean accuracy in our experiment. This is consistent with results in previous works that use the same ResNet-50 backbone and computing budgets (200 epochs) as our experimental setup (He et al. 2020; Chen et al. 20206). However, ImageNet pre-trained representa- tion has the lowest robust accuracy by large margin. Unsupervised representation learning (MoCo v2) is already more robust than ImageNet pre-trained weights. URRL is able to further improve both the clean and robust accuracy, and regularizing the weights to be similar that of the pre-traines weights is also effective (URRL-L2SP). Nevertheless, the proposed similarity regularization signifi- cantly improves the robust accuracy by over 10% on average for both MoCo-SR and our URRL-SR, with a much smaller loss of clean accuracy compared to that of adversarial training. URRL-SR has the highest robust accuracy for most of the datasets among representation learning approaches, while losing much less clean accuracy than adversarial training. This shows that both the proposed URRL and similarity regularization are able to improve the transferable robustness without much overhead. In particular, the proposed similarity regularization is also better for robustness compared to 1.-SPP because similarity measure loss directly regularizes the robustness induced by contrastive learning augmented data during training.
Results.
3.3 UNCERTAINTY CALIBRATION OF TRANSFER LEARNING
We further evaluate the uncertainty calibration of transfer learning. Improving the uncertainty cali- bration of downstream task models is important aspect of transferable robustness. If the learned representation can improve calibration of downstream tasks, then even if the downstream task model cannot handle all kinds of new inputs outside its training distribution, it at least knows when it is likely to be mistaken. This alleviates the potential impact of unforeseen inputs.
Experimental Setup. We use CIFAR-10-C and CIFAR- 100-C for this experiment, where pertur- bations that are similar to ImageNet-C are applied to CIFAR-10 10 and CIFAR-100 instead of Ima- geNet (Hendryeks & Dietterich) 2019). We fine-tune models CIFAR-10 and CIFAR-100 and
7

---

Under review as a conference paper at ICLR 2021
evaluate CIFAR-1O-C and CIFAR- 100-C, respectively. We evaluate the RMS Calibration Error following the previous work of Hendrycks et al. (2020b), where each bin has 100 predictions for approximation.
The RMS calibration errors are shown in Table The proposed similarity regularization significantly improves the calibration error on both datasets (half the error for URRL-SR compared to URRL on CIFAR-1O-C). URRL-SR further improves calibration error compared to MoCo-SR. This shows the importance of similarity regularization in combination with the proposed URRL for transfer learning robustness.
Results.
4 RELATED WORK
Unsupervised representation learning. There have been significant recent advances in unsuper-
vised representation learning. Contrastive learning (Gutmann & Hyvärinen (010)) has become a
predominant design leading to state-of-the-art performance (Oord et al.) 2018 Bachman al. al.] 2019)
Chen et al. 2020c by He et al., [020] Misra & Maaten 2020) Tian et al. |2200] et al. (020d)
Latest methods even outperform supervised pre-training on several transfer learning tasks (Caron itt al. 2020). Most existing works benchmark representations by directly fine-tuning the pre-trained models on downstream tasks. However, as observed by He et al. (2019), ImageNet pre-training does not necessarily improve performance on large-scale downstream tasks under such settings. We be- lieve part of the reason lies in the catastrophic forgetting, especially when the scale of a downstream task is large. The proposed similarity regularization casts new insights towards addressing this issue.
Robustness from perturbed inputs. Besides playing an important role in unsupervised represen- tation learning, training with perturbed/coorupteed inputs has been widely shown to render stronger robustness against adversarial attacks (Eghbal- -zaech et al. 2020; Madry et al.
2018) and corrup-
tion (Hendrycks et al. 2020b a). as well as improved network attention towards better task perfor-
mance (Ghiasi et al.] 2018; (Singh & Lee| 2017 Yun et al. 2019 Zhong et al.)
2020). Similar
approaches (Cubuk et al. 2020) have also been widely adopted in unsupervised domain adapta-
tion (French et al.] 2018)) and semi-supervised learning (Berthelot et al.] 2011b a Laine & Aila,
2017 Sohn et al. 2020, Xie al., 2220). We develop new methods to improve the transferable
robustness of unsupervised representation learning, which is an important yet under-explored area.
Robustness and representation learning. In this work, we show that combining standard pretext task with AugMix improves both the clean accuracy and robustness of unsupervised representation learning. This is consistent with recent findings on the benefit of having better representation for su- pervised adversarial robustness. It has been shown that self-supervised learning (Chen et al. [2020a] Hendrycks et al., 2019b, Jiang et al.] 2020 Kim et al.) 2020), supervised pre-training (Hendrycks) eet al. 2019a), unlabeled data (Alayrac et al. 2019 Carmon et al. 2019). and multi-task learn- ing (Ma et al. 2020) are beneficial to adversarial robustness when jointly optimized with adversar- ial training. However, the main goal of these approaches is to complement supervised adversarial training. whether the approach itself is supervised or unsupervised. The robustness of unsupervised representation learning rather under-explored. Recent work even shows that self-supervised model lacks noise robustness (Geirhoo et al.] 2020). Our goal is to address this challenge and improve the transferable robustness of representations without adversarial training and further data augmenta- tion. The most related work to ours are Alayrac et al. (2019) and Kim et al. (2020), which aim to improve adversarial robustness without labeled data. However, we still see an accuracy-robustnes trade-off. Our work shows that the trade-off is not necessary for unsupervised representation learn- ing, and covers a wider range of robustness, such as common corruptions and uncertainty calibration.
Transfer learning robustness. Our emphasis on the transferable robustness is closely related to the works on the robustness of transfer learning. Rezaei & Liu (202)) have shown that downstream task models are vulnerable to attacks based on the pre-trained representation. (Cinn et al. (2220) proposes a noisy feature distillation as a defense for the aforementioned attack. Shaafai et al. (2020) also shows that learning without forgetting (LwF) (Li & Hoiem, 2017) based loss from ad adversarially trained source model can improve adversarial robustness of the downstream task model. Our goal of learning more robust representation without supervision is mutually beneficial to these efforts by serving as a better starting point for robust transfer learning. Our similarity regularization has a sim- ilar goal with the LwF loss, but is less restrictive (only pairwise relationship constraint). Since these
8

---

Under review as a conference paper at ICLR 2021
regularizations do not require test-time labels, they can also be applied to test-time optimization (Sun et al. 2020) to regularization the self-superrvsed learning loss in the target/test domain.
5 CONCLUSIONS
We present Unsupervised Robust Representation Learning (URRL), new framework that com- bines the advantages of unsupervised representation learning and supervised learning robustness. This upends the commonly assumed accuracy-robustees trade-off in the case of unsupervised rep- resentation learning. Moreover, we show that the proposed similarity regularization can preserve URRL's robustness during fine-tuning, Our full model URRL-SR significantly improves the down- stream task robustness without adversarial training and further data augmentation. We believe an impactful future direction is how even without any regularization. While our regularization is task-agnostic, it is still desirable that one can straightforwardly improve the robustness of fine-tuned models without any modification.
learn representation that is still robust after fine-tuning
REFERENCES
lean-Baptiste Alayrac, Jonathan Uesato, Po-Sen Huang, Alhussein Fawzi, Robert Stanforth, and Pushmeet Kohli. Are labels required for improving adversarial robustness? In NeurIPS, 2019.
Philip Bachman, R Devon Hjelm, and William Buchwalter. Learning representations by maximizing mutual information across views. In NeurIPS, 2019.
David Berthelot, Nicholas Carlini, Ekin D Cubuk, Alex Kurakin, Kihyuk Sohn, Han Zhang, and Colin Raffel. Remixmatch: Semi-super learning with distribution alignment and augmenta- tion anchoring. In ICLR, 2019a.
David Berthelot, Nicholas Carlini, Ian Goodfellow, Nicolas Papernot, Avital Oliver, and Colin A Raffel. Mixmatch: A holistic approach to semi-supervised learning. In NeurlPS, 2019b.
Yair Carmon, Aditi Raghunathan, Ludwig Schmidt, John C Duchi, and Percy Liang. Unlabeled data improves adversarial robustness. In NeurIPS, 2019,
Mathilde Caron, Ishan Misra, Julien Mairal, Priya Goyal, Piotr Bojanowski, and Armand Joulin. Unsupervised learning of visual features by contrasting cluster assignments. In NeurlPS, 2020.
Tianlong Chen, Sijia Liu, Shiyu Chang, Yu Cheng, Lisa Amini, and Zhangyang Wang. Adversarial robustness: From self-supervised pre-training fine-tuning. In CVPR, 2020a.
Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey Hinton. A simple framework for contrastive learning of visual representations. In ICML, 2020b.
Ting Chen, Simon Kornblith, Kevin Swersky, Mohammad Norouzi, and Geoffrey Hinton. Big self- supervised models are strong semi-supervised learners. arXiv preprint arXiv:2006.10029, 2020c.
Xinlei Chen, Haoqi Fan, Ross Girshick, and Kaiming He. Improved baselines with momentum contrastive learning. arXiv preprint arXiv:2003.0 04297, 2020d.
Ting- Wu Chin, Cha Zhang, and Diana Marculescu. Improving the adversarial robustness of transfer learning via noisy feature distillation. arXiv preprint arXiv:2002.02999 2020.
Ekin D Cubuk, Barret Zoph, Jonathon Shlens, and Quoc V Le. Randaugment: Practical automated data augmentation with reduced search space. CVPR Workshops, 2020.
J. Deng, W. Dong, R. Socher, L.JJ Li, K. Li, and L. Fei-Fei. ImageNet: A Large-Scale Hierarchical Image Database. CVPR, 2009.
Hamid Eghbal-zadeh, Khaled Koutini, Paul Primus, Verena Haunschmid, Michal Lewandowski, Werner Zellinger, Bernhard A Moser, and Gerhard Widmer, On data augmentation and adversarial risk: An empirical analysis. arXiv preprint arXiv:2007.02653 2020.

---

Under review as a conference paper at ICLR 2021
Geoffrey French, Michal Mackiewicz, and Mark Fisher. Self-ensembling for visual domain adapta- tion. In ICLR, 2018.
Robert Geirhos, Kantharaju Narayanappa, Benjamin Mitzkus, Matthias Bethge, Felix A Wichmann, and Wieland Brendel. On the surprising similarities between supervised and supervise mod- cls. arXiv preprint W0X1v:2010008377, 2020.
Golnaz Ghiasi, Tsung-Y Lin, and Quoc Le. Dropblock: regularization method for convolu- tional networks. In NeurIPS, 2018.
Michael Gutmann and Aapo Hyvärinen. Noise-contrastive estimation: new estimation principle for unnormalized statistical models. In Proceedings of the Thirteenth International Conference on Artificial Intelligence and Statistics, pp. 297-304, 2010.
Kaiming He, Ross Girshick, and Piotr Dollár. Rethinking imagenet pre-training. In Proceedings of the IEEE international conference on computer vision, pp. 4918-4927, 2019.
Kaiming He. Haoqi Fan, Yuxin Wu, Saining Xie, and Ross Girshick. Momentum contrast for unsupervised visual representation learning. In CVPR, 2020.
Dan Hendrycks and Thomas Dietterich. Benchmarking neural network robustness to common cor- ruptions and perturbations. ICLR, 2019.
Dan Hendrycks, Kimin Lee, and Mantas Mazeika. Using pre-training can improve model robustness and uncertainty. JCML, 2019a.
Dan Hendrycks, Mantas Mazeika, Saurav Kadavath, and Dawn Song. Using self-supervised learning can improve model robustness and uncertainty. NewrIPS, 2019b.
Dan Hendrycks, Steven Basart, Norman Mu, Saurav Kadavath, Frank Wang, Evan Dorundo, Rahul Desai, Tyler Zhu, Samyak Parajuli, Mike Guo, al. The many faces of robustness: A critical analysis of out-of-distribution generalization. arXiv preprint arXiv:2006. 16241, 2020a.
Dan Hendrycks, Norman Mu, Ekin D Cubuk, Barret Zoph, Justin Gilmer, and Balaji Lakshmi- narayanan. Augmix: A simple data processing method to improve robustness and uncertainty. ICLR, 2020b.
Andrew lyas, Shibani Santurkar, Dimitris Tsipras, Logan Engstrom, Brandon Tran, and Aleksander Madry. Adversarial examples are not bugs, they are features. In NeurlPS, 2019.
Z. Jiang, Chen, T. Chen, and Z. Wang. Adversarial contrastive learning: Harvesting more robust- from unsupervised pre-training. In NeurIPS, 2020.
Minseon Kim, Jihoon Tack, and Sung Ju Hwang. Adversarial self-supervised contrastive learning. In NeurlPS, 2020.
Ananya Kumar, Percy Liang, and Tengyu Ma. Verified uncertainty calibration. In NeurIPS, 2019.
Samuli Laine and Timo Aila. Temporal ensembling for semi-supervised learning. In
2017. 2017.
Xuhong Li, Yves Grandvalet, and Franck Davoine. Explicit inductive bias for transfer learning with convolutional networks. In ICML, 2018.
Zhizhong Li and Derek Hoiem. Learning without forgetting. TPAMI, 40(12):2935-2947.22
Aleksander Madry, Aleksandar Makelov, Ludwig Schmidt, Dimitris Tsipras, and Adrian Vladu. Towards deep learning models resistant adversarial attacks. ICLR, 2018.
Chengzhi Mao, Amogh Gupta, Vikram Nitin, Baishakhi Ray, Shuran Song, Junfeng Yang, and Carl Vondrick. Multitask learning strengthens adversarial robustness. ECCV, 2020.
Ishan Misra and Laurens van der Maaten. Self-supervised learning of pretext-invane invariant representa- tions. In CVPR, 2020.
10

---

Under review as a conference paper at ICLR 2021
Aaron den Oord, Yazhe Li, and Oriol Vinyals. Representation learning with contrastive predic- tive coding. arXiv preprint arXiv:1807.03744, 2018.
Shahbaz Rezaei and Xin Liu. A target-agnostic attack on deep models: Exploiting security vulner- abilities of transfer learning. ICLR, 2020.
Hadi Salman, Andrew Ilyas, Logan Engstrom, Ashish Kapoor, and Aleksander Madry. Do adver- sarially robust imagenet models transfer better? In ArXiv preprint arXiv:2007.0844 08449, 2020.
Ali Shafahi, Mahyar Najibi, Mohammad Amin Ghiasi, Zheng Xu, John Dickerson, Christoph Studer, Larry Davis, Gavin Taylor, and Tom Goldstein. Adversarial training for free! In NeurlPS, 2019.
Ali Shafahi, Parsa Saadatpanah, Chen Zhu, Amin Ghiasi, Christoph Studer, David Jacobs, and Tom Goldstein. Adversarially robust transfer learning. ICLR, 2020.
Krishna Kumar Singh and Yong Jae Lee. Hide-and-seek: Forcing a network to be meticulous for weakly-supervised object and action localization. ICCV, 2017.
Kihyuk Sohn, David Berthelot, Chun-Liang Li, Zizhao Zhang, Nicholas Carlini, Ekin Cubuk, Alex Kurakin, Han Zhang, and Colin Raffel. Fixmatch: Simplifying semi-supervised learning with consistency and confidence. arXiv preprint arXiv:2001.07653 2020.
Yu Sun, Xiaolong Wang, Zhuang Liu, John Miller, Alexei A Efros, and Moritz Hardt. Test-time training with self-supervision for generalization under distribution shifts. ICML, 2020.
Yonglong Tian, Chen Sun, Ben Poole, Dilip Krishnan, Cordelia Schmid, and Phillip Isola. What makes for good views for contrastive learning. arXiv preprint arXiv:2005./024: 2020.
Zhirong Wu, Yuanjun Xiong, Stella X Yu, and Dahua Lin. Unsupervised feature learning via non- parametric instance discrimination. CVPR, 2018.
Qizhe Xie, Minh-Thang Luong, Eduard Hovy, and Quoc V Le. Self-training with noisy student improves imagenet classification. In CVPR, 2020.
Yuo-Yuan&uo Yang, Cyrus Rashtchian, Hongyang Zhang. Ruslan Salakhutdinov, and Kamalika Chaud- huri. Adversarial robustness through local lipschitzness. arXiv preprint arXiv:2003.024660 2020.
Sangdoo Yun, Dongyoon Han, Scong Joon Oh, Sanghyuk Chun, Junsuk Choe, and Youngjoon Yoo. Cutmix: Regularization strategy to train strong classifiers with localizable features. In ICCV, 2019.
Haichao Zhang and Jianyu Wang. Towards adversarially robust object detection. In ICCV, 2019.
Hongyang Zhang, Yaodong Yu, Jiantao Jiao, Eric Xing, Laurent Ghaoui, and Michael I Jordan. Theoretically principled trade-off between robustness and accuracy. arXiv preprint arXiv: 0901308573 2019.
Zhun Zhong, Liang Zheng, Guoliang Kang, Shaozi Li, and Yang. Random erasing data augmen- tation. In AAAI, 2020.

---

Under review as a conference paper at ICLR 2021
A
LŁ-SP HYPERPARAMETER ANALYSIS
97.75
no L2-SP
no L2-SP
@ 0.00001
0.00001
97.50
0.0001
97.25
8 0.001
0.001
t= 0.01
G 0.01
97.00
0
96.75
96.50
0
0.001
0.01
8
0.000
B 0.01
(a) L2-SP CIFAR-I0 Clean Accuracy
(b) L2-SP CIFAR-I0 Robust Accuracy
Figure 2: L3-SP results CIFAR-10.
Figure shows the SP results CIFAR-10, where conduct the same hyperparameters anal- ysis as Li et al. (2018). We can see that their best performing hyperparameters (a = 0.1, 3 0.01) actually hurt both the clean accuracy and robust accuracy in case. This is due ences of our work compared to theirs: First, our focus is on unsupervised representation learning- Compared to the representations learned from supervised learning on large-scale datasets like Im- ageNet and Places365, our unsupervised representations typically need further updates in order to do well on downstream classification tasks. this case, I.2 regularization not enough by itself to improve the performance. Second, robustness is not the goal of Li et al. (2018), and L is not de- signed to improve the robustness of transfer learning. In contrast, the primary goal our work is to improve the transferable robustness of learned representations, and the design of our similarity mea- sure loss motivated by preserving the robustness learned representations during fine-tuning. As discussed in Section 2.2 the robustness of feature representation comes from how it measures the similarities between images, and the proposed approach explicitly regularizes during fine-tuning. Our results in Table also show the importance of the proposed approach.
key differ-
B GENERALITY OF OUR AUGMENTATION SCHEME
Table 4: Results on applying URRL to multiple unsupervised representation learning approaches.
ImageNet top-1
ImageNet-CC mCE
Calibration Error
InstDis InstDis w/ URRL
59.50
104.93
19.86
60.42
101.64
18.30
PIRL
61.70
103.15
18.81
PIRL w/ URRL
62.47
99.74
18.19
We conduct the experiments in Section 3.1 on other unsupervised representation learning ap- proaches, including InstDis (Wu et al.) 2018) and PIRL (Misra & Maaten (0220) to show the gener- ality of our approach. Results are shown in Table 4 Trep and AugMix (AllOp) as Trob in Eq. Similarly for PIRL and PRIL w/ URRL. URRL im- proves both of these methods, which supports the generality of our approach. We hope this facilitates its broad adoption in sellssupervieednnsuprrisssd learning
InstDis w/ URRL means that we use InstDis as
12


Reviews:
[{"review_id": "ysTPLuahTjB", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 7, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper uses a different data augmentation (AugMix) scheme to improve self-supervised representation learning. It improves accuracy and (corruption and adversarial) robustness by a sufficiently interesting amount. The paper's presentation is clear, but the paper could be more thorough. Since the technique is simple and general, it could easily be broadly applicable to the burgeoning area of self-supervised learning.\n\nOther pros:\n_On the surprising similarities between supervised and self-supervised models_ shows that most techniques for self-supervised learning don't improve robustness at all, so self-supervised learning produces representations that have limited use downstream.\nThis work counteracts this limitation by also assessing the robustness of self-supervised representations, and the technique they propose improves robustness. They improve robustness with data augmentation, a volatile ingredient in self-supervised learning. SimCLR reminds us that the choice of image modifications is crucial and must be carefully selected, so it was not obvious a priori that their technique should help.\n\nOther cons:\nThe paper would be stronger if it showed their augmentation scheme helping with more than MoCo. That would demonstrate generality. I'll boost my score if it improves another self-supervised learning technique.\nThe paper could show results with another robustness benchmark, such as ImageNet-R. This would make a stronger case for robustness. This would be quick to add.\nURRL-SR uses embedding distances, but this seems like logit pairing. Logit pairing can lead to an overestimate of adversarial robustness. Please include results with an adversarial attack with restarts. Also state whether you're using l2 or l_infty attacks, rather than having refer to Ilyas et al.\nThe paper has one only figure, and Table 2's formatting is slightly unintuitive.\n\nUpdate: I am happy with the changes and am keeping my score.", "extra_scores": {}, "extra_text": {}}, {"review_id": "cV6iNfZge1v", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The authors improve the fine-tuning of a data representation allowing\nresistance and recognition of adversarial or corrupt inputs.  Their method,\nURRL, uses an auxiliary unsupervised task and robust data augementation to\nimprove performance for both clean and altered inputs.  A key addition is that,\nwhile fine-tuning on the auxiliary (downstream) task, performance on the\noriginal data is maintained by favoring small differences between original\nrobust representation and the fine-tuned representation.\n\nThe work is well presented, and the main significance is in providing a fast\nway to partially gain some of the robustness available in much more difficult\nadversarial training.  This might be of significance in applications like\nonline learning, where only fast fine-tuning is practical.\n\nTheir approach limits how much fine-tuning is allowed to deviate from the\noriginal, robust representation.  It is a *fast* way to partially recover some \nof the benefits of a much more expensive adversarial training.  The problem\nthey address is important.  For example, observations in related works often \nnote a slight decrease in performance on the original supervised task during\nlater fine-tuning.  The issue to resolve is how to nicely limit degradation,\neven in face of aggressive or long-term fine-tuning.\n\nTo maintain robustness during fine-tuning, one (expensive) option is to\nadversarially fine-tune the downstream task.  The authors propose a quicker\napproach: a loss function maintaining similarity between fine-tuned and\noriginal representation (e.g. low fine-tuning distortion of dot-product\nsimilarity).\n\nThe experiments nicely evaluate robustness with a number of relevant metrics,\nusing ImageNet with many corruption types.  Results are presented as improvements\nto the MoCo v2 codebase.  The use of a linear classifier stage seemed reasonable to me.\n\nFirstly they address what sets of data augmentations are most effective for\nimages, using a mechanism that probabilistically switches between augmentation\ntypes for supervised and unsupervised tasks.\n\nSecondly, the more interesting results with similarity loss showed very good (usually best)\nadversarial performance.  Accuracy on clean data was degraded by ~ 10%, still less than the\ndegradation caused by adversarial training.  Similarity loss can be applied to other existing\ntechniques (like their MoCo baseline).\n\nQ: Does the similarity measure loss still make sense if fine-tuning uses\ndata sampled from a new/changing input distribution?  It might\n(see also \"Test-Time Training with Self-Supervision\nfor Generalization under Distribution Shifts\", Sun et al.)\n\n---\n\nThe authors' comments strengthen the argument for the method.   I (perhaps liberally) conceptualize\ntheir approach as one of tethering \"close to a rotation\" but still don't have a simple understanding of\nwhy/when this should be better than tethering \"close to original parameters\". My original rating is unchanged.", "extra_scores": {}, "extra_text": {}}, {"review_id": "-60iXO0XUXM", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "## Paper summary\nThis paper proposes *Unsupervised Robust Representation Learning* (URRL), a framework that combines several data augmentation schemes and a similarity-based loss. The goal is to improve the robustness of visual representations to image perturbations. A further goal is to maintain the robustness properties of pre-trained representations after fine-tuning the network to downstream tasks.\n\nThe proposed method consist of two components: A scheme for sampling combinations of image augmentations, and a loss that is applied during transfer learning and penalizes deviations of the pairwise similarity between datapoint representations from the similarities before fine-tuning.\n\nThe method is evaluated on clean data performance and adversarial robustness, as well as calibration error.\n\n## Arguments for acceptance\n1. The paper is well written and clear. \n2. The proposed method improves on MoCo-v2 and Augmix in ImageNet accuracy, ImageNet-C corruption error, RMS calibration error and adversarial robustness.\n\n## Arguments against acceptance\n3. The proposed augmentation approach appears to be very similar to the methods used in MoCo, AugMix, and related work. Section 2.1 of the paper argues that these methods represent two different families of augmentation ($\\mathcal{T}_\\text{rep}$ and $\\mathcal{T}_\\text{rob}$). The paper suggests that augmentations need to be sampled from both of these families in a particular way (Eq. 2) to overcome the issue that one of the families affects the color histogram too much. This approach is only a small and heuristic variant of existing methods. It has been known since long before MoCo and AugMix that image augmentations improve unsupervised representation learning and robustness. Formal arguments for the distinction between $\\mathcal{T}_\\text{rep}$ and $\\mathcal{T}_\\text{rob}$ are not given. I am not sure whether the proposed variant is original enough for publication at ICLR. Further, the performance improvement of URRL over MoCov2 is small, less than one percentage point for most metrics.\n\n4. The proposed similarity regularization leads to more significant improvements in fine-tuning performance. However, this method is not compared against baselines. Regularization approaches for improving transfer performance have been proposed before, e.g. see [Li et al., 2018](https://arxiv.org/pdf/1802.01483.pdf). These should be compared to the proposed method. In particular, well-tuned weight decay towards the frozen pre-trained weights ($L^2$-$SP$ in Li et al.) is an essential baseline. I suspect that the proposed similarity regularization may be functionally equivalent to decaying weights towards their initial values.\n\n## Conclusion\nWhile the paper is well organized and clear, in its current form it does not meet the originality and significance standards of ICLR and lacks crucial baselines. Suggestions for improvement: \n\n5. Provide a more formal argument for the proposed augmentation framework to show that it is a significant conceptual advance over the current knowledge that augmentations are important.\n6. Provide a more formal motivation for the similarity regularization, including a formal comparison to $L^2$-$SP$.\n7. Compare the similarity regularization to a well-tuned $L^2$-$SP$-decay baseline.\n\n## Update after rebuttal:\nThe authors provided convincing evidence that the proposed similarity regularization performs better with regards to robustness than $L^2$-$SP$. I adjusted my rating accordingly. I still think the contributions are borderline because the proposed regularization is not backed up by theoretical arguments and the augmentation approach is incremental.", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: The authors propose a framework which combines pretext tasks and data augmentation schemes with the goal of improving robustness of image representations. The authors show that this approach empirically can lead to increased accuracy on both corrupted and uncorrupted data simultaneously. Furthermore, the authors propose a regularization procedure which can be used to maintain a robust representation during transfer to arbitrary downstream tasks. 

The studied problem is significant and highly relevant to the ICLR community. The reviewers agreed that the work is timely, and appreciated the clarity of exposition. At the same time, the reviewers remained in disagreement in terms of novelty and significance of the results -- the proposed method is seen as a clear incremental application of existing techniques in the self-supervised setting. The authors argue that it was not clear that such augmentations would improve the robustness as well as accuracy, but these methods were developed and optimised to improve robustness. In fact, in [1] the authors conclude that “...today's supervised and self-supervised training objectives end up being surprisingly similar” as well as point out that SimCLR is more robust than competing self-supervised methods. Hence, establishing that there is indeed some empirical benefit is a step in the right direction, but not sufficient to meet the bar of acceptance. Furthermore, given the recent trend of scaling-up existing approaches, in particular in terms of the neural architectures and the batch sizes, the computational costs of the proposed regulariser in Eq (4) coupled with the additional hyperparameter to optimize make the approach less practical and general than claimed. In addition, the reviewers pointed out the lack of comparison to a proper baseline, as well as the issue with hyperparameter selection for the baseline after the author response. Finally, given access to additional data augmentations and one more hyperparameter to tune the results should substantially outperform the baselines.

For the reasons outlined above and the incremental nature of the work, I will recommend rejection. That being said, this was a borderline case, and I urge the authors to carefully revise the manuscript with the received feedback.

[1] https://arxiv.org/abs/2010.08377

Decision: Reject