Pushing the limits of self-supervised learning: Can we outperform supervised learning without labels?
Authors: Nenad Tomasev
Ioana Bica
Brian McWilliams
Lars Holger Buesing
Razvan Pascanu
Charles Blundell
Jovana Mitrovic
Venue: Submitted to ICLR 2023 (2023)
Keywords: self-supervised learning
contrastive learning
ImageNet

Abstract: Despite recent progress made by self-supervised methods in representation learning with residual networks, they still underperform supervised learning on the ImageNet classification benchmark, limiting their applicability in performance critical settings. Building  on  prior  theoretical  insights  from  RELIC  [Mitrovic et  al.,  2021],  we  include  additional  inductive  biases  into  self-supervised  learning. We propose a new self-supervised representation learning method, RELICv2,which  combines  an  explicit  invariance  loss  with  a  contrastive  objective  over  avaried set of appropriately constructed data views to avoid learning spurious cor-relations and obtain more informative representations. RELICv2 achieves 77.1% top-1 classification accuracy on ImageNet using linear evaluation with a ResNet50 architecture and 80.6% with larger ResNet models, outperforming previous state-of-the-art self-supervised approaches by a wide margin. Most notably, RELICv2 is the first unsupervised representation learning method to consistently outperform the supervised baseline in a like-for-like comparison over a range of ResNet architectures. Finally, we show that despite using ResNet encoders, RELICv2 is comparable to state-of-the-art self-supervised vision transformers.

Under review as a conference paper at ICLR 2023
PUSHING THE LIMITS OF SELF-SUPERVISED RESNETS: CAN WE OUTPERFORM SUPERVISED LEARNING WITHOUT LABELS ON IMAGENET?
Anonymous authors Paper under double-blind review
ABSTRACT
Despite recent progress made by self-supervised methods in representation learn- ing with residual networks, they still underperform supervised learning on the ImageNet classification benchmark, limiting their applicability in performance- critical settings. Building on prior theoretical insights from RELIC [Mitrovic jet al. 202T]. we include additional inductive biases into self-supervised learn- ing. We propose new self-supervised representation learning method, RELICv2, which combines an explicit invariance loss with a contrastive objective over a varied set of appropriately constructed data views to avoid learning spurious cor- relations and obtain more informative representations. RELICV2 achieves 77.1% top-1 classification accuracy ImageNet using linear evaluation with ResNet50 architecture and 80.6% with larger ResNet models, outperforming previous state- of-the-art self-supervised approaches by wide margin. Most notably, RELICv2 is the first unsupervised representation learning method to consistently outperform the supervised baseline like-for-like comparison over a range ResNet ar- chitectures. Finally, we show that despite using ResNet encoders, RELICv2 is comparable to state-of-the-an superviee vision transformers.
INTRODUCTION
Large-scale foundation models [Bommasani al. 2021 in particular for language (Devlin al., 2018 Brown et al., 2020 and multimodal domains [Radford et al.) [021] are an important recent development in representation learning. The idea that massive models can be trained without la- bels in an unsupervised (or self-supervised) manner and be readily adapted, in few- or zero-shot setting, to perform well on a variety of downstream tasks is important for many problem areas for which labeled data is expensive or impractical to obtain. Contrastive objectives have emerged as successful strategy for representation learning [Chen et a 2020aa] He et al., 2209], However, down- stream utility" of these representations has until now never exceeded the performance of supervised training of the same architecture, thus limiting their usefulness.
In this work, tackle the question "Can outperform supervised learning without labels Im- ageNet?". As such, our focus on learning good representations for high-level vision tasks such image classification. In supervised learning we have access to label information which provides a signal of what features are relevant for classification. In unsupervised learning there is no such signal and we have strictly less information available from which to learn compared supervised learning. Thus, the challenge of outperforming supervised learning without labels might seem im- possible. Comparing supervised and contrastive objectives we see that they are two very different approaches to learning representations that yield significantly different representations. While su- pervised approaches use labels as targets in within cross-entropy objective, contrastive methods rely on comparing against similar and dissimilar datapoints. Thus, supervised representations end up encoding a small set of highly informative features for downstream performance, while contrastive representations encode many more features with some of these features not related downstream performance. This intuition is also supported by the observation that when visualizing the encoded
Downstream utility is commonly measured by how well method performs under the standard linear evaluation protocol on ImageNet; see section 3

---

Under review as a conference paper at ICLR 2023
information of contrastive representations through reconstruction, they are found to retain more detailed information of the original image, such as background and style, than supervised represen- tations (Bordes et al., 2021].
Based on this, we hypothesize that one of the key reasons for the current subpar performance of con- trastive (and thus self-supervised) representations, is the presence of features which are not directly related to downstream tasks, i.e. so-called spurious features. In general. basing representations on spurious fea- tures can have negative consequences for the gener- alization performance of the model and thus avoiding to encode these features is paramount for learning in- formative representations.
Saps Ro43 our
Reaaba rod
"carouuze
dro.
Swon Oai
Sanvcun (411
"occu cenoi
In this paper, we propose to equip self-supervised methods with additional inductive biases to obtain more informative representations and overcome the lack of additional information that supervised meth- ods have access to. We use as our base self- supervised approach the performant RELIC method (Mitrovic et al., [021] which combines contrastive loss with an invariance loss. We propose to extend RELIC by adding inductive biases that penalize the learning of spurious features such as background and style, e.g. brightness; we denote this method as RELICv2. First, we propose new fully unsupervised saliency masking pipeline which enables us to separate the image foreground from its background. We include this novel saliency masking approach as part of the data augmen- tation pipeline. Leveraging the invariance loss, this enables us to learn representations that do not rely on the spurious feature of background. Second, while RELIC operates on just two data views of the same size, we extend RELICv2 to multiple data views of varying sizes. Specifically, we argue for learning from a large number of data views of standard size as well as including a small number of data views that encode only small part of the image. The intuition for using smaller crops is that this enables learning of more robust representations as not all features of the foreground might be contained in the small crop. Thus, the learned representation is robust against individual object features being absent as is the case in many real-world settings, e.g. only parts of an object are visible because of occlusion. On the other hand, using multiple large crops enables us to learn representations that are invariant to object style. We extend the contrastive and invariance losses of RELIC to operate over multiple views of varying sizes.
25M SCM
1894
2PM parariders
4EPM
Figure 1: Top-1 linear evaluation aceu- racy on ImageNet using ResNet50 encoders with 1x, 2x, 4x width multipliers and a ResNet200 with 2x width multiplier.
Empirically we show RELICv2 achieves state-of-the-art performance in self-supervised learning on wide range of ResNet architectures (ResNet50, ResNet 50 2x, ResNet 50 4x, ResNet101, ResNet152, ResNet200, ResNet 200 2x). As shown in figure [ RELICV2 is the first self-supervised representation learning method that outperforms a standard supervised baseline on linear ImageNet evaluation across wide range of ResNet architectures. On top-1 classification accuracy on Ima- geNet, RELICV2 achieves 77.1% with a ResNet50, while with a ResNer200 2x it achieves 80.6%. Furthermore, RELICV2 learns representations that achieve state-of-the-arr performance on wide range of downstream tasks and datasets such as semi-supervised and transfer learning, robustness and out-of-distributtin generalisation. Although using ResNets, RELICv2 demonstrates compara- ble performance to recent vision transformers (figure 6). These strong experimental results across different vision tasks, learning settings and datasets showcase the generality of our proposed repre- sentation learning method. Moreover, we believe that the concepts and results developed in this work could have important implications for wider adoption of self-supervised pre-training in variety of domains as well as the design of objectives for foundational machine learning systems.
Summary of contributions, We tackle the question "Can we outperform supervised learning with- out labels on ImageNet?". We propose to extend the self-supervised method RELIC [Mitrovic et al.) 202T with inductive biases to learn more informative representations. We develop fully unsuper- vised saliency masking pipeline and use multiple data views of varying sizes to encode these induc- tive biases. The resulting method RELICv2 achieves state-of sfte-offtheear performance in self-supervised learning across a wide range of ResNet architectures of different depths and width. Furthermore, RELICV2 is the first self-supervised representation learning method that outperforms a standard supervised ResNet50 baseline on linear ImageNet evaluation across 1 Lx, 2x and 4x ResNet50 vari-
2

---

Under review as a conference paper at ICLR 2023
ants well larger ResNet architectures such as ResNet101, ResNet152 and ResNet200 also highlight the generality of RELICv2 through state-of-the-ar ahe self-suuerrviee performance on transfer learning, semi-supervised learning, and robustness and out-of-distribution generalization.
] We
METHOD
2.1 BACKGROUND
Representation Learning via Invariant Causal Mechanisms (RELIC) [Mitrovic et al., 2021] learns representations by comparing two differently augmented data views through two mechanisms instance classification and invariant prediction. As common in prior work, RELIC tackles the instance classification problem with a contrastive objective, i.e. it learns representations by maxi- mizing the similarity between two differently augmented views of the same datapoint (positives), while minimizing the similarity of that datapoint with other datapoints (negatives). In addition to that, RELIC introduces an invariance objective which ensures that the distribution of similarities of datapoint is invariant across augmentations of that datapoint.
Given a randomly sampled batch of datapoints {>r) with N the batch size, RELIC learns an encoder that outputs the representation 2, i.e. 2, (x;)) Following [Chen al. 2020a Grill iet al. 2020), RELIC creates two views of the data by applying two distinct augmentations randomly sampled from the data augmentation pipeline proposed in [Chen et al. 2020a] i.e. L, T; this
yields two augmented batches {=]]/= N and { Ji=1. To solve the instance classification problem, RELIC maximizes the following probability
gr (+(iine)))
p(x):=4) e. (1110)) EngeNten e (stag)
where (,,,,, (x(f(x;)), g(g(x)) measures the similarity between representations with the temperature parameter. RELIC adopts the target network setting of [Grill et al. 2020 such that and g have the same architecture, but the weights of g are an exponential moving average of the weights of f: also, h and g are multi-layer perceptrons with h playing the role of the composition of the projector and predictor from [Grill et al. 2020] and y being the exponential moving average of the projector network. Nax,) represents the set of negatives; uniformly randomly sample small number of points from the batch to serve as negatives following [Mitrovic et al.) 2020).
In order optimize for invariant prediction, RELIC introduces invariance loss defined as the Beilback-Leibler divergence between the likelihood of the two augmented data views as
DRL (p(()( (p(x))((((())
(e log logp(z);a - -Epettie") log paal at).
(2)
The invariance loss enforces that the similarity of f(x!) and f(at') relative to the positives and negatives is the same; the stop gradient operator, sgl-), does not affect the computation of the KL- divergence but avoids degenerate solutions during optimization [Xie et al. 2019). Taken together, RELIC learns representations by optimizing the following loss
N
Σ
log logp(at:a + (
(3)
UPNT f=1
with 3 scalar weighting the relative importance of the contrastive and invariance.
2.2 RELICV2
We extend Representation Learning via Invariant Causal Mechanisms (RELIC) [Mitrovic et al. 2021] to include additional inductive biases that prevent the learning of spurious features which are hurting the learning of informative representations. We encode these inductive biases through novel fully unsupervised saliency masking pipeline well as scaling up RELIC to accommodate multiple data views of varying sizes; we call this method RELICv2 which we highlight in figure 2
2Concurent work in [Lee et al. 2021 outperforms the same standard supervised haseline only ResNet50 2x encoder.
3

---

Under review as a conference paper at ICLR 2023
lerge views
S smail views
EMA
Figure 2: (a) RELICv2 uses saliency masking as part of the data augmentations pipeline and views of various sizes to learn representations that are invariant to spurious correlations. Note that the L (differently augmented) large views are passed through both the encoder and target network, while the small S views are only passed through the encoder. The learning objective is computed by comparing each of the large and small view passed through the encoder with each large view passed through the target. (b) Learning objective used for each comparison that combines the contrastive (instance discrimination) loss, i.e the cross-entropy (x-entropy) loss computing the similarity scores and the invariance loss, i.c. the Kullback-Leible (KL) divergence between the similarity scores.
Saliency masking. To localize the semantically relevant parts of the image, we propose to use saliency masking. In order to separate foreground from background, we develop a fully unsuper- vised saliency masking pipeline that does not use any additional data. We learn saliency masking network without access to any labels or any additional data outside of the ImageNet training dataset which we also use for pre-training the representation. We start by using a small subset of Ima- geNet (2500 randomly selected images) and compute initial estimates of saliency masks using a number of handcrafted methods. Specifically, we use Robust Background Detection [Zhu et al., 2014), Manifold Ranking (Yang et 2013), Dense and Sparse Reconstruction [Li et al. [0133] and Markov Chain [Jiang et 2013]. Next, we use a ResNet50 2x network pretrained on ImageNet in self-supervised way the initial saliency detection network. We adopt the self-supervised refine- ment mechanism from DeepUSPS [Nguyen et al. 2019) and incrementally refine the handcrafted saliency masks using a self-supervised objective. Subsequently, the saliency detection network is trained by fusing the refined handcrafted saliency masks. The trained saliency detection network is randomly used for computing the saliency masks in our pipeline. During RELICv2 pre- pre-training, apply the saliency mask to input images with a small probability Pm- By learning from images where the background has been removed RELICV2 is learns representations that focus on foreground fea- tures and are robust to background changes as well as any spurious features that might be found in the background. For more details see appendix.
Views of varying sizes. We propose to enforce invariance over multiple randomly augmented data views of varying sizes to learn more informative representations. We propose use large num- ber of views encoding the whole randomly augmented image as well as a small number of smaller views which contain only a portion of the randomly augmented image As the number of views of the whole image increases so does the variation of object styles seen during training as each view represents a random augmentation manipulating the style of the image. Thus, explicitly enforcing invariance over an increasing set of object styles enables us to learn representations which are in- creasingly invariant to the spurious changes in object style. Incorporating small views which are random crops of the original image into the learning serves two purposes. First, these views represent a small part of the original image, it is likely that some parts of the objects of interest might be occluded. Thus, we learn representations through invariance, using these small views enables us to learn representations which are more robust to object occlusions, a common issue in real-world data. Note that it is important to use only small number of small views (in comparison to the number of large views) as using too many small views can result in representations that fail to encode important features as different features might be occluded in different small views and we are learning representations by enforcing invariance Second, we hypothesize that small crops
Note that DeepUSPS cannot be directly applied in our setting as it relies on labelled information and additional datasets (CityScapes) [Cordts et al. 2016
Most other methods use only 2 data views the whole image.
Foor this purpose, we use only small views in our experiments and provide an ablation analysis the different number of large and small views.
4

---

Under review as a conference paper at ICLR 2023
play synergistic role to saliency masking taking small crop of the image is likely to remove potentially large parts of the background; see analysis section for some experimental validation o
Objective. Let o be large size view and o small size view under augmentation t ~ Tsal respec- tively. RELICv2 optimizes a combination of the contrastive and invariance losses across differently augmented large and small views. In particular, RELICv2 optimizes the following objective:
C
Σ
(-
looggpp; : (1th)) 3DKL(p(a)) (p(() +
(4)
Σ 15$58
logg(aaf;
BDLL (p(a, Dp(o
with l1. ~ Taa and la ~ T randomly selected data augmentations, and L and S the number of large and small crops, respectively. Note that we leverage the differently sized data views in different ways for learning representations. We use the large views both for updating the encoder f as well as for computing learning targets through the target network y. i.e. x; appears on both sides of p. On the other hand, we only use the small views for updating the encoder f and not as learning targets, ic. कद appears only on the left hand side of p. c.f. equation We do not use small views as learning targets as potentially informative parts of the image might be occluded and as such the corresponding features removed from the representation. For the precise architectural and implementation details, as well as a pseudo-code for RELICV2 see the appendix.
OUTPERFORMING SUPERVISED LEARNING ON IMAGENET
We use RELICV2 to pretrain representations on the train- ing set of the ImageNet ILSVRC-2012 dataset [Rus- sakovsky et al., 2015) without using labels. As our use the supervised ResNet50 model used through the representation learning literature
Table 1: Top-1 accuracy (in %) under linear evaluation on the ImageNet test set for ResNet50 encoder set for dif- ferent representation learning methods.
supervised baseline
Method
Top-1
(c.f. (Chen et al. 2020at Grill et al.) 2020; Caron et al. 2020 Dwibedi et al. 2021). This model is trained with a cross-entropy loss, a cosine learning rate schedule and full access to labels using the same set of data augmen- tations as proposed by [Chen et al. [020aa]. While the
Supervised
Chen et
2020a
76.5
SimCLR
Chen ml.
3020
69.3
MoCo v2
Chen.etall
2020b
71.1
recent work of (Wightman et al. [021] proposes a series of elaborate optimization and data augmentation tricks to improve the performance of supervised ResNet50 ImageNet, we leave the application of these heuristics for future work. Instead, in this work, focus fair like- for-like comparison between self-supervised methods and supervised learning
InfoMin Aug.
Tian
2020
73.0
BYOL
Gt
2020
74.3
RELIC
Minovicc
74.8
SwAV
Caron ctal
2020
75.3
NNCLR
Duibodi
75.6
C-BYOL
Lee et 2021
75.6
RELICv2 (ours)
77.1
We evaluate RELICV2's representations by training linear classifier on top of the frozen represen-
tation according the procedure described in [Chen et al. 2020a Grill al. 2020 Dwibedi al.)
2 and the appendix. From [ for the ResNet50 encoder, we see that RELICv2 outperforms all previous state-of-the-ar self-supervised approaches by a significant margin in terms of top-1 accuracy on the ImageNet test set. Remarkably, RELICV2 outperforms the standard supervised baseline in terms of t accuracy despite using no label information to pretrain the representation.
In table 2 we also note that RELICv2 outperforms the supervised baseline under varied set of ResNet encoders of different sizes, spanning ResNet50, ResNet101, ResNet152 and ResNet200. Refer to the appendix for additional results and more details. Overall, our work can be viewed as a first step in surpassing supervised learning without access to labels across many different ResNet architectures for the image classification task. Furthermore, RELICv2 also performs competitively to the latest vision transformer architectures at similar parameter counts (see figure 6
6 'Nte that saliency masking is not perfect at removing the background and struggles especially settings where there is little color contrast between the background and foreground.
NNoe that using saliency masking to create hackground augmentations for training the supervised baseline does improve performance highlighted in Ryali al. 2021b
5

---

Under review as a conference paper at ICLR 2023
Table 2: Top-1 acc, (in %) under linear evaluation on ImageNet for different ResNet architectures.
Method
ResNet50 2x
ResNer50 4x
ResNet101
ResNet152
ResNer200
ResNer200 2x
Supervised
77.8
78.9
78.0
79.1
79.3
80.1
BYOL
Ond aaadde MA0
77.4
78.6
76.4
77.3
77.8
79.6
RELICY (ours)
79.0
79,4
78.7
79.3
80.6
Scaling. Figure shows the ImageNet linear evalua- tion accuracy obtained by representations learned using RELICv2 as a function of the number of images seen during pre-training using the ImageNet training set. It can be seen that in order to reach 70% accuracy the ResNet50 model requires approximately twice the num- ber of iterations as the ResNer295 model. The ResNer295 has approximately 3.6x the number of parameters as the ResNe150 (87M vs 24M, respectively). This finding is in accordance with other works which show that larger mod- els are more sample efficient (i.c. they require fewer sam- ples reach given accuracy) [Zhai et al. 2021].
a1e was
Figure 3: ImageNet accuracy obtained by RELICv2 as a function of number of images seen during pre-training for several of ResNet architectures (Num- ber of model parameters in brackets).
3.1 ANALYSIS
Class concentration. To quantify the overall structure of the learned latent space, we examine within- and between-class distances. Figure 4 compares the distribution of ra- tios of between-class and within-class fz-distances of the representations of the Ima- geNet test set learned by RELICv2 against those learned by the supervised baseline A larger ratio means that the representation is better con- centrated within the corresponding classes as well as better separated between classes and therefore more easily linearly separable (c.f. Fisher's 's linear discriminants (Friedman et al. 2009)). We see that RELICv2's s distribution is shifted to the right, i.c. has higher ratio compared to the supervised baseline, suggesting that the better representations have been learned. For additional analysis of the distances between learned representations of closely related classes, refer to the appendix.
ReLICv2
Supervised
Denaty
0.4 0.6
discriminat ratio
Figure 4:
Distribution of the lin- ear discriminant ratio: the ratio of between-class distances and within- class distances of embeddings com- puted on the ImageNet validation set.
Views of varying sizes. Most prior work uses 2 views of size 224 x 224 to learn representations. RELICV2 proposes instead the use of a large number of views of size 224 224 combined with a small number smaller views of size 96 x 96. We ablate the use of different numbers of large and small views in RELICv2 using only SimCLR augmentations (i.e. without saliency masking). We report below the top-1 ImageNet test performance (under linear evaluation) of ResNer50 pretrained for 1000 epoch on ImageNet; L,, S] denotes using L large views and small views.
Views [2, 0]
[2 2]
12, 6]
14, 0]
[4, 2]
[6. 2]
[8, 2]
Top-1
74.8
76.2
76.0
75.5
76.8
76.5
76.5
There seems be a performance plateau going beyond 6 large views and a slight performance penalty going beyond 4 large views. For small views, we also observe performance penalties going beyond 2 small views, while there is a significant performance boost going from no small views to 2 small views, i.e. +1.3% in the case of 4 large views. This is almost double the performance improvement one gets from adding large views, i.e the difference between [2, 0J and [4, 0j of +0.7% Thiss supports our hypothesis that small views significantly contribute to learning more robust representations, but that we only need a small number of them to learn informative representations. Note that our of large and small views is exactly the opposite compared to SWaV (Caron ct al. 2018) which argue for using smaller views as computationally less expensive alternatives to large views and rely heavily on using large number of small views; in particular, they argue for using 2 large views and small views.
Both RELICv2 and the standard supervised baseline were trained on the ImageNet training set.

---

Under review as a conference paper at ICLR 2023
Saliency masking. To isolate the contribution of saliency masking, measure the performance gain (in terms of top-1 accuracy under linear evaluation on ImageNet) when applying saliency mask- ing to just 2 large views. This improves performance from 74.8% to 75.3%, i.. gain of +0.5% which is a boost comparable to having two additional large views (see above). We also explored using different datasets for pretraining our unsupervised saliency masking pipeline well vary- ing the probability with which we apply saliency masking to the inputs and found that RELICv2 is robust these choices; see the appendix for details.
Top-1 64.5%
To assess the importance of enforcing invari- ance over background removal and object styles, we ex- tend SimCLR to use saliency masking and large views and small views, and compare to RELICV2. We train both methods for 100 epochs and report top-1 accuracy on ImageNet. As we can see from the table, invariance- based methods profit from multiple views and saliency masking significantly more than purely contrastive methods. Thus, invariance plays crucial role in learning better representations.
Invariance.
SimCLR SimCLR
cher
saliency
66.2%
RELIC Miswii
66.2% (+1,7%)
RELICVZ (ours)
67.5% (+6.4%)
ADDITIONAL EXPERIMENTAL RESULTS
In addition to outperforming the supervised baseline across a wide range of different ResNet ar- chitectures on ImageNet classification, RELICv2 also achieves state-of-the-ar performance wide range of downstream tasks and datasets. We showcase the generality and wide applicability of RELICV2 transfer, semi-supervised. robustness and out-of-distribution (OOD) generalization tasks when pretraining the representations on ImageNet, but also on the much larger and more com- plex Joint Foto Tree (JFT-300M) dataset (Hinton et al.) 20155 Chollet, 2017, Sunn al.) 2017). We report here the results on the transfer, robustness and OOD generalization downstream tasks and on the JFF-300M dataset. For a complete set of results, in particular in the sunissupervieed setting and on JFT-300M, and experimental details refer to the appendix.
4.1 ROBUSTNESS AND OOD GENERALIZATION
We evaluate the robustness and out-of-distribun (OOD) generalization of RELICv2 on a wide variety of datasets. We use ImageNetV2 [Recht al. 2019 and ImageNet- (Hendrycks and Diet- terich 2019 to evaluate robustness, while ImageNet- R Hendrycks et al.) [2021]. ImageNet-Skette Wang et al. 2019) and ObjectNet [Barbu et al. 2019] are used to evaluate OOD generalization. We evaluate the representations of a standard ResNet50 encoder under linear evaluation akin to i.e.
we train a linear classifier on top of the frozen representation using the labelled ImageNet training set; the test evaluation is performed zero-shot, i.e no training is done on the above datasets. In table 3 we see that RELICv2 learns more robust representations outperforming both supervised perfor- mance and competing self-supervised methods ImageNetV2 and ImageNet-C. RELICv2 also learns representations that outperform competing self-supervised methods while being on par with supervised performance in terms of OOD generalization.
Table 3: Top-1 Accuracy (in %) under linear evaluation on ImageNetV2 and ImageNet-C (robust- ness), and ImageNet-R (IN-R), ImageNet-Skeech (IN-S) and ObjectNet geNetv2 has three variants matched frequency (MF), Threshold 0.7 (T-0.7) and Top Images (TT). The results for ImageNet-C (IN-C) are averaged across the 15 different corruptions.
Ima-
Robustness
OOD Generalization
Method
MF
T-0.7
Ti
IN-C
IN-R
IN-S
ObjectNet
Supervised
65.1
73.9
78.4
40.9
24.0
6.1
26.6
SimCLR [Chen et al. 2020a
53.2
61.7
68.0
31.1
18.3
3.9
14.6
BYOL Grill al. 2020
62.2
71.6
77.0
42.8
23.0
8.0
23.0
RELIC Mitrovic et al.
2021
63.1
72.3
77.7
44.5
23.8
9.1
23.8
RELICV2 (ours)
65.3
74.5
79.4
44.8
23.9
9.9
25.9
4.2 TRANSFER TO OTHER CLASSIFICATION DATASETS AND SEMANTIC SEGMENTATION
Classification. We perform linear evaluation and fine-tuning on a wide range of classification benchmarks. We follow established evaluation protocols from the literature described in the ap-
7

---

Under review as a conference paper at ICLR 2023
pendix and report standard metrics for each dataset on held-out test sets. Figure compares the transfer performance of BYOL, NNCLR and RELICv2 pre-trained representations relative to su- pervised pre- tre-training. RELICv2 improves upon both the supervised baseline and competing meth- ods, performing best on 7 out 11 tasks. RELICv2 has an average relative improvement of over 5% across all tasks over double that of closest competing self-supervised method NNCLR. For detailed results on linear evaluation and fine-tuning see appendix.
Figure 5: Transfer performance relative to the supervised baseline (a value of 0 indicates equal performance supervised).
Semantic Segmentation. We evaluate the ability of
Table 4: Transfer performance
RELICv2 to facilitate successful transfer of the learned representations to the PASCAL [Everingham et al. [2010]
mantic segmentation tasks.
Method
PASCAL
Cityscapss
and Cityscapes [Cordis et al. 2016] semantic segmenta-
BYOL Grill at. 2020
75.7
74.6
tion tasks. In accordance with [ee et al. 2019). we use the RELICv2 ImageNet represstation to initialise fully
RELIC Mitrovic 2021
72.7
75.0
ReLICVI
77.9
convolutional backbone, which we fine-tune on the PASCAL train aug2012 2 set for 45 epochs and report the mean intersection over union (mloU) on the va12012 set. The fine-tuning on Cityscapes is done on the train_fine set for 160 epochs and evaluated on the val_fine set. From table 4 we see that RELICv2 significantly outperforms BYOL on both datasets, and also provides an improvement over the original ReLIC method. Interestingly, RELICv2 outperforms even DetCon [Hénaff et al. 2021 a method specifically trained for detection, on PASCAL where achieves 77.3 (mloU).
4.3 PRETRAINING JOINT FOTO TREE (JFT-300M)
We also test how well RELICV2
Table 5: Top-1 accuracy (in %) on ImageNet when learning representations using the JFT-300M dataset.
scales to much larger datasets by pretraining representations using the
Top-1 3000 epochs
Joint Foto Tree (JFT-300M) dataset
Method
1000 epochs
50000 epochs
which consists of 300 million images from more than 18k classes [Hinton
RELIC Maseico
61.5
BYOL Saleral ban
67.0
67.6
67.9 (5000 epochs)
Divide and Conmist
67.9
69.8
70.7 (4500 opochs)
al. 2015 Chollet 2017 Sun al.]
(ours)
(5000 epochss
2017 We then evaluate the learned
the ImageNet test set under the same linear evaluation protocol as described in section 3 We compare RELICv2 against BYOL and Divide and Contrast (Tian et al.) [021], method that was specifically designed to handle large and uncurated datasets and represents the cur- rent state-of-art in self self-supervised JFT-300M pretraining. Table [5] reports the top-1 accuracy when training the various methods using the standard ResNet50 architecture as the backbone for different number of ImageNet equivalent epochs on JFT-300M. RELICV2 improves over Divide and Contrast Tian et al. 2021) by more than 2% when training on JFT for 1000 epochs and achieves better over- all performance than competing methods while needing a smaller number of training epochs. For experimental details and additional results on JFT-300M, refer to the appendix.
representations
5 RELATED WORK
Recently, contrastive multi-view approaches have become an important area of research owing to their excellent performance in visual recognition tasks [Oord et al. 2018 Bachman et al., 2019 Chen et al. 2020a He et al., 2019 Dwibedi et al. 2021 Grill et al. 20200] Moreover, bootstrapping- based mulli-view learning has also achieved comparable performance [Grill et al. 2020). The afore-
mentioned methods implicitly enforce invariance by maximizing the similarity between views. On
8

---

Under review as a conference paper at ICLR 2023
the other hand, (Caron et al. 2020 incorporates explicit clustering step more direct way of enforeing some notion of invariance. However, neither of these strategies can be directly linked theo- retically to learning more compact representations. [Mitrovic et al. [02T]] approach invariance from a causal perspective and use an explicit invariance loss in conjunction with a contrastive objective to learn representations. They show that invariance must be explicitly enforced- -vi invariance loss in addition to a contrastive loss-in order to obtain guaranteed generalization performance.
Background augmentations have recently been proposed as part of the data augmentation pipeline in (Zha et al. 2021 and further discussed in concurrent work to our manuscript [Ryali et al. 202
In particular, Ryali et al.) 2021a b) obtain saliency masks by training an unsupervised saliency detection network using the self supervuee refinement mechanism from DeepUSPS [Nguyen et al.)
2019]. While [Ryali et al. 202 use an additional dataset, MSRA-B [Liu et al. 2010 to train their saliency network, we use subset of ImageNet to learn the saliency network thus enabling a fair comparison to prior work. RELICV2 differs from these methods in three key points. First, enforce an explicit invariance loss between view with and without the background, while [Zhao et al. 2021 Ryali et al. b02 do not have any invariance loss. Second, RELICv2 learns representations using multiple views of varying sizes, while these methods only two large views. Third, unlike in [Zhao et al. [222]] Ryali et al. 2021a bj the focus of our paper is to explore whether is possible achieve better than supervised performance on wide range of ResNet architectures, rather than to just investigate the benefits of background augmentations combined with a number of self-supervised methods on just the ResNet 50 Ix encoder. To learn the spatial structure of the image and as a computationally favourable alternative to large views, [Caron et al. 2020 propose to use multiple smaller views in addition to the two large views. In notable difference to [Caron et al. 2020]. RELICv2 employs small views to increase the representation robustness and proposes to use only a small number of small views. Additionally, RELICv2 imposes an invariance loss between pairs of large and small views, and uses more large views. Outside of contrastive learning, (Lee) jet al., 2021] take a conditional entropy bottleneck approach which results better-than-supervise performance albeit only the ResNet50 (2x) architecture (see figure I See appendix for more detailed comparison of related works.
6 DISCUSSION
We proposed novel self-supervised representation Iearn- ing method RELICv2 which learns representations by en- forcing invariance over multiple different views of vary- ing sizes and saliency masking. This enables us to learn robust representations which are more invariant to the above spurious features as evidenced by substantial im- provement over existing state-of-the-arr in our extensive experimental analysis across wide range of downstream settings, tasks and datasets, In particular, RELICv2 is the new state-of-the-art in linear evaluation, transfer learning and robustness on ImageNet across ResNet-based self- supervised methods. Moreover, RELICV2 is the first method that demonstrates that representations learned without access labels can consistently outperform standard supervised baseline on ImageNet which is first step in surpassing supervised learning.
VCM
Figure 6: Comparison of ImageNet top- accuracy between RELICV2 and re- cent vision transformer- tures (Swin (Liu et al.) 2021] is fully supervised transformer baseline).
architec-
Vision transformers (ViTs) (Dosovitskiy et al. 2020
have recently emerged as promising architectures for visual representation learning. Figure com- com- pares recent ViT-based methods against RELICV2 using variety of larger ResNet architectures. Notably, RELICV2 outperforms recent self-supervised ViT-based methods DINO (Caron et al.) 2021 and MoCov3 [Chen et al.] 2021 as well as exhibiting similar performance to ViT [Li et al. 2021 for comparable parameter counts despite these methods using more powerful archi- tectures and more involved training procedures. Our results suggest that combining the insights we have developed with RELICV2 alongside recent architectural innovations could lead to further improvements in representation learning and more powerful foundation models.

---

Under review as a conference paper at ICLR 2023
REFERENCES
Mahmoud Assran, Mathilee Caron, Ishan Misra, Piotr Bojanowski, Armand Joulin, Nicolas Ballas, and Michael Rabbat. Semi-supervised learning of visual features by non-parametrically predicting view assignments with support samples. arXiv preprint arXiv:2 13963, 2021.
Philip Bachman, Devon Hjelm, and William Buchwalter. Learning representations by maximizing mutual information across views. In Advances in Neural Information Processing Systems, pages 15509-15519, 2019.
Andrei Barbu, David Mayo, Julian Alverio, William Luo, Christopher Wang, Dan Gutfreund, Josh Tenenbaum, and Boris Katz. Objectnet: A large-scale bias-controlle dataset for pushing the limits of object recognition models. In H. Wallach, H. Larochelle, A. Beygelzimer, F. d'Alché- Buc, E. Fox, and R. Garnett, editors, Advances in Neural Information Processing Systems, volume 32. Curran Associates, Inc., 2019. URL http::tstptttcceeppitesst meerrddpiiiinnnnnn.ppeee.. www.inuu...... cc/ Paper.
Thomas Berg, Jiongxin Liu, Seung Woo Lee, Michelle Alexander, David W Jacobs, and Peter N Belhumeur. Birdsnap: Large-scale fine-grained visual categorization of birds. In Proceedings of the JEEE Conference on Computer Vision and Pattern Recognition, pages 2011-2018, 2014.
Rishi Bommasani, Drew A Hudson, Ehsan Adeli, Russ Altman, Simran Arora, Sydney von Arx, Michael S Bernstein, Jeannette Bohg, Antoine Bosselut, Emma Brunskill, et al. On the opportu- nities and risks of foundation models. arXiv preprint 10Xiv:2108.07258 2021.
Florian Bordes, Randall Balestriero, and Pascal Vincent. High fidelity visualization of what your self-supervised representation knows about, 2021. URL
09164
Lukas Bossard, Matthieu Guillaumin, and Luc Van Gool. 1ood-101-mining discriminative compo- nents with random forests. In European conference computer vision, risees 446-461 Springer 2014.
Tom B Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. arXiv preprint 2005 2020.
Mathilde Caron, Piotr Bojanowski, Armand Joulin, and Matthijs Douze. Deep clustering for unsu- pervised learning of visual features. In Proceedings of the European Conference on Computer Vision (ECCV), pages 132-149, 2018
Mathilde Caron, Ishan Misra, Julien Mairal, Priya Goyal, Piotr Bojanowski, and Armand Joulin. Unsupervised learning of visual features by contrasting cluster assignments. arXiv preprint arXiv:2006.09882. 2020.
Mathilde Caron, Hugo Touvron, Ishan Misra, Hervé Jégou, Julien Mairal, Piotr Bojanowski, and Armand Joulin. Emerging properties in self-supervised vision transformers. arXiv preprint arXiv:2104. 14294, 2021.
Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey Hinton. A simple framework for contrastive learning of visual representations. arXiv preprint arXiv:2002.05709, 2020a.
Xinlei Chen, Haoqi Fan, Ross B. Girshick, and Kaiming He. Improved baselines with momentum contrastive learning. ArXiv, 0bs/2003.04299,, 2020b.
Xinlci Chen, Saining Xie, and Kaiming He. An empirical study of training self-supervised vision transformers. arXiv preprint arXiv:2104.02057, 2021.
François Chollet. Xception: Deep learning with depthwise separable convolutions. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 1251-1258. 2017.
Ching Yao Chuang, Joshua Robinson, Lin Yen-Chen, Antonio Torralba, and Stefanic Jegelka. De- biased contrastive learning. arXiv preprint arXiv:2007.00224 2020.
10

---

Under review as a conference paper at ICLR 2023
Mircea Cimpoi, Subhransu Maji, lasonas Kokkinos, Sammy Mohamed, and Andrea Vedaldi. De- scribing textures the wild. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 3606-3613, 2014.
Marius Cordts, Mohamed Omran, Sebastian Ramos, Timo Rehfeld, Markus Enzweiler, Rodrigo Benenson, Uwe Franke, Stefan Roth, and Bert Schiele. The cityscapes dataset for semantic urban scene understanding. In Proc. of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2016.
Jacob Devlin, Ming- Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv:1810.04805 2018.
Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al. An image is worth 16x16 words: Transformers for image recognition at scale. arXiv preprint arXiv:2010. 1929, 2020.
Debidatta Dwibedi, Yusuf Aytar, Jonathan Tompson, Pierre Sermanet, and Andrew Zisserman. With little help from my friends: Nearest-neighbor contrastive learning of visual representations. arXiv preprint arXiv:2104. / 14548, 2021.
M. Everingham, L. Van Gool, C. K. Williams, J. Winn, and A. Zisserman. The pascal visual object classes (vOc) challenge. International Journal of Computer Vision, 88(2):303-338, June 2010.
Li Fei-Fei, Rob Fergus, and Pietro Perona. Learning generative visual models from few training examples: An incremental bayesian approach tested on 101 object categories. In 2004 conference computer vision and pattern recognition workshop, pages 178-178. IEEE, 2004.
Jerome Friedman, Trevor Hastie, and Robert Tibshirani. The elements of statistical learning- Springer series in statistics New York, 2009.
Jean-Bastion Grill, Florian Strub, Florent Altché, Corentin Tallec, Pierre H Richemond, Elena Buchatskaya, Carl Doersch, Bernardo Avila Pires, Zhaohan Daniel Guo, Mohammad Gheshlaghi Azar, al. Bootstrap your own latent: A new approach to self-supervised learning. arXiv preprint arXiv:2006.07733, 2020.
Kaiming He, Haoqi Fan, Yuxin Wu, Saining Xie, and Ross Girshick. Momentum contrast for unsupervised visual representation learning. arXiv preprint arXiv:1911.05722. 2019.
Dan Hendrycks and Thomas Dietterich. Benchmarking neural network robustness to common cor- ruptions and perturbations. arXiv preprint arXiv: 1903. 12261, 2019.
Dan Hendrycks, Steven Basart, Norman Mu, Saurav Kadavath, Frank Wang. Evan Dorundo, Rahul Desai, Tyler Zhu, Samyak Parajuli, Mike Guo, al. The many faces of robustness: A critical analysis of out-of-distribution generalization. In Proceedings of the IEEE/CVF International 2021.
Conference Computer Vision, pages
Geoffrey Hinton, Oriol Vinyals, and Jeff Dean. Distilling the knowledge neural network. arXiv preprint arXiv:1503.025333 2015.
Olivier J. Hénaff, Skanda Koppula, Jean- -aptiste Alayrac, Aaron den Oord, Oriol Vinyals, and João Carreira. Efficient visual pretraining with contrastive detection, 2021.
Bowen Jiang. Lihe Zhang. Huchuan Lu, Chuan Yang, and Ming-Hsuan Yang. Saliency detection via absorbing markov chain. In Proceedings of the IEEE international conference on computer vision, pages 1665-1672, 2013.
Jonathan Krause, Michael Stark, Jia Deng, and Fei-Fei. 3d object representations for fine-graing grained categorization. In Proceedings of the IEEE international conference on computer vision work- shops, pages 554-561,770 2013.
Alex Krizhevsky, Geoffrey Hinton, et al. Learning multiple layers of features from tiny images, 2009.
11

---

Under review as a conference paper at ICLR 2023
Kuang- Huei Lee, Anurag Amab, Sergio Guadarrama, John Canny, and Ian Fischer. Compressive visual representations. arXiv preprint arXiv:2109.12909, 2021.
Chunyuan Li, Jianwei Yang, Pengchuan Zhang. Mei Gao, Bin Xiao, Xiyang Dai, Lu Yuan, and Jianfeng Gao. Efficient self-supervised vision transformers for representation learning. arXiv preprint 17Xiv:210609970 2021.
Xiaohui Li, Huchuan Lu, Lihe Zhang, Xiang Ruan, and Ming-Hsuan Yang. Saliency detection via dense and sparse reconstruction. In Proceedings of the IEEE international conference on computer vision, pages 2976 2983, 2013.
Sungbin Lim, Ildoo Kim, Taesup Kim, Chiheon Kim, and Sungwoong Kim. Fast autoaugment. Advances in Neural Information Processing Systems, 32:6665-6675, 2019.
Tie Liu, Zejian Yuan, Jian Sun, Jingdong Wang, Nanning Zheng, Xiaoou Tang, and Heung- Yeung Shum. Learning detect salient object. IEEE Transactions Puttern analysis and machine intelligence, 33(2):353-367 2010.
Ze Liu, Yutong Lin. Yue Cao. Han Hu, Yixuan Wei, Zheng Zhang, Stephen Lin, and Baining Guo. Swin transformer: Hierarchical vision transformer using shifted windows. arXiv preprint arXiv:2103. 14030, 2021.
Subhransu Maji, Esa Rahtu, Juho Kannala, Matthew Blaschko, and Andrea Vedaldi. Fine-grained visual classification of aircraft. arXiv preprint arXiv: 1306.530..5111 2013.
Jovana Mitrovic, Brian McWilliams, and Melanie Rey. Less can be more in contrastive learning. PMLR, 2020.
Jovana Mitrovic, Brian McWilliams, Jacob Walker, Lars Buesing, and Charles Blundell. Repre- sentation learning via invariant causal mechanisms. In International Conference on Learning Representations (ICLR), 2021.
Due Tam Nguyen, Maximilian Dax, Chaithanya Kumar Mummadi, Thi Phuong Nhung Ngo, Thi Hoai Phuong Nguyen, Zhongyu Lou, and Thomas Brox. Deepusps: Deep robust unsupervised saliency prediction with self-supervision. Advances in Neural Information Processing Systems, 2019.
Maria-Elena Nilsback and Andrew Zisserman. Automated flower classification over a large number of classes. 2008 Sixth Indian Conference on Computer Vision, Graphics & Image Processing, pages 722-729. IEEE, 2008
Aaron van den Oord, Yazhe Li. and Oriol Vinyals. Representation learning with contrastive predic- tive coding, arXiv preprint arXiv: 180,0088,,,,3748,, 2018.
Omkar M Parkhi, Andrea Vedaldi, Andrew Zisserman, and CV Jawahar. Cats and dogs. In 2012 IEEE conference on computer vision and pattern recognition, pages 3498-3505. IEEE, 2012.
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. arXiv preprint arXiv:2103.00020 2021.
Benjamin Recht, Rebecca Roelofs, Ludwig Schmidt, and Vaishaal Shankar. Do imagenet classifiers generalize to imagenet? In International Conference on Machine Learning, pages 5389-5400. PMLR, 2019.
Joshua Robinson, Ching Yao Chuang, Suvrit Sra, and Stefanie Jegelka. Contrastive learning with hard negative samples. arXiv preprint arXiv:2010.04592, 2020.
Olga Russakovsky, Jia Deng, Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma, Zhiheng Huang, Andrej Karpathy, Aditya Khosla, Michael Bernstein, al. Imagenet large scale visual recognition challenge. International journal of computer vision, 1 115(3):211-252, 2015.
Chaitanya Ryali, David J Schwab, and Ari Morcos. Learning background invariance improves generalization and robustness in self-supervised learning on imagenet and beyond. 2021a.
12

---

Under review as a conference paper at ICLR 2023
Chaitanya K Ryali, David J Schwab, and Ari S Morcos. Characterizing and improving the robustness of self-supervised learning through background augmentations. arXiv preprint arXiv:2103.12719V,, 2021b.
Nikunj Saunshi, Orestis Plevrakis, Sanjeev Arora, Mikhail Khodak, and Hrishikesh Khandeparkar. A theoretical analysis of contrastive unsupervised representation learning. In International Con- ference on Machine Learning, pages 5628-5637, 2019.
Chen Sun, Abhinav Shrivastava, Saurabh Singh, and Abhinav Gupta. Revisiting unreasonable ef- fectiveness of data in deep learning era. In Proceedings of the IEEE international conference computer vision, pages 843-852, 2017.
Yonglong Tian, C. Sun, Ben Poole, Dilip Krishnan, C. Schmid, and Phillip Isola. What makes for good views for contrastive learning. ArXiv, abs/2005.10243, 2020.
Yonglong Tian, Olivier Henaff, and Aaron van den Oord. Divide and contrast: Self-supervised learning from uncurated data. arXiv preprint arXiv:2105.0m 08054, 2021.
Haohan Wang, Songwei Ge, Zachary Lipton, and Eric P Xing. Learning robust global representa- tions by penalizing local predictive power. Advances Neural Information Processing Sys- tems, pages 10506-10518, 2019.
Ross Wightman, Hugo Touvron, and Hervé Jégou. Resnet strikes back: An improved training procedure timm. arXiv preprint arXiv:2 10.0046, 2021.
Jianxiong Xiao, James Hays, Krista A Ehinger, Aude Oliva, and Antonio Torralba. Sun database: Large-scale scene recognition from abbey to zoo. In 2010 IEEE computer society conference on computer vision and pattern recognition, pages 3485-3492. IEEE, 2010.
Qizhe Xie, Zihang Dai, Eduard Hovy, Minh-Thang Thang Luong, and Quoc V Le. Unsupervised data augmentation. arXiv preprint arXiv: 12848, 2019.
Chuan Yang, Lihe Zhang, Huchuan Lu, Xiang Ruan, and Ming-Hsuan Yang. Saliency detection via graph-based manifold ranking. In Proceedings of the IEEE conference on computer vision and paltern recognition, pages 3166-3173, 2013.
Yang You, Igor Gitman, and Boris Ginsburg. Large batch training of convolutional networks. arXiv preprint rexiiv:7700 03888, 2017.
Fisher Yu, Vladlen Koltun, and Thomas Funkhouser. Dilated residual networks. In Proceedings of the IEEE conference computer vision and pattern recognition, pages 472-480, 2017.
Xiaohua Zhai, Avital Oliver, Alexander Kolesnikov, and Lucas Beyer. S41: Self-supervised semi- supervised learning. In Proceedings of the IEEE international conference on computer vision, pages 1476-1485, 2019.
Xiaohua Zhai, Alexander Kolesnikov, Neil Houlsby, and Lucas Beyer. Scaling vision transformers. arXiv preprint arXiv:2106.045500 2021.
Nanxuan Zhao, Zhirong Wu, Rynson WH Lau, and Stephen Lin. Distilling localization for self- supervised representation learning. Association for the Advancement of Artificial Intelligence, 2021.
Wangjiang Zhu, Shuang Liang, Yichen Wei, and Jian Sun. Saliency optimization from robust back- ground detection. In Proceedings of the IEEE conference on computer vision and pattern recog- nition, pages 2814-2821, 2014.
Wenbin Zou and Nikos Komodakis. Harf: dieraachy-assocciae rich features for salient object detection. In Proceedings of the IEEE international conference computer vision, pages 406- 414, 2015.
13

---

Under review as a conference paper at ICLR 2023
A COMPARISON BETWEEN SELF-SUPERVISED METHODS
In this review we focus on how important algorithmic choices: namely explicitly enforcing invari- ance and more considered treatment of positive and negative examples are key factors in improving downstream classification performance of unsupervised representations.
Method
Contrastive
Invariance
Positives
Negatives
SimCLR
Chen ct al.
2020a
X
aug();)
full batch
BYOL
Grill al. 2020
f2
aug(I:)
n/a
NNCLR Dwibedi
2021
X
aug(::)) nn(2r;)
full batch
MoCo
Hc ct al. 2019
X
aug(x;))
queue
SwAV
Caron et al. 2020
X
aug(#:)) me(ins), proto (1,))
proto (ara)
Debiased
Chuang al. 2020
X
aug(>:)
importance sample
Hard Negutives Robinson et al.
2020
X
aug(x:)
importance sample
ReLICv1 Mitrovic al. 2021
DEL
aug(r:)
subsample
ReLICv2 (ours)
Dx.
aug(::)) mo((n;), sal(ar.)
subsample
Table 6: The role of positives and negatives in recent unsupervised representation learning algo- rithms.
Negatives. A key observation of CCen et al. 2020a was that large batches (up to 4096) improve results. This was partly attributed the effect of more negatives. This motivated the incorpora- tion of queues that function as large reservoirs of negative examples into contrastive learning (He) al., 2019). However subsequent work has shown that naively using a large number of negatives can have detrimental effect on learning [Mitrovic et al.) [020]] [Saunshi et al.) 2019 Chuang et al.) 2020 Robinson et al., 2020]. One reason for this is due to false negatives, that is points the set of negatives which actually belong to the same latent class as the anchor point. These points are likely to have high relative similarity to the anchor under e and therefore contribute disproportionately to the loss. This will have the effect of pushing apart points belonging to the same class in repre- sentation space. The selection of true negatives is a difficult problem as the absence labels it necessitates having access reasonably good representations access to these representations, but are instead trying to learn them, there has been limited success in avoiding false and selecting informative negatives. This phenomenon explains the limited success of attempts to perform hard negative sampling.
begin with. As do have
Subsampling-baat approaches have been proposed avoid false negatives via importance sam- pling to attempt to find true negatives which are close to the latent class boundary of the anchor point [Robinson al. 2020). or uniformly-at-random sampling a small number of points to avoid false negatives [Mitrovic et al. 2020).
Positives and invariance. Learning representations which are invariant to data augmentation is known to be important for self-supervised learning. Invariance is achieved heuristically through comparing two different augmentations of the same anchor point. Incorporating an explicit clus- tering step is another way of enforcing some notion of invariance [Caron et al. 2020). However, neither of these strategies can be directly linked theoretically to learning more compact representa- tions. More rigorously [Mitrovic et al., [021] approach invariance from causal perspective. They show that invariance must be explicitly enforced- -va an invariance loss addition to the con- trastive loss in order to obtain guaranteed generalization performance. Most recently (Dwibedi et al. 2021] and [Assran et al.) 2021] use nearest neighbours to identify other elements from the batch which potentially belong to the same class as the anchor point.
providos detailed comparison in terms how prominent representation learning methods utilize positive and negative examples and how they incorporate both explicit contrastive and invari-
14

---

Under review as a conference paper at ICLR 2023
ance losses. Here au; refers to the standard set of SimCLR augmentations [Chen et al. 2020a nn(x;) refers to a scheme which selects nearest neighbours of 24, mc(x;) are multicrop au augmenta- tions (c.f. [Caron et al. 2020)). proto" (æt) and proto (zz) refer to using prototypes computed via an explicit clustering step c.f. CCron et al. 2020J. Finally, sal(2;; refers to a scheme which com- putes saliency masks of r; and removes backgrounds described in section Note that SwAV first computes a clustering of the batch then contrasts the embedding of the point and its nearest cluster centroid (proto against the remaining K 1 cluster centroids (proto invariance is implicitly enforced in the clustering step.
15

---

Under review as a conference paper at ICLR 2023
B PRETRAINING ON IMAGENET IMPLEMENTATION DETAILS AND ADDITIONAL RESULTS
Similar previous work [Chen al.
2020a Grill al. 2020) minimize objective using
the LARS optimizer (You et al. 2017 with a cosine decay learning rate schedule without restarts. Unless otherwise indicated, we train our models for 1000 epochs with a warm-up period of 10 epochs and a batch size of [B] = 4096. In our experiments, we use 4 views of the standard size 224 224 and views of the smaller size 96 96 each coming from an image augmented by a different randomly chosen data augmentation; the smaller size views are centered crops of the randomly augmented image. For a detailed ablation analysis on the number of large and small crops see section 3.1
B. LINEAR EVALUATION
Following the approach of [Chen et al., 2020a Grill et al. 2020 Caron et al. 2020 Dwibedi et al,, 2.21]. use the standard linear evaluation protocol on ImageNet. We train linear classifier on top of the frozen representation which has been pretrained, i.e. the encoder parameters as well as the batch statisties are not being updated. For training the linear layer, we preprocess the data by applying standard spatial augmentations, i.c. randomly cropping the image with subsequent resizing to 224 x 224 and then randomly applying a horizontal flip. At test time, we resize images to 256 pixels along the shorter side with bicubic resampling and apply 224 x 224 center crop to it. Both for training and testing, after performing the above processing, we normalize the color channels by substracting the average channel value and dividing by the standard deviation of the channel value (as computed on ImageNet). To train the linear classifier, we optimize the cross-entropy loss with stochastic gradient descent with Nestorov momentum for 100 epochs using batch size of 1024 and a momentum 0.9; we do not use any weight decay or other regularization techniques.
In the following tables, report the top-1 and top- 5 accuracies of different methods under a varied set of ResNet encoders of different sizes, spanning ResNe150, ResNet101, ResNet152 and ResNet200 and layer widths of 1 1x, 2x and 4x. ResNet50 with 2x and 4x wider layers has 94 and 375 million parameters, respectively. ResNet101, ResNet152, ResNet200 and ResNer200 2x have 43, 58, 63 and 250 million parameters, respectively.
Method
Top-1
Top-5
Supervised
Chan
2110ss
76.5
93.7
SimCLR
et 2020a
69.3
89.0
MoCo v2 Chen ct 20206
71.1
InfoMin Aug.
Tinnetal. 2020
73.0
91.1
BYOL
2020
74.3
91.6
RELIC Minrstc al. 201
74.8
92.2
SWAV
Caron al. 2026
75.3
In table R we present results under linear evalua- tion on the ImageNet test set a varied set of ResNet architectures; we compare against different unsuper- vised representation learning methods and use as the supervised baselines the results reported in [Chen] ct al. 2020al Grill et al.) 2020). Note that the su- pervised baselines reported in [Chen et al. 2020a] are extensively used throughout the self-supervised literature in order to compare performance against supervised learning. For architectures for which supervised baselines are not available in [Chen et al.) 2020a]. we use supervised baselines reported in Grill et al. 2020 which use stronger augmentations for training supervised models than (Chen et 2020a] and as such do not represent a direct like-for-lik- comparison with self-supervised methods. Across this varied set of ResNet architectures, RELICv2 outperforms all competing self- supervised methods while also outperforming the supervised baselines in all cases with margins up 1.2% absolute terms.
ANCLR
Doibediet al. 2021
75.6
92.4
C-BYOL
Lee et 2021
75.6
92.7
RBLICv2 (ours)
77.1
93.3
Table 7: Top-1 and top-5 accuracy (in %) un- der linear evaluation on the ImageNet test set for a ResNet50 encoder set for different rep- resentation learning methods.
16

---

Under review as a conference paper at ICLR 2023
Method
Top-1
Top-5
Method
Top-1
Top-S
Supervised Chen al.
MoCo
2020a
77.8
65.4
Supervised
Chen al.
2020a
78.9
SimCLR
BYOL Ghill at. 2020
Clene al.
2020s
77.4
74.2
93.6
92.0
MoCo Ho
SimCLR Chen ci
2019
2020a
76.5
68.6
93.2
SWAV
77.9
SWAV Carun
C-B B-BOL
Loe 2021
2020
78.8
77.3
94.5
BYOL Crill et al.
RELICV v2 (ours)
2020
78.6
79.4
94.2
94.3
RELIC (ours)
79.0
94.5
(b) ResNet50 1x encoder.
(a) ResNe150 2x encoder.
Method
Top-1 1
Top-5
Method
Top-1 1
Top-5
Supervised Grill 2020
78.0
94.0
Supervised Grill
79.1
94.5
BYOL Grill al 2020
76.4
93.0
BYOL Grill al 2020
77.3
93.7
RELICvZ (ours)
78.7
94.4
CBLICV2 (ours)
79.3
94.6
(c) ResNet101 encoder.
(d) ResNet152 encoder.
Method
Top-1 1
Top-5
Method
Top-1 1
Top-5
Supervised Grill et.l.
2020
79.3
94.6
Supervised Grill et 2020
80.1
95.2
BYOL Grill all 2020
77.8
93.9
BYOL Grill al 2020
79.6
94.8
RELICV2 v2 (ours)
79.8
95.0
RELICVZ (ours)
80.6
95.2
(e) ResNet200 encoder.
(f) ResNer200 2x encoder.
Table 8: Top-1 and top-5 accuracy (in %) under linear evaluation on the ImageNet test set for varied set of ResNet architectures.
B.2 SEMI-SUPERVISED LEARNING
We further test RELICV2 representations learned on bigger ResNet models in the semi-supervised setting. For this, we follow the semi-supervised protocol as in [Zhai al. 2019: Chen et al. 20200a] Grill et al. 2020; Caron et al., 2020). First, we initialize the encoder with the parameters of the pretrained representation and we add on top of this encoder a linear classifier which is randomly initialized. Then we train both the encoder and the linear layer using either 1% or 10% of the ImageNet training data; for this we use the splits introduced in [Chen et al. [020a which have been used in all the methods we compare to [Grill et al. 2020: Caron et al. 2020 Dwibedi et al. 2021 Lee et al., [221]. For training, we randomly crop the image and resize
224 224 and then randomly apply a horizontal flip. At test time, we resize images to 256 pixels along the shorter side with bicubic resampling and apply 224 x 224 center crop to it. Both for training and testing, after performing the above processing, average channel value and dividing by the standard deviation of the channel value (as computed on ImageNet). Note that this is the same data preprocessing protocol protocol. To train the model, we use a cross entropy loss with stochastic gradient descent with Nesterov momentum of 0.9. For both 1% and 10% settings, we train for 20 epochs and decay the initial learning rate by a factor 0.2 at 12 and 16 epochs. Following the approach of [Caron] al. 2020), we use the optimizer with different learning rates for the encoder and linear classifier parameters. For the 1% setting. we use batch size of 2048 and base learning rates of 10 and 0.04 for the linear layer and encoder, respectively; we do not use any weight decay or other regularization technique. For the 10% setting, we use batch size of 512 and base learning rates of 0.3 and 0.004 for the linear layer and encoder, respectively; we use a weight decay of le 5, but do not use any other regularization technique.
normalize the color channels by substracting the
in the linear evaluation
T and top-5 accuracy on the ImageNet test set is reported table RELICV2 outperforms both the standard supervised baseline and all previous state-of-the-ar self-supervised methods when us- ing 10% of the data for fine-tuning. When using 1% the data, only C-BYOL performs better than RELICV2. From table 10 we that RELICv2 outperforms competing self- supevvupee methods on ResNet50 2x in both the 1% and 10% setting. For larger ResNets. ResNet50 4x and ResNet200 2x, RELICV2 is state-of-the-ae with respect to top-1 accuracy for the low-data regime of 1 %. On
17

---

Under review as a conference paper at ICLR 2023
Method
Top-
Top-5
1%
10%
1%
10%
Supervised Thai 2019
25.4
56.4
48.4
80.4
SimCLR Chen
48.3
65.6
75.5
87.8
BYOL Grill et
2020
53.2
68.8
78.4
89.00
SWAV
Caron etal
2020
53.9
70.2
78.5
89.9
NNCLR
Dwihedi
2021
56.4
69.8
80.7
89.3
C-BYOL
Lec ctal
2021
60.6
70.5
83.4
90.0
RELICv2 2 (ours)
58.1
72.4
81.3
91.2
Table 9:
Top-1 and top-5 aceuracy (in %) on the ImageNet test set after semi-supervised training with fraction of ImageNet labels ResNet50 encoder for different representation learning methods.
these networks for the higher data regime of 10% BYOL outperforms RELICv2. Note that BYOL trains their semi-supervised models for 30 or 50 epochs whereas RELICv2 trained only for 20 epochs. We hypothesize that longer training (e.g. 30 or 50 epochs as BYOL) is needed for RELICv2 representations on larger ResNets as there are more model parameters.
Method
Top-1
Top-5
1%
10%
1%
10%
ResNe150 encoder
SimCLR Chen
58.5
71.7
83.0
91.2
BYOL Grill at 200
62.2
73.5
84.1
91.7
RELICV2 v2 (ours)
64.7
73.7
85.4
92.0
ResNer50 4x encoder
SimCLR Chen et
2020a
63.0
74.4
85.8
92.6
BYOL Grill ot al. 2020
69.1
75.7
87.9
92.5
RELICV2 v2 (ours)
69.5
74.6
87.3
91.6
ResNer200 2x encoder
BYOL Grill et al. 2020
71.2
77.7
89.5
93.7
RELICV2 (ours)
72.1
76.4
89.5
93.0
Table 10: Top-1 and top-5 accuracy (in %) after semi-supervised training with fraction of ImageNet labels for different ResNet encoders and unsupervised representation learning methods. Results reported on the ImageNet test set.
B.3 TRANSFER
We follow the transfer performance evaluation protocol outlined in Griil et al. 2020 Chen al., 2020a]. We evaluate RELICV2 both transfer settings linear evaluation and fine-tuning. For the linear evaluation protocol freeze the encoder and train only randomly initialized linear classifier which is put on top of the encoder. On the other hand, for fine-tuning in addition to training the randomly initialized linear classifier, we also allow for gradients propagate to the encoder which has been initialized with the parameters of the pretrained representation. In line with prior work [Chen al. [2200a] Grill al.) 2020 Dwibedi et al.) [02T], J we test RELICv2 representations the following datasets: FoodTO1 [Bossard et 2014], CIFARIN 10 Krizhevsky al. 2009). CIFARTON (Krizhessky et al.] 2009), Birdsnap [Berg et al.) 2014). SUN397 (split
1) Xiao al. 2010), DTD (split 1) [Cimpoi al. 2014], Cars Krause et al. 2013 Aircraft [Maji
ct al. 2013), Pets Parkhi et al. 20122] Caltech101
Fei-Fei et al.
2004). and Flowers [Nilsback and
Zisserman 2008).
Again in line with previous methods [Chen et al. 2020a Grill et al.
2020 Dwibedi et al. 2021], for
Food101 [Bossard et al. 2014), CIFARI0 [Krizhevsky al. [2099]
CIFARIO0 Krizhevsky al. 2010), DTD (split 1) [Cimpoi we report the Top-1 accuracy the test set, and for
2009], Birdsnap [ergg et al. 2014). SUN397 (split 1) [Xiao et al.
al. 2014), and Cars [Kause al. 2013)
18

---

Under review as a conference paper at ICLR 2023
Caltech101 FFii-Fei et al. 2004 and Flow- ers [Nilsback and Zisserman 2008 we report the mean per-class accuracy as the relevant metric in the comparisons. For DTD and SUN397, only use the first split, of the 10 provided splits in the dataset as per [Chen et al. 2020a Grill et al. 2020 Dwibedi et al. 2021].
Aircraft (Maji et al. 2013) Pets Parkhi et al. 2012)
We train the training sets of the individual datasets and sweep over different values of the models hyperparameters. To select the best hyperparameters, we use the validation sets of the individual datasets. Using the chosen hyperparameters, validation data and test on the held out test data in order to obtaín the numbers reported in table
train the appropriate using the merged training and
11 We swept over learning rates {.01, 0.1, 0.2, 0.25. 0.3, 0.35, 0.4, 1., 2.}. batch sizes {128, 256, 512, 1024), weight decay between {1e-6, 6, 1e-5, 1e-4, 1e-3, 0.01, 0.1}. warmup epochs {0, 10), momentum (0.9, 0.99}, Nesterov (True, False), and the number of training epochs. For linear transfer we considered setting epochs among (20, 30, 60, 80, 100), and for fine-tuning, we also considered {150, 200, 250). for datasets where lower learning rates were preferable. Models were trained with the SGD optimizer with momentum.
As can be seen from table I1 RELICv2 representations yield better performance than both state- of-the-art self-supervised method well as the supervised baseline across wide range of datasets. Specifically. RELICv2 is best on 7 out of 11 datasets and on out datasets the linear and fine-tuning settings, respectively.
Method
Food101
CIFARIO CIFAR100 Birdsnap SUN397 Cars Aireraft DTD Pets Caltech101 Flowers
Linear evaluation:
Supervised-IN IN Ornna
72.3
93.6
78.3
53.7
61.9
66.7
61.0
74.9
91.5
94.5
94.7
SimCLR Oee Cicte
68.4
90.6
71.6
37.4
58.8
50.3
50.3
74.5
83.6
90.3
91.2
BYOL
onteraa SIOn
75.3
91.3
78.4
57.2
62.2
67.8
60.6
75.5
90.4
94.2
96.1
NNCLK Dested
93.7
79.0
61.4
62.5
64.1
91.3
ReLICV2 (OUTS)
80.6
92.8
78.2
65.4
66.2
75.1
64.8
77.4
92.4
92.8
95.6
Fine-tuned:
Random Init Chesetaa ba35d
86.9
95.9
80.2
76.1
53.6
91.4
85.9
64.8
81.5
72.6
92.0
Supervised-Ih IN Cheneta ponna
88.3
97.5
86.4
75.8
64.3
92.1
86.0
74.6
92.1
93.3
97.6
SimCLR aae B000e
88.2
97.7
85.9
75.9
63.5
91.3
88.
73.2
89.2
92.1
97.0
cnal 2000
97.8
86.1
76.3
63.7
91.6
88
76.2
91.7
93.8
97.0
ReL.CCV (OUIS)
88.7
97.7
85.3
76.7
64.7
92.3
88.7
76.9
92.2
93.2
97.9
Table 11: Accuracy (in %) of transfer performance of ResNet50 pretrained on ImageNet.
B.4 ROBUSTNESS AND OOD GENERALIZATION
The robustness and distriibuttiitoo (OOD) generalization abilities of RELICv2 representations tested on several detasets. We use ImageNetV2 [Recht et al. 2019 and ImageNet-C [Hendrycks) and Dietterich 2019) datasets to evaluate robustness. ImageNet |Recht et al. 2019 has three sets of T0000 images that were collected to have similar distribution to the original ImageNet valida- tion set, while ImageNet-C [Hendrycks and Dietterich] 2019) consists of 15 synthetically generated corruptions (e.g. blur, noise) that are added to the ImageNet validation set.
For OOD generalization we examine the performance on ImageNet-R [enddryck et al. 202 Ima- geNetSketch [Wang et al.) 2019] and ObjectNet [Barbu et al.) 2019]. ImageNet-R [Hendrycks al. 2021 consists of 30000 different renditions (e.g. paintings, cartoons) of 200 ImageNet classes, [Wang al. 2019) consists of 50000 images. 50 for each ImageNet class, of object sketches in the black-and-white color scheme. These datasets aim to test robustness to different textures and other naturally occurring style changes and are out-o--distribution the Ima- geNet training data. ObjectNet [Barbu al. 2019 has 18574 images from differing viewpoints and backgrounds compared to ImageNet.
while ImageNet-Skeree
On all datasets we evaluate the representations of standard ResNet50 encoder under linear eval- uation protocol akin to Section 3 i.e. we freeze the pretrained representations and train a linear classifier using the labelled ImageNet training set; the test evaluation is performed zero-shot, i. training done on the above datasets. As we have seen in table 3 RELICv2 learns more robust representations and outperforms both the supervised baseline and the competing self-supervised
19

---

Under review as a conference paper at ICLR 2023
methods on ImageNetV2 and ImageNet-C. We provide detailed breakdown across the different
ImageNet-C corruptions in table I2
Blur Mosion
Weather
Digital
Method
Shoc
Impuise
Defocus
Frost
Contrast
lilastic
JPEX
Supervised
StnCe
Coreod pola
29.]
26.1
17.3
22.1
20.0
27.2
46.2
59.7
53.9
24.1
43.9
BYOL
41.5
58.7
31.9
37.8
22.5
31.6
29.6
35.1
42.9
00.1
09.0
58.4
41.5
46.3
55.9
RELIC
43.4
40.7
36.6
40.5
24.5
34.3
30.3
36.6
43.8
61.4
09.5
59.5
42.8
46.8
57.3
Ral 1V2
41.6
390
31.1
99.7
22.6
35.2
34.5
40.1
46.1
64.5
71.0
60.0
44.6
46.6
58.4
Table 12: Top-1 accuracies for for Gauss, Shot, Impulse, Blur, Weather, and Digital corruption types on ImageNet-C.
20

---

Under review as a conference paper at ICLR 2023
C ANALYSIS
Class confusion. To understand the effect of the invariance RELICV2, we look at the distances between learned representations of closely related classes. Figure / illustrates the Euclidean distances between nearest-neighhour represen- tations learned by RFL.CCv2 and BYOL on ImageNet using the pro- Here
RAUO22
BYOL
in
tocol described in section pick two breeds of dog and two breeds of cat. Each of these four classes has 50 points associated with from the ImageNet validation set, ordered contiguously. Each row rep- resents an image and each coloured point in row represents one of the five nearest neighbours of the representation of that image where the colour indicates the distance between the image and the nearest neighbour. Representations which align perfectly with the underlying class structure would exhibit perfect block-diagonal structure, i.e. their nearest neighbours all belong to the same under- lying class. We see that RELICv2 learns representations whose nearest neighbours are closer and exhibit less confusion between classes and super-classes than BYO..
Figure 7: Distances between nearest-neighhour representa- tions. Each coloured point row represents one of the five nearest neighbours of the representation of that image where the colour indicates the distance between the points.
21

---

Under review as a conference paper at ICLR 2023
D PRETRAINING ON JOINT FOTO TREE (JFT-300M) IMPLEMENTATION DETAILS AND ADDITIONAL RESULTS
D.1 LINEAR EVALUATION
We test how well RELICV2 scales much larger datasets by pretraining representations using the Joint Foto Tree UFT-300M dataset which consists of 300 million images from more than 18k classes
(Hinton al. 2013 Chollet 2017: Sun et al. 2017]. We then evaluate the learned representations
on the ImageNet va validation set under the same linear evaluation protocol as described in section (3 We compare REIICV2 against BYOL and Divide and Contrast (DnC) (Tian al. 202T], J, method that was specifically designed to handle large and uncurated datasets and represents the current state-of-art in self-supervised JFT-300M pretraining. Table 13 reports the top-1 accuracy when training the various methods using the standard ResNet50 architecture as the backbone for different number of ImageNet equivalent epochs on JFT-300M; implementation details can be found in the supplementary material. RELICv2 improves over DnC by more than 2% when training JFT for 1000 epochs and achieves better overall performance than competing methods while needing a smaller number of training epochs.
Method
Epochs
Top-1
BYOL [Grill al.
2020
1000
67.0
Divide and Contrast
Tian et al.
2021
1000
67.9
RELICv2 (ours)
1000
70.3
BYOL [rrill et al.
2020
3000
67.6
Divide and Contrast
Tian al. 2021
3000
69.8
RELICv2 (ours)
3000
71.1
BYOL [Grill et al.
2020
5000
67.9
Divide and Contrast TTan al. 2021
4500
70.7
RELICv2 (ours)
5000
71.4
Top-1 accuracy (in %) on ImageNet when learning representations using the JFT-300M dataset. Each method is pre-trained on JFT-300M for an ImageNet-equivall number of epochs and evaluted on the ImageNet validation set under a linear evaluation protocol.
Table 13:
For results reported in table 13 we use the following training and evaluation protocol. To pre- train RELICv2 on the Joint Foto Tree (JFT-300M) dataset, we used a base learning rate of 0.3 for pretraining the representations for 1000 imageNet-equivalent epochs. For longer pretraining of 3000 and 5000 ImageNet-equuialent epochs, we use lower base learning rate of 0.2. We set the target exponential moving average to 0.996, the contrast scale to 0.3. temperature to 0.2 and the saliency mask apply probability to 0.15 for all lenghts of pretraining. For 1000 and 5000 ImageNet- equivalent epochs use 2.0 as the invariance scale, while for 3000 ImageNet-eeuiivaenn epochs, we use invariance scale 1.0. We then follow the linear evaluation protocol on ImageNet described in Appendix [..]] We train linear classifier on top of the pretrained representations from JFT-300M with stochastic gradient descent with Nesterov momentum for 100 epochs using batch size of 256, learning rate of 0.5 and momentum of 0.9.
D.2 TRANSFER
We evaluate the transfer performance of JFT-300M pretrained representations under the linear eval- uation protocol. For this, we freeze the encoder and train only linear classifier on top of the frozen encoder output, i.e. representation. As before in B.3 we follow the transfer performance evalu- ation protocol as outlined
2020a In line with prior work, for (Grill et al. 2020 Chen et al..
Food101 [Bossard et al., 2014], CIFARIU 10 [Krizhessky al. 2009 CIFARI00 (Krizhevsky et al.) 2009]. Birdsnap [Berg et al. 2014]. SUN397 (split 1) [iiao et al. 2010 DTD (split 1) [Cimpoi al. 2014], and Cars [Kause al. 2013] report the top-1 accuracy on the test and for Aircraft [Maji et al., 2013]. Pets Parkhi et al.] 2012). Caltech101 [Fei-Fei et al. 2004], and Flow- report the mean per-class accuracy the relevant metric in
NNlshack and Zisserman 2008
22

---

Under review as a conference paper at ICLR 2023
the comparisons. For DTD and SUN397, only use the first split, of the 10 provided splits in the dataset.
We train the training sets of the individual datasets and sweep over different values of the models hyperparameters. To select the best hyperparameters, we use the validation sets of the individual datasets. Using the chosen hyperparameters, training and validation data and test on the held out test data in order to obtain the numbers reported in table 14 We swept learning rates {.01, 0.1, 0.2, 0.25, 0.3, 0.35, 0.4, 1., 2.). batch sizes (128, 256, 512, 1024), weight decay between {1e-6, le-5, 5, le-4, 1e-3, 0.01, 0.1), warmup epochs {0. 10). momentum {0.9. 0.99). Nesterov (True, False), ), and the number of training epochs {60, 80, 100}. Models were trained with the SGD optimizer with momentum.
train the linear layer from seratch using the merged
As can be seen from table [4 longer pretraining benefits transfer performance of RELICV2. Al- though DrC (Tian et al. 2021 was specifically developed to handle uncurated datasets such as JFT-300M, we see that RELICV2 has comparable performance DnC in terms of the number of datasets with state-of-the-ar performance among self-supervised representation learning methods; this showcases the generality of RELICV2.
Method
Food101
CIFARIO
CIFAR 100 Birdsnap SUN397 Cars Aircraft DTD
Pets
Caltech101 Flowers
BYOL-55k
73.3
89.8
72.4
38.2
61.8
64.4
54.4
75.5
77
90.1
94.3
DrC-44.5k
78.7
91.7
74.9
42.1
65.0
75.3
54.1
76.6
86.1
90.2
98.2
ReLICv2- IK (ours)
77.5
90.2
72.6
47.4
64.5
74.4
62.9
77.0
84.9
92.2
94.5
ReLICv2-5k (ours)
78.3
89.9
73.0
49.4
65.6
76.9
65.5
76.8
85.1
91.4
95.7
Table 14: Accuracy (in %) transfer performance a ResNet50 pretrained on JFT under the linear transfer evaluation protocol. xk refers to the length of pretraining in ImageNet-equivalent epochs, e.g. 1k corresponds 1000 ImageNet-equiialent epochs of pretraining.
23

---

Under review as a conference paper at ICLR 2023
D.3 ROBUSTNESS AND OOD GENERALIZATION
We also tested the robustness and out-of-distribution (OOD) generalization of RELICv2 represen- tations pretrained JFT. We use the same set-up described B.4 where we freeze the pretrained representations on JFT-300M, train a linear classifier using the labelled ImageNet training set and perform zeroshot test evaluation on datasets testing robustness and OOD generalization. As in B.4 we evaluated robustness using the ImageNetV2 [Recht et al., 2019] and ImageNet-C [Hendrycks and Dietterich 2019] datasets and OOD generalization using ImageNet-R (Hendrycks al. 2021], ImageNetSketch [Wang et al., 2019] and ObjectNet [Barbu et al., 2019) datasets. We report the robustness results in table 15a and the OOD generalization results in table [15b] We notice that RELICv2 representations pretrained on JFT-300M for different number of ImageNNet-equivalent epochs have worse robustness and OOD generalization performance compared to RELICv2 repre- sentations pretrained directly ImageNet (see table 3 for reference). Given that the above datasets have been specifically constructed to measure the robustness and OOD generalization abilities of models pretrained ImageNet (as they have been constructed in relation to ImageNet), this result not entirely surprising. We hypothesize that this is due to there being a larger discrepancy be- tween datasets and JFT-300M than these datasets and ImageNet and such JFT-300M-pretraine representations perform worse than ImageNet-pretrained representations, Additionally, note that pretraining on JFT-300M for longer does not necessarily result in better downstream performance on the robustness and out-of oittt--iiitiiuttionn datasets.
Epochs
IN-R
IN-Sketch
ObjectNet
Epochs
MF
T-0.7
Ti
IN-C
1000
20.4
6.7
20.3
1000
57.6
66.7
73.0
32.9
3000
20.3
8.7
21.3
3000
58.6
67.5
73.4
32.8
5000
20.3
5.4
20.9
5000
59.1
67.3
73.3
33.5
(b) ImageNet-R,
ImageNet-Sketch and ObjectNet
(a) ImageNetv2 dataset.
datasets.
Table 15: Top-1 Accuracy (in %) under linear evaluation on the the ImageNet-R (IN-R), ImageNet- Sketch (IN-S) and ObjectNet out-of-distribution datasets and ImageNetV2 dataset for RELICv2 pre-trained on JFT-300M for different numbers of ImageNet-equivalent epochs. We evaluate on all three variants ImageNetV2 matched frequency (MF), Threshold 0.7 (T-0.7) and Top Images (TI). The results for ImageNet-C (IN-C) are averaged across the 15 different corruptions.
24

---

Under review as a conference paper at ICLR 2023
E RELICV2 PSEUDO-CODE IN JAX
Listing provides PyTorch-like pseudo-code for RELICv2 detailing how we apply the saliency masking and how the different views of data are combined in the target network setting. We also provide a direct comparison with the pseudo-code for RELIC [Mitrovic et al. [021] highlighed in
listing? 2 Note that loss-relic is computed using equation
[H]
[H]
fLo: online network: anooder comparison net
online network: erooder comparlaon_nie
g_t: tazget network: encoder comparison_net
3 g_t: target network: encoder + comparissn__et
ganma: target
coefficient
garma: target BMA coefficient
ALe: number of nagatives
R_e: numbez of negatives ..r
p_m: moak apply probability f..
load batch B samplea
for in batch: lead batch B samples
For x batch:
Apply saliyy background
remove
Apply augmertations
10
x1 - aug
x_2 wemove pawove_bcckgrouddttt
aug(x)
11
for i in range (aun_argg__viers]
# Salect ehe original
o1, o2 f_oix]), f_o(x2)
background removedd
21, t2 [_t01), f_t (x2)
13
# Image with probsbility p.m Sernoulli (p_m)
14
Compute loas between augmented vieus
15
# Do large zandon view and sugment
17
loss loss__ellc t2, a_e) loss_relic(o2, ti, a_e)
xl_i a
17
loas /=
18
ol. [) t1_ g_t(xl_)
19
loss backward()
20
for i range inum_small :
Compute grads, update online and target
22
# Do smail zandon view and augnent
setworks
xs_i aug jeroperpa (x))
(oss.bsckwardd)
24
# Small 1 views only through the online network
25
update (f
g_t
gamma
ganma) [_o
25
os_i {_olxs_i)
Listing 2: Pseudo-code for RELIC.
26
28
Compute loss between all pairs of lazge
29
for 1. in range Inum larp_larre_vvvee
30
lines Ioz in zenge (nwumlargee,....mm views) loss +- (ol_ ti_s,
P_e)
33
Compute loss between small views and large views
34
for 1 in range taun_smaa somu_smalllam seaalsmlliiii [iews) foz in zeage largee loss loss_relic(os__, ti_j,
35
36
P_e)
(nun_large_views
smcl_mmal1_.m views)
38
num _an_large_viee
39
loss /- acale
40
Compute grads, update online and target networks
backward()
43
update (f_o)
g_t ganma
q_t ganma) [_o
Listing I: Pseudo-code for RELICv2.
25

---

Under review as a conference paper at ICLR 2023
F IMAGE PREPROCESSING
F.1 AUGMENTATIONS
Following the data augmentations protocols of (Chen et al. 2020al Grill et al. 2020 Caron al., 2020 RELICv2 uses a set of augmentations to generate different views of the original image which has three channels, red r, green and blue with r, g. b E [0, 1].
The augmentations used, in particular (corresponding to aug in Listing 2 are the same as in Grill al. 2020] and are generated as follows; for exact augmentations parameters see table I6) The following sequence of operations is performed in the given order.
Crop the image: Randomly select patch of the image, between a minimum and maximum crop area of the image. with aspect ratio sampled ogg-unformmly in [3/4, 4/3]. Upscale the patch, via bicubic interpolation, to a square image size s x s.
2. Flip the image horizontally.
3. Colour jitter: randomly adjust brightness, contrast, saturation and hue of the image, in random order, uniformly by value in |-a, a) where is the maximum adjustment (speci-
fied below).
4. Grayscale the image, such that the channels are combined into one channel with value 0.2989r 0.5870g 0.1140b.
5. Randomly blur. Apply a 23 23 Gaussian kernel with standard deviation sampled uni- formly in [0.1, 2.0].
6. Randomly solarize: threshold each channel value such that all values less than 0.5 are replaced by and all values above equal 0.5 replaced with 1.
Apart from the initial step of image cropping, each step is executed with some probability to generate the final augmented image. These probabilities and other parameters are given in table [6] separately for augmenting the original image z; and the positives P(æ;). Note that we use 4 large views of size 224 224 pixels and 2 small views of 96 96 pixels; to get the first and third large views and the first small view we use the parameters listed below for odd views, while for the second and fourth large view and the second small view we use the parameters for even views.
Parameter
Even views
Odd views
Probability of randomly cropping 50%
50%
Probability of horizontal flip
50%
50%
Probability of colour jittering Probability of grayscaling Probability of blurring
80%
80%
20%
20%
100%
10%
Probability of solarization
0%
20%
Maximum adjustment a of brightness
0.4
0.4
Maximum adjustment a of contrast
0.4
0.4
Maximum adjustment a of saturation
0.2
0.2
Maximum adjustment of hue
0.1
0.1
Crop size
224
96 (small), 224 (large)
Crop minimum area
8%
5% (small), 14% (large)
Crop maximum area
100%
14% (small), 100% (large)
Table 16: Parameters of data augmentation scheme. Small/large indicates small or large crop.
F.2 SALIENCY MASKING
Using unsupervised saliency masking enables us to create positives for the anchor image with the background largely removed and thus the learning process will rely less the background to form representations. This encourages the representation to localize the objects in the image (Zhao al,
2021]
26

---

Under review as a conference paper at ICLR 2023
We develop fully unsupervised saliency estimation method that uses the self- self-supervised refinement mechanism from DeepUSPS [Nguyen et al. 2019] to compute saliency masks for each image in the ImageNet training set. By applying the saliency masks top of the large views, we obtain masked images with the background removed. To further increase the background variability, instead of using black background for the images, apply homogeneous grayscale to the background with the grayscale level randomly sampled for each image during training. We also use a foreground threshold such that we apply the saliency mask only if covers at least 5% of the image. The masked images with the grayscaled background are used only during training. Specifically, with small probability Pm we selected the masked image of the large view in place of the large view. Figure shows how the saliency masks are added on top of the images to obtain the images with grayscale background.
Figure 8: Illustration of how for each image in the ImageNet training set (left) we use our unsuper- vised version of DeepUSPS obtain the saliency mask (middle) which we then apply on top of the image to obtain the image with the background removed (right).
F.2.1 TRAINING THE SALIENCY DETECTION NETWORK TO OBTAIN SALIENCY MASKS
DeepUSPS Nguyen et al. 2019 is saliency prediction method that uses self-supervision to refine pseudo-labels from a number of handcrafted saliency methods. To obtain saliency masks for the images in ImageNet, refinement mechanism from DeepUSPS [Nguyen et al.] 2019]. To this end, we firstly sample a random subset of 2500 ImageNet images; note that the original implementation of DeepUSPS uses 2500 images from the MSRA-B dataset. We instead use randomly selected subset of the ImageNet training set of the same size to ensure fair comparison to previous work. We compute initial
build new saliency detection method that leverages the self-supervised
saliency masks for the 2500 ImageNet images using the following handcrafted methods: Robust
Background Detection (RBD) /Zhu al. 2014), Manifold Ranking (MR) (Yang et al.
2013 Dense
and Sparse Reconstruction (DSR) [L at al. [0033] and Markov Chain (MC) [Jiang et al. that these methods do not make use of any supervised label information.
2013
Note
We then follow the two-stage mechanism proposed by DeepUSPS [Nguyen et al. saliency prediction network. In the first stage, the noisy pseudo-labed from each handcrafted method iteratively refined. In the second stage, these refined labels from each handcrafted saliency method are used to train the final saliency detection network. The saliency detection network is then used to compute the saliency masks for all images in the ImageNet training set. For the refine- ment procedure and for training the saliency detection network, we adapt the publicly available code for training DeepUSPS: https::/ /tsttps:/tiiyytn http://tiiuurlrr com/ wt 1hgo3] Note that the official implemen- tation for DeepUSPS uses as backbone a DRN-network (Yu et al. [2017] which was pretrained on CityScapes [Cordts et al. 2016) with supervised labels. To be consistent with our fully-unsupervised setting, we replace this network with a ResNet50 2x model which was pretrained on ImageNet us- ing the self-supervised objective from SWaV [Caron al. 2020). We used the publicly available pretrained SWaV model from: httpt: / thub. facebookress
2019 to obtain
rch/ swav
To account for this change in the architecture, we adjust some of the hyperparameters needed for the the two-stage mechanism of DeepUSPS. In the first stage, the gpeudo-generation networks used for refining the noisy pseudo-labels from each of the handcrafted methods are trained for 25 epochs in three self-supervised iterations. We start with learning rate of le 5 which is doubled during each iteration. In the second stage, the saliency detection network is trained for 200 epochs using a learning rate of 1e 5. We use the Adam optimizer with momentum set to 0.9 and batch size of 10. The remaining hyperparameters are set in the same way as they are in the original DeepUSPS code.
27

---

Under review as a conference paper at ICLR 2023
Next, for different probabilities Pm of removing the background of the large augmented views during training, we report below the top-1 accuracy under linear evaluation on ImageNet.
Pm
0.0
0.1
0.15
0.2
0.25
Top-1
76.8
77.1
76.8
76.8
76.7
Applying the saliency masks 10% of the time results in the best performance and significantly im- proves over not using masking (pm = 0).
F.2.2 EXAMPLES OF SALIENCY MASKS
In Figure we provide random sample of the saliency masks obtained from the salieney detection network that are used as part of the data augmentations pipeline in RELICv2 pretraining. We notice that while the obtained saliency masks can, in general, segment the object from the background well, some common failure cases include the setting where the object fills the entire image (i.e there is no background), or when the object is difficult to distinguish from the background. This is why, we believe it is important only use the saliency masking with small probability during training.
28

---

Under review as a conference paper at ICLR 2023
Figure 9: Examples of saliency masks used during training ReLICv2. Each of the two columns contains randomly selected examples of image (left), mask obtained from our saliency detection networks (middle) and image with applied masks, without background grayscaling (right).
29

---

Under review as a conference paper at ICLR 2023
G ABLATIONS
In order to determine the sensitivity of 2 to different model hyperparameters, we perform an extensive ablation study. Unless otherwise noted, in this section report results after 300 epochs of pretraining. As saliency masking is one of the main additions of RELICv2 on top of RELIC and was not covered extensively in the main text, we start ablation analysis with looking into the effect of different modelling choices for it.
G.1 USING DIFFERENT DATASETS FOR OBTAINING THE SALIENCY MASKS
we used a saliency detection network trained only on randomly selected subset of 2500 ImageNet images using the refinement mechanism proposed by
In the main text in Sections 3 4.2
DeepUSPS [Nguyen al. 2019 Here we explore whether using additional data could help improve the performance of the saliency estimation and of the overall representations learnt by RELICv2. For this purpose, we use the MSRA-B dataset [Liu et al. 2010), which was originally used by DeepUSPS to train their saliency detection network. MSRA-B consists of 2500 training images for which handcrafted masks computed with the methods Robust Background Detection (RBD)
[Zhu et al.] 2014), Hicrarchy-associin
Rich Features (HS) [Zu and Komodakis, 2015]. Dense
and Sparse Reconstruction (DSR) (Li et al., 2013) and Markov Chain (MC) [Jiang et al., 2013 already available. We use the same hyperparameters
described in Section F.2 to train
saliency detection network on MSRA-B; note that these are the same hyperparameters we use for training our saliency detection network on ImageNet.
We explored whether using saliency masks obtained from training the saliency detection network on the MSRA-B affects performance of RELICV2 pre-training on ImageNet. We noticed that for RELICv2 representations pretrained on ImageNet for 1000 epochs, we get 77 77. top-1 1 and 93.3% top-5 accuracy under linear evaluation on the ImageNet validation The slight performance gains may due to the larger variety of images in MSRA-B used for training the saliency detection network, as opposed the random sample of 2500 ImageNet images that we used for training the saliency detection network directly on the ImageNet dataset.
for ResNet50 (Ix) encoder.
We also explored training the saliency detection network on 5000 randomly selected images from the ImageNet dataset and this resulted in the model overfitting, which degraded the quality of the saliency masks and resulted in RELICv2 performance of 76.7% top-1 and 93.3% top-5 accuracy on the ImageNet validation set after 1000 epochs of pretraining ImageNet training set.
The results for RELICV2 in Section D are obtained by applying the saliency detection network trained on MSRA-B to all images in JFT-300M and then applying the saliency masks to the large augmented views during training as described Section
G.2 ANALYSIS AND ABLATIONS FOR SALIENCY MASKS
Using saliency masking during RELICV2 training enables to learn representations that focus the semantically-reeevaa parts of the image, i.e. the foreground objects, and as such the learned representations should be more robust background changes. We investigate the impact of using saliency masks with competing self-supervised benchmarks, the effect of the probability Pm of applying the saliency mask to each large augmented view during training as well as the robustness of RELICv2 to random masks and mask corruptions. For the ablation experiments described in this section, we train the models for 300 epochs.
Using saliency masks with competing self-supervied methods. We evaluate the impact of using saliency masks with competing self-supervised methods such as BYOL (Grill et al., 2020). This method only uses two large augmentented views during training and we randomly apply the saliency masks, in a similar way as described in Section F2 to each large augmented view with probability Pm- We report in table 17 the top-1 1 and t accuracy under lincar evaluation on ImageNet for different settings of pm for removing the background of the augmented images. We notice that saliency masking also helps improve performance of BYOL.
Mask apply probability. We also investigate the effect of using probabilities ranging from 0 to 1 for applying the saliency mask during training for RELICy2. In addition, explore further the
30

---

Under review as a conference paper at ICLR 2023
Mask probability Pm
0.15
0.2
0.25
0.3
BYOL
Top-I Top-5
73.1
73.4
73.2
73.3
72.8
71.8
91.2
91.3
91.2
91.3
90.8
90.1
Table 17: Top-1 and top-5 accuracy (in %) under linear evaluation on the ImageNet validation set for BYOL trained using different probabilities of using the saliency mask to remove the background of the augmented images. Models are trained for 300 epochs.
effect of using different datasets for training the saliency detection network that is subsequently used for computing the saliency masks. Table 18 reports the top-1 and top-5 accuracy for varying the mask apply probability Pm between O and and for using the ImageNet vs. the MSRA-B dataset [Liu et al. 2010 for training our saliency detection network. Note that using the additional images from the MSRA-B dataset to train the saliency detection network results in better saliency masks which translates better performance when using the saliency masks during RELICv2 training.
Sal. network trained on ImageNet
Sal. network trained MSRA-B
Mask probability Pm
Top-1
Top-5
Top-1
Top-5
0
75.2
92.4
75.2
92.4
0.05
75.3
92.6
75.2
92.6
0.1
75.4
92.5
75.3
92.4
0.15
75.2
92.5
75.5
92.5
0.2
75.2
92.5
75.6
92.6
0.25
75.0
92.3
75.3
92.5
0.3
75.1
92.3
74.8
92,4
0.4
75.0
92.3
75.3
92.5
0.5
74.7
92.2
75.0
92.4
0.6
75.0
92.3
75.0
92.3
0.7
74.4
92.3
74.6
92,0
0.8
73.9
91.7
75.0
92.1
0.9
74.0
91.7
74.6
92.0
1.0
73.7
91.7
74.5
92.0
Table 18: Top-1 and top-5 accuracy (in %) under linear evaluation the ImageNet validation set for a ResNet50 (Ix) encoder set for different probabilities pm of using the saliency mask to remove background of the large augmented views during training and for using different datasets to train the saliency detection network for computing the saliency masks. Models are trained for 300 epochs.
Random masks and mask corruptions. To understand how important having accurate saliency masks for the downstream performance of representations is we also investigated using random masks, corrupting the saliency masks obtained from saliency detection network and using bounding box around the saliency masks during RELICv2 training.
We explored using completely random masks, setting the saliency mask to be a random rectangle of the image and also a centered rectangle. As ImageNet images generally consists of images with objects centered in the middle of the image, we expect that using random rectangle that centered around the middle will cover a reasonable portion of the object. Table 19 reports the performance under linear evaluation on the ImageNet validation set when varying the size of the random masks to cover different percentage areas ap of the full image. We notice that improving the quality of the masks, by using random rectangle patches instead of completely random points in the image as the mask, results in better performance. However, the performance with random masks is 1% lower than using saliency masks from our saliency detection network. As expected, using centered rectangles instead of randomly positioned rectangles masks results in better peformance.
Moreover, to test the robustness of RELICv2 to corruptions the saliency masks, we add/remove from the masks rectangle proportional to the of the saliency mask. The mask rectangle is added/removed from the image center. Table 20 reports the results when varying the area of the rectangle be added/removed cover different percentages my of the saliency masks. We notice
31

---

Under review as a conference paper at ICLR 2023
Random
Rectangle
Centered
Rectangle
Image percentage area dp
Top-1
Top-5
Top-1
Top-5
Top-1
Top-5
10%
70.8
89.9
70.9
90.3
71.3
90.1
20%
72.2
90.7
73.1
91.3
73.4
91.3
30%
72.9
91.3
73.8
91.8
73.8
91.9
40%
73.1
91.4
74.2
91.9
74.1
92.0
50%
73.3
91.5
74.0
92.0
74.3
92.0
60%
73.6
91.8
74.2
92.1
74.3
92.2
70%
73.7
91.9
74.4
92.1
74.4
92.2
80%
74.1
92.1
74.4
92.2
74.2
92.1
90%
74.1
92.2
74.4
92.1
74.2
92.2
Table 19: Top-1 I and top-5 accuracy (in %) under linear evaluation on the ImageNet validation set for a ResNet50 (Ix) encoder set for using different types of random masks that cover various percentage areas (ap) of the full image. These random masks are applied on top of the large augmented views during training with probability 0.1. Models are trained for 300 epochs.
that while RELICv2 is robust to small corruptions of the saliency mask its performance drops in line with the quality the saliency masks degrading.
Add rectangle to mask
Remove rectangle from mask
Mask percentage my
Top-1
Top-5
Top-1
Top-5
10%
75.2
92.5
75.2
92.3
20%
75.3
92.6
75.1
92.4
30%
75.1
92.3
74.7
92.2
40%
74.9
92.2
74.6
92.2
50%
74.9
92.4
74.5
92.0
60%
74.9
92.2
74.0
91.7
70%
74.8
92.2
73.6
91.7
80%
74.8
92.4
73.4
91.4
90%
74.7
92.2
73.0
91.3
100%
74.6
92.3
72.6
90.9
Table 20: Top-1 and top-5 accuracy (in %) under linear evaluation on the ImageNet validation set for a ResNet50 (Ix) encoder set for corrupting the saliency masks by adding/remove a rectangle from the image center. The rectangle is percentage (m,) of the saliency mask area (the higher the percentage the higher the corruption). The corrupted saliency masks are applied on top of the large augmented views during training with probability 0.1.
Finally, we also explore corrupting the masks using bounding box around the saliency mask which results in 74.5% top-1 and 92.2% top-5 accuracy under linear evaluation on the ImageNet validation for ResNet50 (Ix) encoder trained for 300 epochs with mask apply probability that this performance is comparable to using random rectangles to mask the large augmented views during training (see table 19) and is lower than directly using the saliency masks from the trained saliency detection network.
0.1 Note
G.3 BENEFITS OF SALIENCY MASKING FOR ROBUSTNESS AND OOD GENERALIZATTIN
To further highlight the benefits of using saliency masking, we conducted an additional ablation where we show the impact of only using the multiple views on downstream performance, particularly on robustness and OOD generalization downstream tasks. In Table 21 we compare the performance of ReLICv2 without saliency masking (just using multi-crop) vs ReLICv2 with saliency masking. We report Top-1 Accuracy under linear evaluation on ImageNetV2 and ImageNet-C (robustness), and ImageNet-R (IN-R), ImageNet- Sketch (IN-S) and ObjectNet out-of-distribution)) ImageNetv2 has three variants matched frequency (MF), Threshold 0.7 (T-0.7) and Top Images (TI). The results for ImageNet-CC (IN-C) are averaged across the 15 different corruptions. These results highlight that the of saliency masking for ReI.CV2 results more robust representations.
32

---

Under review as a conference paper at ICLR 2023
Table 21: Top-1 Aceuracy (in %) under linear evaluation on ImageNetV2 and ImageNet-C (ro- bustness), and ImageNet-R (IN-R), ImageNet-Sketch (IN-S) and ObjectNet (out-of-distribution) ImageNetv2 has three variants matched frequency (MF), Threshold 0.7 (T-0.7) and Top Images (TI). The results for ImageNet-C (IN-C) are averaged across the 15 different corruptions.
Robustness
OOD Generalization
Method
MF
T-0.7
Ti
IN-C
IN-R
IN-S
ObjectNet
RELICv2 w/o saliency masking
64.2
73.1
78.0
43.8
22.9
5.7
24.5
RELICv2 (ours)
65.3
74.5
79.4
44.8
23.9
9.9
25.9
G.4
OTHER MODEL IUPERPARAMETER
Now we turn our attention ablating the effect of other model hyperparameters on the downstream performance of RELICv2 representations. Note that these hyperparameters have been introduced and extensively ablated prior work [Grill et al. 2020 Mitrovic et al. 2021 2020
Number of negatives. As mentioned in Section RELICv2 selects negatives by randomly sub- sampling the minibatch in order to avoid false negatives. We investigate the effect of changing number of negatives in table 22 We see that the best performance can be achieved with rela- tively low numbers of negatives, i.e. just 10 negatives. Furthermore, we see that using the whole batch negatives has one of the lowest performances.
In further experiments, we observed that for longer pretraining (e.g. 1000 epochs) there is less variation in performance than for pretraining for 300 epoch which itself also quite low.
Number negatives
Top-1
Top-5
1
75.1
92.4
5
75.2
92.6
10
75.4
92.5
20
75.3
92.7
50
75.5
92.5
100
75.4
92.5
500
75.1
92.4
1000
75.3
92.6
2000
75.4
92.5
4096
75.2
92.6
Table 22: Top-1 and top-5 accuracy (in %) under linear evaluation on the ImageNet validation set for a ResNet50 (Ix) encoder set for different numbers of randomly selected negatives. All settings are trained for 300 epochs.
Target EMA. RELICV2 uses target network whose weights are an exponential moving average (EMA) of the online encoder network which is trained normally using stochastic gradient descent; this is setup first introduced [Grill al. 2020) and subsequently used in [Mitrovic et al.) 202 among others. The target network weights iteration are & ~ (1 7) where EMA parameter which controls the stability of the target network (? = sets & = 0.); 0, are the parameters of the online encoder at time f,, while
the
the parameters of the target encoder time As can be seen from table 23 all decay rates between 0.9 and 0.996 yield similar performance for top-1 accuracy on the ImageNet validation set after pretraining for 300 epochs indicating that RELICv2 is robust to choice of y in that range. For values of of 0.999 and higher, the performance quickly degrades indicating that the updating of the target network
slow. Note that contrary
to Grill et al.) 2020) where top-1 accuracy drops below 20% for = 1, RELICv2 is significantly more robust to this setting achieving double that accuracy.
33

---

Under review as a conference paper at ICLR 2023
Top-1
Top-5
0
73.5
91.5
0.9
74.6
92.2
0.99
75.5
92.6
0.993
75.4
92.5
0.996
74.4
92.0
0.999
70.5
89.8
1.0
39.6
63.6
Table 23: Top-1 1 and top-5 accuracy (in %) under linear evaluation on the ImageNet validation set for ResNet50 (Ix) encoder set for different setting of the target exponentially moving average (EMA). All settings are trained for 300 epochs.
34


Reviews:
[{"review_id": "lM5ayJa93oi", "reviewer": "Reviewer_hHxH", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper is an extension of RELIC self supervised learning method.\n\nIn this extension, two main augmentation method is added. (1) Saliency map (2) Views of varying sizes.\nThe loss function has also been modified.\n\nExtensive experiments had been perform to study the effects of these extensions.", "summary_of_the_review": "Well written paper, however paper can be strengthen with error bars and with experiments showing how results differs with different hyper-parameters.", "strength_and_weaknesses": "(1) Strength: a lot of experiments are being carried out with a lot of analysis\n(2) Strength: Results seems to be better than previous methods.\n(3) Weakness: However there is no error bars to convince readers about the robustness of the reported results. for example, multiple duplicated experiments should be perform and p-value calculated when comparing the results of various methods.\n(4) Weakness: it is not clear if the reported results are due to more careful tuning of hyper-parameters. A good way to silence this criticism is to show evidence of robustness of results with respect to hyper-parameter values.\n(5) Weakness: Seems that the major contributions are two more augmentation methods. ", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "_WyY21_O9QJ", "reviewer": "Reviewer_Z39N", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper proposes a novel self-supervised learning framework called ReLIC v2. This framework builds upon ReLIC's loss and combines multicrops and background mask. The method achieves the state-of-the-art and better linear probe accuracy than supervised ResNet50.\nExtensive transfer experiments and ablation study verifies the effectiveness of the proposed method.   ", "summary_of_the_review": "State-of-the-art self-supervised learning framework using ReLIC loss plus multicrops and background mask. Overall limited novelty but a lot of novel engineering design that's useful for the community.", "strength_and_weaknesses": "Strengths:\n1. This method achieves state-of-the-art ImageNet linear probe accuracy. It combines a few existing effective building components including ReLIC loss function, multicrops and background mask. Even though these components are already existing in the literature, I find that some special settings including 4x big + 2x small multicrops are novel.\n2. The authors conducted a scaling study on a larger dataset such as JFT300M. Even though they derived relatively poor scaling performance, I find such study helpful and insightful, i.e. how augmentation-based joint-embedding method can benefit from scaling. \n\nWeaknesses:\n1. Overall the paper has very limited novelty.\n2. From ablation study, the background mask only shows 0.3% improvements. The main advantage seems to come from the multicrops. Using 4x big views + 2x small views means effectively much higher computation cost. This method still uses 4096 batch size. As far as I can tell, the previous works limiting to 2x big views + many small views with this batch size are mainly caused by GPU memory. It is not discussed how their method can afford this. \n3. The background mask here is very hand-crafted. Figure 8 seems too good to be true for arbitrary input. Is there any failure case for the background removing?", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "XQhHsciQxm", "reviewer": "Reviewer_aWZZ", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "================================================================= \n\n[Summary] \n\nThis paper proposes an improved version of ReLIC, dubbed ReLICv2, by including views of varying sizes and saliency masking into the ReLIC's training loss, and presents benchmarking results using ResNet encoders in several experiments.\n\n================================================================= \n\n", "summary_of_the_review": "Using views of varying sizes and saliency masking to improve the original ReLIC's training loss is intriguing, but, it does not appear to make a significant impact, as do the many flavors of contrastive self-supervised approaches. ", "strength_and_weaknesses": "================================================================= \n\n[Main Strengths] \n\nThe key merit of this paper is that the authors' approaches for improving the ReLIC indeed produce better benchmark results than previous ResNet-based methods at a different model and dataset sizes.\n\n================================================================= \n\n[Main Weaknesses] \n\nThe fundamental shortcoming of this study is that, while the basic idea is interesting, it does not appear to have the intended impact, as the title implies, much like the various flavors of contrastive methods. Given the richness of learned visual representation, focusing primarily on linear evaluations in the classification tasks is a too narrowed contribution.  I encourage authors to explore further into specific image domains and/or paired with Vision transformers to see whether ReLICv2 is still effective. Furthermore, I don’t see the implementation GitHub repo is intended to be provided.\n\n================================================================= \n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "CfLuj8yhnRk", "reviewer": "Reviewer_jUzd", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper proposes a few tricks that can be added on top of ReLIC for self-supervised learning. First, saliency masking is used to explicitly reduce spurious correlations in the model. Second, augmentations of small sizes are used. This has the effect of occluding some regions of the image. Using these two tricks, the authors were able to improve the performance of ReLIC by a good margin. Moreover, the authors also achieve SOTA results on Resnet based self-supervised learning methods. Several ablation studies are also performed.\n\n", "summary_of_the_review": "Overall, I think this is a very good paper in terms of ideas proposed and rigorous experimental study. I advocate for acceptance.\n\n\n======================================\n\nPost rebuttal:\n\nAfter looking at other reviews and having discussion with some reviewers, I feel like the paper has the following issues:\nSimilarity with prior work: As pointed out by reviewer Z39N, background augmentation only has marginal improvements. So, this makes me a bit worried about the main message the paper gives. Its probably that improvements mainly comes because of multi-crop at scale. If this is the case, the authors need to make this very clear (if the paper gets accepted). I also agree with other reviewers that claiming that they beat supervised learning is bit of a overclaim and I would appreciate if the authjors address this.\n\nBut, I also feel that even if the paper proposes a few tricks, I am fine with accepting the paper as the computer vision field has made progress through a series of tricks. But the authors should make that message very clear.\n\nSince I was not aware of these concerns at the time of giving my review, I downvote my rating to 6. \n\n", "strength_and_weaknesses": "Strengths:\n\nThis is a well written paper. The authors clearly state what their main contribution is. The two tricks that are proposed is intuitive and looks promising. The paper is also explained well and I was able to understand the main contributions.\n\nThe experiments are very extensively performed. Comparisons with existing Resnet based models are included. Experiments are performed with different model sizes. Results are also shown on OOD, transfer, semi-supervised leanrning and for other tasks such as segmentation. Experimental section is quite strong and I don't have any concerns.\n\nI also apprecitate the authors for providing detailed experimental settings in appendix.\n\nWeaknesses:\n\nThe main tricks proposed in this work seem to be general purpose. Do you think this would work on ViTs?\nIn this regard, the self-attentions in ViTs might alrealy be doing some sort of saliency detection. For instance, they have been shown to do semantic segmentation. So, I am curious if thesr tricks can be used with ViTs?\n\nWhy did you stick only with Resnet based architectures? Do you think ReLICv2 would work with architectures like Swin? If so, would you set SOTA results there?\n\n\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works."}, "extra_text": {}}]

Author rebuttal:

Decision: Reject