The Close Relationship Between Contrastive Learning and Meta-Learning
Authors: Renkun Ni
Manli Shu
Hossein Souri
Micah Goldblum
Tom Goldstein
Venue: ICLR 2022 Poster (2022)
Keywords: meta-learning
contrastive learning
self-supervised learning

Abstract: Contrastive learning has recently taken off as a paradigm for learning from unlabeled data. In this paper, we discuss the close relationship between contrastive learning and meta-learning under a certain task distribution. We complement this observation by showing that established meta-learning methods, such as Prototypical Networks, achieve comparable performance to SimCLR when paired with this task distribution. This relationship can be leveraged by taking established techniques from meta-learning, such as task-based data augmentation, and showing that they benefit contrastive learning as well. These tricks also benefit state-of-the-art self-supervised learners without using negative pairs such as BYOL, which achieves 94.6\% accuracy on CIFAR-10 using a self-supervised ResNet-18 feature extractor trained with our meta-learning tricks.  We conclude that existing advances designed for contrastive learning or meta-learning can be exploited to benefit the other, and it is better for contrastive learning researchers to take lessons from the meta-learning literature (and vice-versa) than to reinvent the wheel. 

Published as a conference paper at ICLR 2022
THE CLOSE RELATIONSHIP BETWEEN CONTRASTIVE LEARNING AND -ETA-LEAE LEARNING
Renkun Ni University of Maryland rn92mè cs. umd edu
Manli Shu University of Maryland man 1 lisiccs umd. edu
Hossein Souri
Johns Hopkins University hsouri jhuu edu
Micah Goldblum University of Maryland goldbl umd. edu
Tom Goldstein
University of Maryland tomgêcs umd. edu
ABSTRACT
Contrastive learning has recently taken off as paradigm for learning from un- labeled data. In this paper, we discuss the close relationship between con- trastive learning and meta-learning under a certain task distribution. We com- plement this observation by showing that established meta-learning methods, such as Prototypical Networks, achieve comparable performance to SimCLR when paired with this task distribution. This relationship can be leveraged by taking established techniques from meta-learning, such as task-based data aug- mentation, and showing that they benefit contrastive learning well. These tricks also benefit state-of-the-art self-supervised learners without using nega- tive pairs such as BYOL, which achieves 94.6% accuracy on CIFAR-I0 using a self-supervised ResNet-18 feature extractor trained with our meta-learning tricks. We conclude that existing advances designed for contrastive learning or meta- learning can be exploited to benefit the other, and it is better for contrastive learning researchers versa) than to reinvent the wheel. Our Pytorch implementation can be found on: https: / /github. com http///eeekuni//eeaaatttt..ttee
take lessons from the meta-learning literature (and vice-
INTRODUCTION
Self-supervised visual representation learning (SSL) has recently gathered attention due to its ability
to learn image features without manual supervision, thus allowing for efficient learning on down-
stream tasks such as detection and segmentation (Norozzi & Favaro 2016 Zhang et al., 2016 Oord
ctt al.
| 2018 Hjelm et al.] 2018 |Wu et al.] [2018] Gidaris et al. [2018] He et al. [2020] & Maaten
2020 Tian al.) 2020a Chen et al. 2020a] Kim et al. 2020 Grill et al. 2020 Caron et al.
2020
Kalantidis et al. 2020, Shen et al., 2020 Li et al. 2020 Zbontar et al. 2021 Chen & He
2021
Caron et al. 2021 Among SSL approaches, contrastive learning based methods (Chen et al.
2020a
He et al.] 2020) Chen et al. 2020b) show particularly strong potential and achieve promising results
which are close those fully supervised methods on numerous computer vision benchmarks.
These methods rely on applying various data augmentations such as random crops, flips, color distor- tion, and Gaussian blur the same training sample to create different views of an image. Two such example methods, SimCLR (Chen et al. 2020a) and MoCo (He et al. (2020), involve reducing the distance between features corresponding to positive pairs (different augmented views of the same image), and increasing the distance between features corresponding to negative pairs (augmented views of different images).
Meanwhile, meta-learning is an established popular framework for learning models that quickly adapt to on-the-fly tasks given a small number of examples (Hochreiter et al.) 2000 Finn et al.) 2017 Nichol et al. 2018: Bertinetto et al.) 2018 Lee et al.) (211). The training loop for meta- learners typically involves (i) sampling a random batch of classes and (ii) updating feature ex-
"qqual contribution

---

Published as a conference paper at ICLR 2022
tractor to distinguish between these classes. This procedure mirrors that of contrastive learning, which proceeds by (i) sampling batch of images and augmenting them to generate classes (each "class" is an image plus all of its views), and (ii) updating a feature extractor to distinguish between these classes. Conceptually, this contrastive learning procedure resembles meta-learning where the training tasks generated by computing multiple views of individual images.
In this paper, we discuss the close relationship between contrastive learning and meta-learning. Concretely, show that established meta-learning algorithms, originally designed for few-shot learning, can achieve the same performance as recent contrastive learning algorithms on standard SSL problems when paired with the same data sampling strategy. In addition, we explore tech- niques, originally designed for meta-learning, that can improve contrastive learning. Specifically, we explore ways by which we can adapt data augmentation strategies inspired by recent work in meta-learning (Su et al. 2020; Ni et al. 2021) to SSL. and find that this approach can yield signifi- cant performance boosts.
Our contributions can be summarized as follows:
We formulate a meta- leeta-le based framework for understanding self-supervised learning. and we show that meta-learners can achieve comparable self-supervised performance to contrastive learning methods.
We propose a meta-specific task augmentation strategy which boosts the performance of self-supervised learning. This data augmentation method generalizes methods with no negative pairs, such as BYOL (Grill et al. 2020), as well.
RELATED WORK AND BACKGROUND
2.1 META- FOR FEW-SHOT LEARNING
Meta-learning algorithms for few-shot learning aim Iearn model that can quickly adapt to new tasks with limited data and generalize to unseen examples. To achieve this, the adaptation and eval- uation procedures are both simulated during meta-training. During each episode of meta-learning, we sample a task, T; from a distribution of tasks, often corresponding to combinations of training classes formed into classification problems. Each task consists of support data T,' and query data T.", so that To (T }. When applied to few-shot classification, this task is called k-shot, N N-way classification problem, where denotes the number of training samples per category in the support data. Then, support data will be used to simulate few-shot training data, while query data will be used to simulate novel testing samples.
A weta-leerning model F, in this setting, contains feature extractor and classification strategy, A. This classification strategy can take various forms, such as adding a linear classifier on top of the feature extractor and fine-tuning either the linear layer the whole network end-to-end, this strategy may simply classify samples by selecting the nearest class prototype. Meta-leamning training algorithms have inner loop and an outer loop in each parameter update. In the inner loop, the model is first fine-tuned on support data T,S. Then, in the outer loop, the updated model is used to predict on query data 7,", and a loss is minimized with respect the model's parameters before fine-tuning Intuitively, we update parameters so that the feature extractor extracts better features for the classification strategy, often resulting in tightly clustered features corresponding to each class (Goldblum al., 2020p. Existing works apply various methods for fine-tuning on support data during the inner loop. In line of algorithms, such as MAML and Reptile (Finn et al. Nichol jet al., 2018), all the parameters in the model are updated using gradient descent during fine-tuning on support data. Other algorithms, such as MetaOptNet and R2-D2 (Lee et al., 2019 Bertinetto cct al., [2018), keep the feature extractor frozen during fine-tuning: MetaOptNet uses SVM, and R2- D2 uses ridge regression
[
top of the feature extractor. Similarly, metric learning approaches, such as ProtoNet (Snell et al., 2017 Kye et al. 2020), freeze the feature extractor as well, and create class centroids from the support data in the inner loop. In this paper, we primarily focus on the latter algorithms due to their efficiency and performance as well as the similarity of contrastive learning to metric learning.
Note that algorithms in the vein of Reptile Nichol al.
2018 split the tasks into support and query.
2

---

Published as a conference paper at ICLR 2022
2.2 SCPEVISDE LEARNING
Contrastive SSL. Contrastive methods Oord et al. 2018; Wu et al., 2018 Tian et al. 2020a Chen ctt al. 2020aa He et al., 2020; Chen al. 2020b) achieve promising performance in supf-supe learning. As mentioned previously, contrastive learning maximizes agreement on different aug- mented views of the same image (called positive pairs) while ensuring disagreement on samples generated by different base images (called negative pairs). Given a batch of input images m ran- dom data augmentations are applied on the same batch, generating a set of training samples {x' Im 1== These samples are fed into a backbone network f(-) to obtain the feature representations (h" J'mm Then, a small neural network g(-), usually a non-linear MLP, is applied to project (h') to the latent representations {z') in the space where contrastive loss I(-) is applied, ensuring that latent representations of positive pairs are similar while latent representations of negative pairs are different. In this paper, we mainly focus on this type of self-supervised learning and show its close relation with meta-learning. The batch sampling procedure of contrastive learning can be viewed as sampling new classification problem with a number of classes equal to the number of base im- ages used to generate augmented views. We will see that this on-the-fly sampling of classification problems closely mirrors a common meta-learning setup.
Non- Non-Contrastive SSL. Some contrastive methods are generative approaches, such as auto- encoders (Vincent et al., 2008; 2010 Kingma & Welling, 2013), and adversarial learning (Goodfel-) low et al., 2014), where a distribution is learned over data and a latent embedding. These methods are typically computationally expensive as they require training a learned model which maps latent representations to pixel space. Other non-contrastive methods rely using heuristic designed pre- text tasks (Doersch et al. 2015 Zhang et al. 2016 Noroozi & Favaro 2016 Gidaris et al. 2018) to learn the representation. More recently, B BOL (Grill al.) 2020) showed that by bootstrapping target representation prediction, feature representations can be learned without negative pairs. How- ever, BOL still adopts the data augmentation procedure from contrastive learning, where different augmented views are used as training samples. In section our proposed task augmentation strategy.
5
we show that B BOL still benefits from
Data Augmentation in Meta-Learning and SSL., Data augmentations play an essential role in both meta-learning and self-supervised learning. In meta-learning, [Su et al. (2020): Liu et al.) (2020); Ni et al. 2222)) show that proper data augmentation and meta-specific task augmentations dramatically improve few-shot learning performance by expanding the number of classes available for sampling- In self-supervised learning. Tian et al. (2020b) show contrastive learners can find better feature rep- resentations when views contain less mutual information. In addition, Kimm et al. (220)); (hhnn et al.)
(2020): Li et al. (2020) show that adding harder examples such cut-mixed samples into the training pipeline can improve self-supervised performance. In this paper, we show that similarly to meta- learning, self-supervised learners can benefit from carefully applied data augmentation techniques which mirror task augmentations from the meta-learning literature.
2.3
LEARNING FOR META-LEARNING AND FEW-SHOT LEARNING
Another line of research (Hsu et al. 2018 Khodadadeh et al. 2018 Ye et al. 2020 Medina et al.
2020) focuses on self-supervieed learning for meta-learning and few-shot learning. Hsu et al. (0088)
focuses on partitioning samples from dataset to construct meta- meta-learning tasks and using MAML or
ProtoNet on 4-layer architectures to solve few-shot problems. Similarly, Khodadadeh et al. 2018
and Medina et al. (2020) add more data augmentations, such as Auto Aug (Cubuk et al. 2018
and use MAML and ProtoNet, respectively, to learn a few-shot representation. To further improve few-shot performance, Ye et al. (2020) sample harder mixed support examples and apply task- specific projection head to help generalize to unseen classes. These methods focus on few-shot learning performance, which entails up to 50 training examples per class. In contrast, focus on the unsupervised learning paradigm where large models are pre-trained on samples generated via data augmentation and are applied to downstream tasks such ImageNet classification.
EXPERIMENTAL SETUP
Datasets and Evaluation We conduct sule-supe training on both the CIFAR- 10 and ImageNet datasets (Krizhevsky et al. 2009 Deng al. 2009). Following Chen et al. (020a). we evaluate
3

---

Published as a conference paper at ICLR 2022
Contrastive loss
Meta loss
Projections
Z
A
g
Representations
A
6
Views
Support
Query
Images
a
(a) Training procedure of contrastive learning
(b) Meta-Learning framework for SSL
Figure I: (a) Training procedure of contrastive learning. Two augmented views are generated by applying random transformations the same input batch. A backbone f(-) and projector g(-) is learned through contrastive prediction tasks. (b) Meta-Learning framework for SSL. We adopt the data augmentation operations contrastive learning. We generate b-way classification problem, b is the batch size, by treating each image itself as a class. Two views are separated as support data, on which the network is fine-tuned and the classification strategy learned by A:; query data, on which we apply the updated model and calculate the meta-loss. At the end of training, everything but f() )) is discarded, and h is used as the image representation.
pre-trained representations in a linear evaluation setting, where feature extractors are frozen, and a classification head is stacked on top and tuned. In addition, test the performance of the pre-trained feature extractors on downstream tasks such as transfer learning and semi-supervised learning with 1% and 10% of labeled data. Evaluation and dataset details can be found in Appendix A.2
Pre-Training Details We use a ResNet-18 18 backbone for all experiments on CIFAR- 10 and ResNet- 50 for those ImageNet. We train the model on CIFAR-10 10 with the LARS optimizer (You al., 2019) and batch size 1024 for 1000 epochs (with 4 GPUs). On ImageNet, we use the same optimizer and batch size of 256, and train for 100 epochs (with 8 GPUs). For ImageNet pre-training, follow the hy hyperaarameter setting in Chen et al. (2020a), including baseline data augmentation meth- ods, dimension of the latent space, and learning rate decay schedule. For CIFAR- pre-training, we use the same CIFAR-10 specific hyperparameters as SimCLR again. For BYOL, we use the same learning rate schedule as meta-learners and start with learning rate In addition, both the projector and predictor in BYOL are two-layer MLPs with hidden dimension 2048 and output dimension 256. More details can be found in Appendix A.
4 A META-LEARNING FRAMEWORK FOR SSL
Meta-learners, designed for few-shot learning, and contrastive learners for SSL are built on similar intuitions. Both approaches learn to solve new tasks on-the-fly with each batch - new classification problems the case of meta-learning and differentiating new batch of images in the trastive learning. Furthermore, both approaches hold a goal of learning invariances which generalize novel problems at inference; meta-learnrss should extract similar features for each instance of novel test class, and contrastive learners should extract similar features for each view of an image sample. In this section, we show how one can construct meta-learning framework for SSL which closely mirrors the strategy adopted by recent contrastive learning methods.
of con-
We now describe how to generate meta-learning task distributions p(T) for self-supervised learning. We adopt the data augmentation operations from contrastive learning, where different random aug- mentations are applied to the input batch to generate alternative views. In general, given m random augmented views of batch of input images x. treating all images generated by the same base image as a class. Then, we can divide the data from each class into mi support and ma query samples so that my + ma
create b-way classification problem by
This framework for
4

---

Published as a conference paper at ICLR 2022
Algorithm 1: MetaMe Meeannne Framework for Self- Learning
Require: Base model Fo. classification strategy A, learning rate ", and distribution of data augmentations DD.
Initialize 0, the weights of F;;
while not done do
for j= L..... 71 do
Sample batch of base images x;
Sample m random data augmentations from D obtain augmented views {
Timi
Separate {xt) into support set 7 (x1) and query set 7,¹ (x* kxk) +1m
Fine-tune model on Ty: 0j A(0, T;)):
Compute gradient 9 V [F,, );;;;
end
Update base model parameters: 0 t 0 2 Σ,
end
sampling a batch containing augmented views of base images and dividing them into support and query samples yiclds task distribution p(T). In few-shot learning terminology, each training task Ti is a mi my-shhtt-bwway classification problem, since we have base images which generate classes, and for each such image, have my support samples. Viewed in this way, SimCLR sets and m, = m2 = 1, but SimCLR has important differences from common meta-learners.
Unlike typical meta-learning methods, SimCLR compares every sample with every other sample, while methods like ProtoNet only compare each query sample with each support prototype and not with each other. Moreover, SimCLR samples single large batch of samples for each episode of training which corresponds to sampling single task, while meta-lcarners typically sample a batch of many tasks, i.e., classification problems, during each episode.
Now that we have established framework for sampling tasks, we can directly apply various meta- learning algorithms, such R2-D2 and ProtoNet described in Section 2.1 in order learn the parameters 0 of the base model F. Recall that the base model contains classification strategy and a feature extractor, which is the learning target SSL. We use the same feature extractor here used for contrastive learning in Section 2.2 which consists of backhone f(-) followed by projection head g()). Formally, we solve the meta-lcarning optimization problem.
min E [Csst], LSSL I(For, T
where 0' = A(0, T parameters updated by training support tasks, and / is the loss function, e.g., cross-entropy loss in our work, used in the outer loop of training. After pre-training, only the backbone f(.) will be kept for sule-supervised evaluation. This meta-learning framework for self-supervised learning is summarized in Algorithm and Figure [
We compare the performance of representations learned by meta-learners with SimCLR under the default setting described in Section 3 Vable|annd Table[]show the linear evaluation top-1 accuracy for the feature representations trained and tested on the CIFAR-10 and ImageNet datasets, respec- tively. We observe that representations learned via meta-lcarning (R2-D2 and ProtoNet) can achieve performance on par with SimCLR on CIFAR- 11 but worse on ImageNet. Note that during training, we use the same exact hyperparameter as SimCLR due to computational constraints, which may
Table 1: Linear evaluation CIFAR-10 for representations learned via contrastive learn- ing and our meta-learning framework.
Table 2: Linear evaluation ImageNet for representations learned via contrastive learn- ing and our meta-learning framework.
Method
Backbone
Top-1 Acc(%)
Method
Backbone
Top-1 Acc(%)
SimCLR
ResNet-18
91.4
SimCLR
ResNet-50
58.8
ProtoNet
ResNet-18
91.8
ProtoNet
ResNet-50
57.6
R2-D2
ResNet-18
91.6
R2-D2
ResNet-50
55.5
5

---

Published as a conference paper at ICLR 2022
Table 3: ImageNet Top-1 accuracy (%) models fine-tuned with few labels.
Label fraction
Method
Backbone
1%
10%
Supervised baseline
ResNet-50
25.4
56.4
SimCLR
ResNet-50
32.4
53.6
ProtoNet
ResNet-50
31.0
52.9
R2-D2
ResNet-50
37.9
58.8
not be optimal specifically for our proposed method. We will see in the following experiments that although R2-D2 achieves worse lincar evaluation on ImageNet with this hyperparameter setting, actually performs better than SimCLR on downstream tasks, such as semi-supervin learning and transfer learning, other popular (and plausibly more realistic) evaluation scenarios for SSL methods.
Following Chen et al. (020a). we first evaluate the pre- pre-trained model by semi-supervised learning, where we fine-tune the pre-trained model with only fraction of labeled ImageNet data (1% and 10%). As we see from Table 3 with the same fine-tune setting (See Appendix (.2)). models pre- trained by R2-1D2 can achieve 5% higher top-1 accuracy than those pre-trained by SimCLR after fine-tuning on labeled data. Notably, the supervised baseline from Zhai et al. (2019) is strong due to exhaustive hyperparameter searching and stronger data augmentations used during the training.
To further compare the feature representations learned by different methods, we apply the pre-trained weights transfer learning. We consider 8 datasets with natural images of various categories (Nils- back & Zisserman 20088) Cimpoi et al., 2014 Everingham et al., Maji et al. [0133] Bossard et al.
2014| al.,) 2010 Krizhevsky al., 2009). For each dataset, we use the backbone (ResNet-50)
pre-trained on ImageNet as an initialization for the feature extractor of the downstream classifica- tion model. In contrast to linear evaluation, we fine-tune the entire model on the given dataset for 20,000 iterations with the best hyperparameter setting selected its validation split. Details of our hyperparameter selection are included in Appendix AA.2] All models are pre-trained on ImageNet for 100 epochs. We also include baseline model provided in Chen et al. (2020a) which does not pre- pre-trained weight as an initialization. Note that the baseline model is tuned to achieve comparable performance with a larger search space for hyperparameters, and it is trained for longer duration. From the results in Table we find that R2-D2 initialized model consistently outperforms its con- trastive counterpart on all datasets. These results suggest that for the model trained with R2-D2 works better as an initialization for downstream tasks than one trained with SimCLR. We speculate that this property is connected R2-D2's few-shot learning driven design and simulation of adapting to new tasks during its inner loop.
number of epochs,
Table 4: Transfer learning using ImageNet pre- treitrained weights. We report mean per-class accuracy (%) on the Flowers and Aircraft datasets, mean average precision (mAP) on the VOC2007 classifi- cation dataset, and Top-1 accuracy the remaining datasets.
Flowers 102
DTD
VOC2007
Aircraft
Food101
SUN397
CIFAR-10
CIFAR-
Baseline
92.0
64.8
67.3
85.9
86.9
$3.6
95.9
80.2
SimCLR
92.4
72.7
66.0
83.7
86.3
94.8
79.1
ProtoNet
92.7
71.5
64.7
83.9
86.2
56.4
96.0
79.1
R2-D2
94.5
73.8
69.9
86.2
86.9
59.7
96.7
82.8
5 BOOSTING SSL WITH META-SPECIFIC AUGMENTATION
Now that we have established a relationship between contrastive learning and meta-learning, will apply tools developed in the latter discipline to enhance contrastive learners. Previous work has shown that data augmentations such crops and colorizing play an important role in both contrastive learning and meta-learnin. We focus on a particular augmentation strategy from the meta-learning literature, termed task augmentation, which aims to expand the number of classes

---

Published as a conference paper at ICLR 2022
available for sampling rather than expanding the number of samples per class. Liu et al. (2020); Ni
let al. (202T) show empirically that data augmentations work best when applied to carefully chosen parts of the meta-learning batch, and large rotations can only work as task augmentation, in which rotation by a chosen degree is applied to all images in an entire class, and we then treat them as a new class. Large rotations, and other dramatic transformations used for task augmentation, actually decreases performance when instead applied independently on support and query samples (as a way to increase data within class) rather than uniformly on an entire class (therefore defining new class). Augmentations that exhibit this task augmentation behavior are typically those which transform an image so much that its semantic content looks different to a human. By keeping images with very large augmentations in the same class, we may accidentally encourage models to learn overly strong invariances which do not naturally exist in the data.
Table 5: Linear evaluation (Top-1 accuracy (%)) on CIFAR-10 with feature representations learned by SimCLR and BYOL in default setting (see Section B. Simply adding large rotations to data augmentation hurts the performance of self--upprvvsed learning.
Rotation
SimCLR
BYOL
No Yes
91.4
92.1
89.7
90.6
Although data augmentations such as large rotations have been shown effective for visual pre- training (Feng et al. 2019 Gidaris et al. 2018), how to encode that into the data augmentation framework of contrastive learning remains unclear. In addition, we observe that when applied along with contrastive learning, the phenomenon mentioned above occurs as well. Namely, the same large rotation data augmentation, which can improve the performance of meta-learners via task augmen- tation, also degrades the performance of contrastive learners when applied to samples independently (instead of uniformly to an entire class). Table 5 shows linear evaluation accuracy on CIFAR-10 when add this augmentation to the training pipeline of SimCLR. In this experiment, we ran- domly rotate every augmented view by {90', 180% -.70) with probability 0.25 each. In Table 5 that the accuracy of SimCLR drops by 2%, and this degradation also occurs in self- supervised learning methods without negative pairs such as BYOL, which adopts the same augmen- tation pipeline as SimCLR. Driven by the observation and insights from the meta-learning literature, we are motivated to apply strong augmentations, such as large rotations, to contrastive learning at the task level so that models can benefit from the additional augmentation without learning overly strong and harmful invariances.
In the literature on meta-leaming literature for few-shot classification, large rotations can be used either as a task augmentation or an auxiliary loss (Su et al., 2020; Liu et al.] (2000; Gidaris et al.) 2019) We adopt both methods into our meta-learning inspired pipeline for self-supervised learning and describe the details below. This procedure is illustrated in Figure and Figure 3
Large Rotation as Task Augmentation Instead of randomly rotating each training sample inde- pendently, we rotate all images from the same class (different augmented views of the same original base image) by the same degree (chosen randomly from {0", 90" 180%, 270") }) in both the support and the query data. In such a way, the number of potential classes is enlarged by 3 times. We com- bine large rotation augmentation with the basic augmentations used in contrastive learning during the sampling stage and keep other components of the training procedure unchanged.
Large Rotation as Auxiliary Loss. In addition to task augmentation, large rotations can be used as auxiliary prediction task for the original self-supervised problem, where the angle of a rotated image is used as the target label. To this end, we spin the input batch x by an angle to generate the rotated training samples, {x,, ). where d E (09, .90%, 180", 2220%) J,y {d/90). Then, we stack a 4-way classification head r on top of the shared backbone f and projection head g, to predict the angle of the rotations evaluated with cross-entrope loss LLR:
LLR =
Σ (=)()(
(x
During training, we sum this loss with the original self sufervueer loss to achieve the final objective, f = LSsL + XLLR, where À is coefficient that controls the influence of our prediction task. In
7

---

Published as a conference paper at ICLR 2022
Ingut Irages
Original Class
Novel Class
Novel Class 2
Novel Class 3
Augicial Vivs
Telk Naaicittoooo
(b)
(a)
Figure 2: (a) Examples of novel classes created by rotation by (90", 180%, 270"). (b) Adding task augmentation into the data augmentation pipeline. We first apply random transformations on the input batch, then for each augmented view from the same base image, we rotate them by the same degree.
Images
Views
SSL Algorithm
LSSl
Rotation
Share
L
i 1
x
Feature
r
LLR
Extractor
Figure 3: Training procedure with Large Rotation augmentation as an auxiliary loss. We randomly rotate each augmented view by degree in {0", 90% 180", 270"). We share the feature extractor used the SSL algorithm and apply along with a 4-way linear head r(-) to predict the rotation angle. We sum up the SSL loss, LSSL. with the angle prediction loss, LLR, our ultimate objective, L.
contrast to few-shot classification problems, the batch size self-supervised tasks is often very large. If we rotate all images individually by four different degrees, we will triple the batch size which will consume valuable memory. Instead, propose only sample single rotation for each image. We conduct an ablation study on the number of rotations used to augment each batch on CIFAR-10 pre-raining, and we show that when the batch size is large, rotating each augmented view in the batch once is enough (see Appendix A.3).
Table 6: Linear evaluation (Top-1 accuracy (%) with one standard error) on CIFAR-10 10 with repre- sentations learned via different augmentations. "Baseline" denotes augmentations used in SimCLR, "+ TA" denotes adding large rotation as task augmentations, "+ LI.R" denotes adding large rotation as an auxiliary loss, and "+ Mix" denotes adding image mixing augmentation (Shen et 2020
Augmentations
SimCLR
R2-D2
ProtoNet
BYOL
Baseline
91.52 + 0.19
91.46
0.11
91.84 0.19
92.08 0.13
TA (ours)
91.46 + 0.13
91.24
0.06
91.98
0.13
92.03 + 0.15
LLR (ours)
93.04 + 0.17
0.11
93.16
93.18 0.16
Mix
92.98 - 0.13
93.02
13.36+00111
93.83 + 0.06
Mix LLR (ours)
93.88 + 0.13
93.70
0.20
94.12 0.13
94.57 0.15
Table 6 highlighss the effectiveness of treating large rotations task augmentation
auxil-
iary loss on pre- trairer CIFAR- representation in the default setting. Notably, our method enables BOL achieve 94.6% accuracy with backbone trained entirely unlabeled data, just high
8

---

Published as a conference paper at ICLR 2022
models of the same architecture trained in fully supervised setting (but without augmenta- tions) (Kuang et al. 2017). For task augmentation, we apply large rotations with the probability of 0.25 each on top of the baseline augmentations. For the rotation angle predictor loss, weight the additional loss term with coefficient À = 1 for all experiments except for pre-training with R2-D2, where we set À 0.01. In Table 6 we that by using large rotations can avoid the massive accuracy drop observed in Table In addition, when we add the angle predic- tion loss for large rotation an auxiliary loss, we boost the linear evaluation of all the pre-trained representations by at least 1%.
task augmentation,
Table 7: Linear evaluation on ImageNet with representations learned by SimCLR and with proposed augmentations.
Model
Top-1 (%)
SimCLR
58.8
SimCLR CLR (ours)
59.6
We further show that our proposed method can be combined with existing data augmentation meth- ods by adding harder examples with mix-up or cut-mix in Table 6 Following [Shen al.) (2020). we mix one augmented view from each base image with an augmented view from a second image where the indices used for the second augmented view are reverse ordered. In order mix images, we randomly apply mix-up or cut-mix with equal probability. Our proposed large rotation predic- tion loss consistently improves performance on top of the mixing augmentation by 1% on all pre-training methods, thus leading to more than 2% improvement from the baseline augmentations. On ImageNet, Table 7 displays the top-1 linear evaluation accuracy for methods with and without our proposed rotation angle prediction loss. With the proposed auxiliary loss, we achieve 0.8% im- provement on the representation trained for 100 epochs. Additional evaluations for transfer learning and semi- supervisd learning can be found in Appendix A.4
CONCLUSION
In this work, we discuss the close relationship between contrastive learning and meta-learning. In doing so, we propose a new meta-learning framework for self-supervised learning by converting the contrastive setup into on-the-fly image classification tasks. We show that feature extractors pre- trained via meta-learning achieve comparable results to contrastive learning methods and, in fact, they transfer better to downstream tasks in some cases. In addition, we leverage data augmentation ideas from meta-learning and incorporate them into contrastive learning. We demonstrate that the proposed meta-specific data augmentation consistently improves the performance popular self- supervised learning algorithms on various datasets.
ACKNOWLEDGEMENT
Support for this work was provided by the AFOSR MURI program, the Office of Naval Research, the ONR MURI program, the National Science Foundation (DMS-1912866), DARPA GARD (HR001 12020007), and the DARPA Young Faculty Award Program. Additional funding was pro- vided by Capital One Bank.
REPRODUCIBILITY STATEMENT
Our work is fully reproducible. We release our code in the aforementioned github repositories, and we discuss the training details in Section 3 and evaluation details in Appendix A.2
REFERENCES
Luca Bertinetto, Joao F Henriques, Philip HS Torr, and Andrea Vedaldi. Meta-learning with differ- entiable closed-form solvers. arXiv preprint
2018.

---

Published as a conference paper at ICLR 2022
Lukas Bossard, Matthieu Guillaumin, and Luc Van Gool. ponents with random forests. In European Conference on Computer Vision, 2014.
Food-101 mining discriminative com-
Mathilde Caron, Ishan Misra, Julien Mairal, Priya Goyal, Piotr Bojanowski, and Armand Joulin. Unsupervised learning of visual features by contrasting cluster assignments. arXiv preprint arXiv:2006. 09882, 2020.
Mathilde Caron, Hugo Touvron, Ishan Misra, Hervé Jégou, Julien Mairal, Piotr Bojanowski, and Armand Joulin. Emerging properties in self-supervised vision transformers. arXiv preprint arXiv:2104. 14294, 2021.
Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey Hinton. A simple framework for contrastive learning of visual representations. In International conference machine learning, pp. 1597-1607. PMLR, 2020a.
Xinlei Chen and Kaiming He. Exploring simple siamese representation learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 15750-15758, 2021.
Xinlei Chen, Haoqi Fan, Ross Girshick, and Kaiming He. Improved baselines with momentum contrastive learning. arXiv preprint arXiv:2003. 000::2003.0499,, 2020b.
M. Cimpoi, S. Maji, I. Kokkinos, S. Mohamed, and A. Vedaldi. Describing textures in the wild. In Proceedings of the IEEE Conf. on Computer Vision and Putlern Recognition (CVPR), 2014.
Ekin D Cubuk, Barret Zoph, Dandelion Mane, Vijay Vasudevan, and Quoc Le. Autoaugment: Learning augmentation policies from data. arXiv preprint arXiv:1805.095500 2018
Jia Deng, Wei Dong. Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale scale hi- erarchical image database. In 2009 IEEE conference on computer vision and pattern recognition, pp. leee, 2009.
Carl Doersch, Abhinav Gupta, and Alexei A Efros. Unsupervised visual representation learning by context prediction. Proceedings of the IEEE international conference on computer vision, pp- 1422-1430. 2015.
M. Everingham, L. Van Gool, C. K. I. Williams, J. Winn, and A. Zisserman. The PASCAL Visual Object Classes Challenge 2007 (VOC2007) Results. http://www.paccal
Zeyu Feng, Chang Xu, and Dacheng Tao. Self-supervised representation learning by rotation fea- ture decoupling. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 10364-10374, 2019.
Chelsea Finn, Pieter Abbeel, and Sergey Levine. Model-agnostic meta-learning for fast adaptation of deep networks. arXiv preprint arXiv:/703 703.03400, 2017.
Spyros Gidaris, Praveer Singh, and Nikos Komodakis. Unsupervised representation learning by predicting image rotations. arXiv preprint 1833v.1803.07728, 2018.
Spyros Gidaris, Andrei Bursuc, Nikos Komodakis, Patrick Pérez, and Matthieu Cord. Boosting few-shot visual learning with self-supervision. In Proceedings of the IEEE/CVE International Conference on Computer Vision, pp. 8059-8068, 2019.
Micah Goldblum, Steven Reich, Liam Fowl, Renkun Ni, Valeriia Cherepanova, and Tom Gold- stein. Unraveling meta-learning: Understanding feature representations for few-shot tasks. arXiv preprint arXiv:22022.0675: 2020.
Ian Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley. Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative adversarial nets. Advances in neural information processing systems, 27, 2014.
Grill, Florian Strub, Florent Altché, Corentin Tallec, Pierre H Richemond, Elena Buchatskaya, Carl Doersch, Bernardo Avila Pires, Zhaohan Daniel Guo, Mohammad Gheshlaghi Azar, et al. Bootstrap your own latent: A new approach to self-supervised learning. arXiv preprint arXiv:2006. 07733, 2020.
10

---

Published as a conference paper at ICLR 2022
Kaiming He, Haoqi Fan, Yuxin Wu, Saining Xie, and Ross Girshick. Momentum contrast for unsupervised visual representation learning. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 9729-9738, 2020.
R Devon Hjelm, Alex Fedorov, Samuel
Karan Grewal, Phil Bachman, Adam Trischler, and Yoshua Bengio. Learning deep representations by mutual information estimation and maximization. arXiv preprint arXiv: 1808.06670 2018.
Sepp Hochreiter, A Steven Younger, and Peter Conwell. Learning learn using gradient descent. In International Conference on Artificial Neural Networks, pp. 87-94. Springer, 2001.
Kyle Hsu, Sergey Levine, and Chelsea Finn. Unsupervised learning via meta-learning. arXiv preprint 1rX1v:1810.02334, 2018
Yannis Kalantidis, Mert Bulent Sariyildiz, Noe Pion, Philippe Weinzaepfel, and Diane Larlus. Hard negative mixing for contrastive learning. arXiv preprint arXiv:2010.01023 2020.
Siavash Khodadadeh, Ladislau Bölöni, and Mubarak Shah. Unsupervised meta-leaming for few- shot image classification. arXiv preprint arXiv:1811.11819 2018.
Sungnyun Kim, Gihun Lee, Sangmin Bae, and Se- Young Yun. Mixco: Mix-up contrastive learning for visual representation. arXiv preprint arXiv:2010.063000 2020.
Diederik P Kingma and Max Welling. urXiv:1312.66 14, 2013.
Auto-encoding variational bayes. arXiv preprint
Alex Krizhevsky, Geoffrey Hinton, et al. Learning multiple layers of features from tiny images. 2009.
Liu Kuang, Yang Wei, Yang Peiwen, and Ducau Felipe. Train cifar10 with py- torch, 2017. https///gtttbb.ooomuuuuuggittgrt
URL https://githuu: wwt.ggtwtw......uaageeeeooaaaaee com/ kuangliu/pytorch cifar
Seong Min Kye, Hae Beom Lee, Hoirin Kim, and Sung Ju Hwang. Transductive few-shot learning with meta-learned confidence. arXiv preprint 2xX.v:2002.12011 2020.
Kwonjoon Lee, Subhransu Maji, Avinash Ravichandran, and Stefano Soatto. Meta-learning with differentiable convex optimization. In Proceedings of the IEEE Conference on Computer Vision and Puttern Recognition, pp. 10657-10665, 2019.
Chunyuan Li, Xiujun Li, Lei Zhang, Baolin Peng, Mingyuan Zhou, and Jianfeng Gao. Self- supervised pre-training with hard examples improves visual representations. arXiv preprint arXiv:2012.13493, 2020.
Jialin Liu, Fei Chao, and Chih-Min Lin. Task augmentation by rotating for meta-learning. arXiv preprint arXiv:2003.008: 00804, 2020.
Maji, J. Kannala, E. Rahtu, M. Blaschko, and Vedaldi. Fine-grained visual classification of aircraft. Technical report, 2013.
Carlos Medina, Arnout Devos, and Matthias Grossglauser. Self-supervised prototypical transfer learning for few-shot classification. arXiv preprint arXiv:2006. / 1325, 2020.
Ishan Misra and Laurens van der Maaten. Self-supervised learning of pretext-invariant representa- tions. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp. 6707-6717, 2020.
Renkun Ni, Micah Goldblum, Amr Sharaf, Kezhi Kong, and Tom Goldstein. Data augmentation for meta-learning. In International Conference Machine Learning, pp. 8152-811111 8161. PMLR, 2021.
Alex Nichol, Joshua Achiam, and John Schulman. On first-order meta-learning algorithms. arXiv preprint XX33:/803300999 2018
Maria-Elena Nilsback and Andrew Zisserman. Automated flower classification over a large number of classes. In Indian Conference Computer Vision, Graphics and Image Processing, Dec 2008.
11

---

Published as a conference paper at ICLR 2022
Mehdi Noroozi and Paolo Favaro. Unsupervised learning of visual representations by solving jigsaw puzzles. In European conference on computer vision, pp. 69-84. Springer, 2016.
Aaron van den Oord, Yazhe Li, and Oriol Vinyals. Representation learning with contrastive predic- tive coding. arXiv preprini arXiv: 00744,3748 2018.
Zhiqiang Shen, Zechun Liu, Zhuang Liu, Marios Savvides, Trevor Darrell, and Eric Xing. Un- mix: Rethinking image mixtures for unsupervised visual representation learning, arXiv preprint arXiv:2003.080 054388 2020.
Jake Snell, Kevin Swersky, and Richard Zemel. Prototypical networks for few-shot learning. In Advances in neural information processing systems, pp- 4077-4087, 2017.
Gowthami Somepalli, Micah Goldblum, Avi Schwarzschild, C Bayan Bruss, and Tom Goldstein. Saint: Improved neural networks for tabular data via row attention and contrastive pre-training. arXiv preprint arXiv:2106. 01342, 2021.
Jong-Chyi Su, Subhransu Maji, and Bharath Hariharan. When does self-supervision improve few- shot learning? In European Conference on Computer Vision, pp. 645-666. Springer, 2020.
Yonglong Tian, Dilip Krishman, and Phillip Isola. Contrastive multiview coding. In Computer Vision-ECCV 2020: 16th European Conference, Glasgow, UK, August 23-28, 2020, Proceedings, Part XI 16, pp. 776-794. Springer, 2020a.
Yonglong Tian, Chen Sun, Ben Poole, Dilip Krishnan, Cordelia Schmid, and Phillip Isola. What makes for good views for contrastive learning? arXiv preprint arXi::2005. 2020b.
Joaquin Vanschoren, Jan N Van Rijn, Bernd Bischl, and Luis Torgo. Openml: networked science in machine learning. ACM SIGKDD Explorations Newsletter, 15(2):99-60, 2014.
Pascal Vincent, Hugo Larochelle, Yoshua Bengio, and Pierre-Antoine Manzagol. Extracting and composing robust features with denoising autoencoders. In Proceedings of the 25th international conference on Machine learning, pp. 1096-1103, 2008
Pascal Vincent, Hugo Larochelle, Isabelle Lajoie, Yoshua Bengio, Pierre-Antoins Manzagol, and Léon Bottou. Stacked denoising autoencoders: Learning useful representations in deep network with a local denoising criterion. Journal of machine learning research, 11(12), 2010.
Zhirong Wu, Yuanjun Xiong, Stella X Yu, and Dahua Lin. Unsupervised feature learning via non- parametric instance discrimination. Proceedings of the IEFE conference on computer vision and pattern recognition, pp. 3773--3722, 2018.
J. Xiao, J. Hays, K. A. Ehinger, A. Oliva, and A. Torralba. Sun database: Large-scale scene recog- nition from abbey to zoo. In 2010 IEEE Computer Society Conference on Computer Vision and Patlern Recognition, June 2010. doi: 10. N0.109//VPR.22100.55399
Han-Jia Ye, Lu Han, and De-Chuan Zhan. Revisiting unsupervised meta-learning: Amplifying or compensating for the characteristics of few-shot tasks. arXiv preprint arXiv:2011.14660 2020.
Yang You, Jing Li, Sashank Reddi, Jonathan Hseu, Sanjiv Kumar, Srinadh Bhojanapalli, Xiaodan Song, James Demmel, Kurt Keutzer, and Cho-Jui Hsieh. Large batch optimization for deep learning: Training bert in 76 minutes. arXiv preprint arXiv: 1904000690 2019.
Jure Zbontar, Li Jing, Ishan Misra, Yann LeCun, and Stéphane Deny. Barlow twins: Self-supervised learning via redundancy reduction. arXiv preprint trXiv:2103.032:: 2021.
Xiaohua Zhai, Avital Oliver, Alexander Kolesnikov. and Lucas Beyer. S41: Self-supervised semi-
supervised learning. Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 1476-1485, 2019.
Richard Zhang, Phillip Isola, and Alexei A Efros. Colorful image colorization. In European confer- ence on computer vision, pp. 649-666. Springer, 2016.
12

---

Published as a conference paper at ICLR 2022
A APPENDIX
A.1 PRE-TRAINING DETAILS
For ImageNet, we follow similar procedure as Chen et al. (000a) for both SimCLR and our pro- posed meta-learning algorithms, where we use batch size 256, a starting learning rate of 0.3 (fol- lowing LR = 0.3 x BatchSize/256), weight decay 10-6 and temperature 0.1. We accumulate the gradients and update our model every 8 iterations during the pre-training. For C0 we use batch size 1024, starting learning rate of 4. (following LR = 1. x BatchSizze2566, weight decay 10-6 and temperature 0.5.
A.2 EVALUATION DETAILS
Linear Evaluation. For ImageNet, we follow similar procedure as Chen et al. (2020a), where we use batch size 4096, a larger learning rate of 1.6 (following LR = 0. BatchSize/256) and longer training of 90 epochs. For CIFAR-10, we train the linear classifier for 200 epochs with batch size 1024, and a linear decay learning rate schedule starts with rate 0.1. We use SGD as the optimizer for linear evaluation on both datasets.
Semi-supervised Learning. Following Chen et al. (2020a), fine-tune the pre-trained model with 1% and 10% labeled ImageNet data. Due to the computational limit, instead of using batch size 4096, we use the SGD optimizer with batch size of 256 and momentum of 0.9. We linear decay learning rate schedule starts with rate 0.05. We only use the regular data augmentation for ImageNet training (random crop and resize). We do not any regularization such weight decay. For both 1% and 10% of the labeled data, we fine-tune for 60 epochs. The supervised baseline results are imported from Zhai et al. (2019), where they trained several thousand models for hyperparameter tuning and using strong data augmentations.
Transfer Learning. We evaluate the performance of models in transfer learning through fine- tuning. We use SSL pre-trained feature extractors with linear classification head as our model in transfer learning, and we update the entire network during fine-tuning. In this experiment, we use an SGD optimizer with Nesterov momentum (momentum=0.9), We set the batch size to be 256 and train our models for 20,000 iterations. For each dataset, we follow the raining/validatioon/sstigg split in Chen et al. (2020a) and perform the hyperparameter search on the validation set. We search through different learning rates (ranging from 10 to 10-1) and weight decays (ranging from 10-6 to 10 and 0), and choose the best setting by comparing the performance on the validation set. Results in Table are evaluated the test split, and our models trained on the union of training and validation set, using the best hyperparameter settings we have found.
A.3 THE SAMPLE SIZE FOR LARGE ROTATION AUXILIARY LOSS
Table& the linear evaluation accuracy for representations learned by different self-supervised learning algorithms with different rotation methods. A full rotation method rotates all the aug- mented views by times, a half rotation method rotates half of the augmented views by times, and the random rotation only rotates all the augmented views by an angle randomly chosen from (0%, 7000, 180" 270%). With different rotation methods, the number of training examples for rota- tion angle prediction is different. Among all the experiments, we use the same batch size 1024, and the full rotation will generate 1024 x 2 X 4 training examples, the half rotation will generate 512 x 2 x 4 examples and random rotation will only generate 1024 x 2 samples, where 2 represents for the number of the random augmentations and 4 is the length of the degree set. From Table g that even if we have much fewer training samples when applying random rotation, can still achieve comparable results. As a result, we use random rotation in all the experiments as makes the training more memory efficient.
13

---

Published as a conference paper at ICLR 2022
Table Linear evaluation (top-I accuracy (%)) on CIFAR-I0 with representations learned by vari- ous SSL methods with different rotation methods.
Method
SimCLR
R2-D2
PN
Random Rotation
93.0
92.9
93.0
Full Rotation
92.6
92.8
92.9
Half Rotation
92.9
92.6
92.7
A.4 MORE EVALUATIONS FOR LARGE ROTATION
In this section, provide additional evaluations for the model pre- pre-trained with proposed large
rotation in TableDand Table IO We conduct the same transfer learning and semi-supervised learning
experiments in Section 4
Table 9: ImageNet Top-1 accuracy (%) of models fine-tuned with few labels. "+ LLR" denotes adding large rotation as an auxiliary loss during pre-training
Label fraction
Method
Backbone
1%
10%
Supervised baseline
ResNet-50
25.4
56.4
SimCLR
ResNet-50
32.4
53.6
SimCLR + LLR
ResNet-50
32.0
53.5
Table 10: Transfer learning using ImageNet pre-trained weights. We report mean per-class accuracy (%) on the Flowers and Aircraft datasets, mean average precision (mAP) on the VOC2007 classifi- cation dataset, and Top-I accuracy on the remaining datasets. "+ LLR" denotes adding large rotation auxiliary loss during pre-training
Flowers102
VOC2007
Aircraft
Food101
SUN397
CIFAR-10
CIFAR-100
Baseline
92.0
64.8
67.3
85.9
86.9
53.6
95.9
80.2
SimCLR
92.4
72.7
66.0
83.7
86.3
57.4
94.8
79.1
SimCLR + CTR
93.1
73.0
64.4
85,8
86.3
57.0
96.2
82.0
A.5
PRE-TRAINING FOR TABULAR DATASET
Outside of computer vision, contrastive learning pre-training has been shown effective in the tabular data domain as well, especially for fine-tuning with limited data (Somepalli et al. 2021). To further show the effectiveness of meta-learning in this setting, pre-train tabular data with the R2D2 head and evaluate on semi-supervised tasks given only 50 training samples. We apply our methods to 5 datasets from OpenML (Vanschoren et al. 2014), and average over 5 runs with random train- test split on each dataset. Details of the selected datasets can be found in (Somepalli et al. (222I). For datasets with binary classification tasks, we evaluate the algorithm with mean AUROC scores, and for datasets with multi-class classification tasks, we evaluate the methods with mean accuracy. Table [T shows that our meta-learning based pre-training method can indeed achieve comparable accuracy to state-of-tt toee-o--ttherrrr contrastive learning methods on tabular data.
14

---

Published as a conference paper at ICLR 2022
Table 11: Semi-supervised evaluation with 50 labeled training samples on tabular datasets.
Dataset
Pre-train Nethods
Mean Score
Std
adult
84.46
1.82
Contrastive R2-D2
86.66
1.62
84.26
4.34
arrhythmia
72.36
6.20
Contrastive
75.94
4.32
R2-D2
75.64
3.26
telco-customer-chum
79.76
2.22
Contrastive
80.34
1.30
R2-D2
81.38
1.69
eucalyptus
45.06
3.68
Contrastive R2-D2
46.8
8.76
49.17
9.03
volkert
40.80
2.57
Contrastive R2-D2
39.22
2.07
40.44
1.76
15


Reviews:
[{"review_id": "wuZ1-cuZ9U", "reviewer": "Reviewer_KYMy", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strengths\n1. Learning self-supervised visual representations via meta-learning is new and interesting.\n2. The presented meta-learning framework for self-supervised learning improves over the SimCLR in downstream tasks. This shows the potential of using meta-learning for self-supervised visual representation learning.\n3. The tools from meta-learning are proved to benefit SimCLR.\n\nWeaknesses\n1. A core claim and also the title of the paper is \"contrastive learning is just meta-learning\": contrastive learning can be interpreted as a special case of meta-learning with a certain task distribution. However, its main arguments are not convincing to support this statement. In particular, the paper only discusses similar features (and also differences) shared by contrastive learning and meta-learning, such as solving new tasks on-the-fly with each batch and learning invariances which generalize to novel problems at inference. These shared features are not unique or characteristics to either meta-learning or contrastive learning. For example, deep metric learning via a triplet loss also solves new tasks (different anchors, positive/negative examples in each batch) and aims to learn features generalize to new problems (e.g., new faces for face recognition). To claim \"contrastive learning is just meta-learning\", one should present a general meta-learning framework and show how the contrastive learning framework can fit in that framework as a special case. Also, one should think about the \"core characteristics\" of these two frameworks, and show one is a subset of the other. For example, I did not see anything \"meta\" in SimCLR. Otherwise (only showing some shared features), it is similar to proving that \"elephants are just mice\" because they both have four legs, a head and a tail.\n\n2. The evaluation is inconsistent in different parts of the paper. Section 4 presents results from both linear protocol and downstream tasks while Section 5 and 6 only present the former. Some experiments are only on CIFAR-10 while some are only on ImageNet. It is unclear whether the data augmentation and gradient accumulation help meta-learning or SimCLR in downstream tasks.\n\n3. The idea of Large Rotation as Auxiliary Loss reads similar to prior work on classifying random image rotations as a pretext task for self-supervised learning. What is the main difference?\n\n4. The proposed meta-learning framework for self-supervised learning contains iterations in the inner loop. Prior self-supervised work indicates the number of training iterations/epochs can significantly affect the performance. An experiment is needed to the performance of meta-learning and SimCLR under different epochs/iterations. It is also interesting to see the extra training time added by the inner loop.\n\n5. Is the loss l on page 5 a contrastive loss?\n", "summary_of_the_paper": "The paper first shows that the current popular contrastive learning for self-supervised visual representation learning shares some similarity with a meta-learning framework for few-shot learning. This inspires the authors to propose a meta-learning framework for self-supervised learning. In addition, the paper also shows that tools (data augmentation and gradient accumulation) developed in meta-learning can help enhance contrastive learners. \n\nExperimental results show that the proposed meta-learning framework for self-supervised learning outperforms SimCLR, a state-of-the-art contrastive learning method, on multiple downstream tasks in a semi-supervised learning framework, though it performs not as well as SimCLR under the linear evaluation protocol on ImageNet.  Other results show tools developed in meta-learning can help enhance contrastive learners under the linear evaluation protocol.", "summary_of_the_review": "Learning self-supervised visual representations via meta-learning is new and interesting. Experimental results also indicate its potential effectiveness. But the main claim \"contrastive learning is just meta-learning\" is not well supported, as detailed in Main Review. The choice of datasets and tasks in different experiments is heuristic, making the results less convincing. Experiments on the impact of iterations within the inner loop are needed to show more in-depth comparison between the meta-learning framework and SimCLR.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "1: The main claims of the paper are incorrect or not at all supported by theory or empirical results.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "ofiDCPMMA5_", "reviewer": "Reviewer_AjN4", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strengths:\n1. Mathematical formalization of the connection between contrastive learning and meta-learning.\n2. New data augmentation techniques for contrastive learning lead to improvements on standard evaluations.\n3. Paper is clear and easy-to-read.\n4. Code is provided in the supplement.\n\nWeaknesses:\n1. The main weakness with the paper is the lack of comparison to the large body of work comparing unsupervised learning and meta-learning. Most notably, Hsu et al. (2019) showed that meta-learning algorithms can be used for unsupervised learning, and many of the papers that cite it have explored transferring techniques in both directions. The novelty here seems to be just the connection to contrastive learning specifically.\n2. It is unclear if any of the methods proposed here would extend to domains such as text, which is not examined.\n3. [minor] The paper title is aggressive, suggesting that we can stop doing contrastive learning and focus on meta-learning, even though the only connection is in the data generation during training, not in the goals of the two learning paradigms.\n\nComments:\n1. [*The training loop for meta-learners typically involves (i) sampling a random batch of classes and (ii) updating a feature extractor to distinguish between these classes.*] This describes the few-shot learning subset of meta-learning; meta-learning itself is much broader, including applications to supervised learning and RL that do not involve sampling classes.\n2. [*In the inner loop, the model is first fine-tuned on support data T_s_i . Then, in the outer loop, the updated model is used to predict on query data T_q_i , and a loss is minimized with respect to the model’s parameters before fine-tuning.*] Algorithms such as Reptile do not separate task data into support and query data.\n3. [*These methods usually require operations in pixel space, which is computationally expensive.*] What does this mean? Don’t most deep nets apply at least one operation to pixel space?\n\nReferences:\nHsu, Finn, Levine. *Unsupervised Learning via Meta-Learning*. ICLR 2019.", "summary_of_the_paper": "This paper formalizes a connection between the training procedures of (few-shot) meta-learning and contrastive learning. It shows that meta-learning algorithms can be used to pretrain on image data and outperform standard contrastive learning on downstream tasks. The authors further use the connection to develop data augmentation procedures for contrastive learning.", "summary_of_the_review": "The main concern with this work is that the main insight seems to be a somewhat more specific variant of observations made in past work on the connection between modern unsupervised and meta-learning. While there are some interesting experimental results, I lean against accepting given the lack of any analysis concerning what the novelty is here compared to those papers.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "Q5lpT_tIzK3", "reviewer": "Reviewer_AV99", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "1. The idea of integrating self-supervised learning into meta-learning is novel. The insight of treating different augmentation as task augmentation is conceptually interesting. Even though the paper does not extend such a framework to the level of other meta-learning literature that datasets and tasks are largely different, I believe the concept proposed in the paper may be a good inspiration for future self-supervised learning research.\n\n2. The first empirical idea proposed in the paper is exactly just rotNet (arXiv:1803.07728), and combining it with the joint-embedding approach is also not novel (\"Self-Supervised Representation Learning by Rotation Feature Decoupling\" CVPR'19). Therefore, it has limited novelty. I do not see how this setting is different when fitting into the meta-learning framework. It's still working as an auxiliary loss.\n\n3. The second empirical idea seems new to the self-supervised learning framework. Even though the idea itself comes from meta-learning is not novel, I think that demonstrating such an idea can work in a self-supervised learning framework is interesting and can probably benefit the general self-supervised learning research community.\n\n4. The authors provided all experimental details for reproducing the results. I understand that it's not possible to reproduce standard SimCLR due to computational budget but it seems that the comparison is not satisfactory. I do not see any hyperparameter search for the baseline SimCLR/BYOL model. These hyperparameters may not be optimal. For example, using a large learning rate of 4 and a high temperature of 0.5 may not be best for training small batch sizes such as 256. Also, all experiments improvements are incremental e.g. Table 6, 7, 8 only demonstrates +1%. Without an exhaustive hyperparameter search, this +1 % is not convincing enough.\n\n5. Overall, the paper is well written.  ", "summary_of_the_paper": "This paper proposed a framework to integrate contrastive/self-supervised learning into meta-learning literature. The authors demonstrate that contrastive learning principles implemented in meta-learning methods such as R2-D2 can achieve comparable results on various computer vision tasks. Next, the authors proposed two tricks in meta-learning literature to improve SSL models, including (1) rotation prediction (2) batch gradient accumulation. Both methods demonstrate incremental improvement over the standard SimCLR baseline.", "summary_of_the_review": "The concept of combining self-supervised learning and meta-learning is interesting and may have a bigger impact in the future. The first trick of rotation prediction has limited novelty. But the second trick is interesting and demonstrated to be useful. Empirical results are all incremental and not convincing enough.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "KZOKc5Bu0i", "reviewer": "Reviewer_vRLC", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Overall the paper is quite well written and easy to understand and follow. As said above, novelty might be a bit limited but I can see the benefit of highlighting the relationship of meta- and contrastive learning for many readers in the community. Experimental results are in favor of the approach, however a more compressive analysis (e.g., confidence intervals) would have been appreciated.", "summary_of_the_paper": "The paper at hand shows relations from self-supervised-learning (e.g., contrastive learning) and meta-learning (e.g., fine-tuning). Whereas the shown relationship is interesting it's not that novel and I would even argue that several teams have implemented similar ideas already. However it has not made that explicit, especially the gradient accumulation for contrastive learning which is quite relevant for many  learning in real world applications.", "summary_of_the_review": "A good summary of the state-of-the-art and highlighting some interesting relationships.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper was borderline, based on the reviews. The paper points out an interesting connection (somewhat known but not in this specific version) and good experimental results. However, numerous reviewers raised concerns that the paper was lacking a comparison to prior work connecting unsupervised learning and meta-learning, most notably, Hsu et al. (2019).

After reading the revised version of the paper, the authors address this issue and also all the other reviewer comments. In relation to prior work they clarify that they focus on the contrastive unsupervised case and also do a good job in answering other reviewer concerns relative to novelty and results. 

I would also like to point out, as reviewers also did that the previous title was a bit aggressive and provocative. Gladly the authors agree to change it to a more scientific `The Close Relationship Between Contrastive Learning and Meta-Learning”. 

Overall I think the authors have done a good effort on addressing the reviewer concerns and I think the paper would be interesting for ICLR readers.
Decision: Accept (Poster)