Contrastive Learning is Just Meta-Learning

1 INTRODUCTION . Self-supervised visual representation learning ( SSL ) has recently gathered attention due to its ability to learn image features without manual supervision , thus allowing for efficient learning on downstream tasks such as detection and segmentation ( Noroozi & Favaro , 2016 ; Zhang et al. , 2016 ; Oord et al. , 2018 ; Hjelm et al. , 2018 ; Wu et al. , 2018 ; Gidaris et al. , 2018 ; He et al. , 2020 ; Misra & Maaten , 2020 ; Tian et al. , 2020a ; Chen et al. , 2020a ; Kim et al. , 2020 ; Grill et al. , 2020 ; Caron et al. , 2020 ; Kalantidis et al. , 2020 ; Shen et al. , 2020 ; Li et al. , 2020 ; Zbontar et al. , 2021 ; Chen & He , 2021 ; Caron et al. , 2021 ) . Among SSL approaches , contrastive learning based methods ( Chen et al. , 2020a ; He et al. , 2020 ; Chen et al. , 2020b ) show particularly strong potential and achieve promising results which are close to those of fully supervised methods on numerous computer vision benchmarks . These methods rely on applying various data augmentations such as random crops , flips , color distortion , and Gaussian blur on the same training sample to create different views of an image . Two such example methods , SimCLR ( Chen et al. , 2020a ) and MoCo ( He et al. , 2020 ) , involve reducing the distance between features corresponding to positive pairs ( different augmented views of the same image ) , and increasing the distance between features corresponding to negative pairs ( augmented views of different images ) . Meanwhile , meta-learning is an established popular framework for learning models that quickly adapt to on-the-fly tasks given a small number of examples ( Hochreiter et al. , 2001 ; Finn et al. , 2017 ; Nichol et al. , 2018 ; Bertinetto et al. , 2018 ; Lee et al. , 2019 ) . The training loop for metalearners typically involves ( i ) sampling a random batch of classes and ( ii ) updating a feature extractor to distinguish between these classes . This procedure mirrors that of contrastive learning , which proceeds by ( i ) sampling a batch of images and augmenting them to generate classes ( each “ class ” is an image plus all of its views ) , and ( ii ) updating a feature extractor to distinguish between these classes . Conceptually , this contrastive learning procedure resembles meta-learning where the training tasks are generated by computing multiple views of individual images . In this paper , we discuss the close relationship between contrastive learning and meta-learning . Concretely , we show that established meta-learning algorithms , originally designed for few-shot learning , can achieve the same performance as recent contrastive learning algorithms on standard SSL problems when paired with the same data sampling strategy . In addition , we explore techniques , originally designed for meta-learning , that can improve contrastive learning . Specifically , we first explore ways by which we can adapt data augmentation strategies inspired by recent work in metalearning ( Su et al. , 2020 ; Ni et al. , 2021 ) to SSL and find that this approach can yield significant performance boosts . We further investigate the relationship between the number of negative pairs in a batch and the number of training tasks from a meta-learning perspective , and we thus improve the performance of contrastive learning for small batch sizes and memory constraints . Our contributions can be summarized as follows : • First , we propose a meta-learning based framework for understanding self-supervised learning , and we show that meta-learners can achieve comparable self-supervised performance to contrastive learning methods . • Second , we propose a meta-specific task augmentation strategy which boosts the performance of self-supervised learning . This data augmentation method generalizes to methods with no negative pairs , such as BYOL ( Grill et al. , 2020 ) , as well . • Finally , inspired by meta-learning training tasks , we show that a simple technique can improve the performance of contrastive learning in the small batch regime by removing the reliance on massive numbers of negative pairs , thus allowing for gradient accumulation . 2 RELATED WORK AND BACKGROUND . 2.1 META-LEARNING FOR FEW-SHOT LEARNING . Meta-learning algorithms for few-shot learning aim to learn a model that can quickly adapt to new tasks with limited data and generalize to unseen examples . To achieve this , the adaptation and evaluation procedures are both simulated during meta-training . During each episode of meta-learning , we sample a task , Ti , from a distribution of tasks , often corresponding to combinations of training classes formed into classification problems . Each task consists of support data T si and query data T qi , so that Ti = { T si , T q i } . When applied to few-shot classification , this task is called a k-shot , N -way classification problem , where k denotes the number of training samples per category in the support data . Then , support data will be used to simulate few-shot training data , while query data will be used to simulate novel testing samples . A meta-learning model F , in this setting , contains a feature extractor and a classification strategy , A . This classification strategy can take various forms , such as adding a linear classifier on top of the feature extractor and fine-tuning either the linear layer or the whole network end-to-end , or this strategy may simply classify samples by selecting the nearest class prototype . Meta-learning training algorithms have an inner loop and an outer loop in each parameter update . In the inner loop , the model is first fine-tuned on support data T si . Then , in the outer loop , the updated model is used to predict on query data T qi , and a loss is minimized with respect to the model ’ s parameters before fine-tuning.1 Intuitively , we update parameters so that the feature extractor extracts better features for the classification strategy , often resulting in tightly clustered features corresponding to each class ( Goldblum et al. , 2020 ) . Existing works apply various methods for fine-tuning on support data during the inner loop . In a line of algorithms , such as MAML and Reptile ( Finn et al. , 2017 ; Nichol et al. , 2018 ) , all the parameters in the model are updated using gradient descent during fine-tuning on support data . Other algorithms , such as MetaOptNet and R2-D2 ( Lee et al. , 2019 ; Bertinetto et al. , 2018 ) , keep the feature extractor frozen during fine-tuning ; MetaOptNet uses SVM , and R2D2 uses ridge regression on top of the feature extractor . Similarly , metric learning approaches , such as ProtoNet ( Snell et al. , 2017 ; Kye et al. , 2020 ) , freeze the feature extractor as well , and create class centroids from the support data in the inner loop . In this paper , we primarily focus on the latter algorithms due to their efficiency and performance as well as the similarity of contrastive learning to metric learning . 2.2 SELF-SUPERVISED LEARNING . Contrastive SSL . Contrastive methods ( Oord et al. , 2018 ; Wu et al. , 2018 ; Tian et al. , 2020a ; Chen et al. , 2020a ; He et al. , 2020 ; Chen et al. , 2020b ) achieve promising performance in self-supervised 1Note that algorithms in the vein of Reptile ( Nichol et al. , 2018 ) do not split the tasks into support and query . learning . As mentioned previously , contrastive learning maximizes agreement on different augmented views of the same image ( called positive pairs ) while ensuring disagreement on samples generated by different base images ( called negative pairs ) . Given a batch of input images x , m random data augmentations are applied on the same batch , generating a set of training samples { x̃i } mi=1 . These samples are fed into a backbone network f ( · ) to obtain the feature representations { hi } mi=1 . Then , a small neural network g ( · ) , usually a non-linear MLP , is applied to project { hi } mi=1 to the latent representations { zi } mi=1 in the space where a contrastive loss l ( · ) is applied , ensuring that latent representations of positive pairs are similar while latent representations of negative pairs are different . In this paper , we mainly focus on this type of self-supervised learning and show its close relation with meta-learning . The batch sampling procedure of contrastive learning can be viewed as sampling a new classification problem with a number of classes equal to the number of base images used to generate augmented views . We will see that this on-the-fly sampling of classification problems closely mirrors a common meta-learning setup . Non-Contrastive SSL . Some non-contrastive methods are generative approaches , such as autoencoders ( Vincent et al. , 2008 ; 2010 ; Kingma & Welling , 2013 ) , and adversarial learning ( Goodfellow et al. , 2014 ) , where a distribution is learned over data and a latent embedding . These methods are typically computationally expensive as they require training a learned model which maps latent representations to pixel space . Other non-contrastive methods rely on using heuristic designed pretext tasks ( Doersch et al. , 2015 ; Zhang et al. , 2016 ; Noroozi & Favaro , 2016 ; Gidaris et al. , 2018 ) to learn the representation . More recently , BYOL ( Grill et al. , 2020 ) showed that by bootstrapping a target representation prediction , feature representations can be learned without negative pairs . However , BYOL still adopts the data augmentation procedure from contrastive learning , where different augmented views are used as training samples . In section 5 , we show that BYOL still benefits from our proposed task augmentation strategy . Data Augmentation in Meta-Learning and SSL . Data augmentations play an essential role in both meta-learning and self-supervised learning . In meta-learning , Su et al . ( 2020 ) ; Liu et al . ( 2020 ) ; Ni et al . ( 2021 ) show that proper data augmentation and meta-specific task augmentations dramatically improve few-shot learning performance by expanding the number of classes available for sampling . In self-supervised learning , Tian et al . ( 2020b ) show contrastive learners can find better feature representations when views contain less mutual information . In addition , Kim et al . ( 2020 ) ; Shen et al . ( 2020 ) ; Li et al . ( 2020 ) show that adding harder examples such as cut-mixed samples into the training pipeline can improve self-supervised performance . In this paper , we show that similarly to metalearning , self-supervised learners can benefit from carefully applied data augmentation techniques which mirror task augmentations from the meta-learning literature . 2.3 SELF-SUPERVISED LEARNING FOR META-LEARNING AND FEW-SHOT LEARNING . Another line of research ( Hsu et al. , 2018 ; Khodadadeh et al. , 2018 ; Ye et al. , 2020 ; Medina et al. , 2020 ) focuses on self-supervised learning for meta-learning and few-shot learning . Hsu et al . ( 2018 ) focuses on partitioning samples from a dataset to construct meta-learning tasks and using MAML or ProtoNet on 4-layer architectures to solve few-shot problems . Similarly , Khodadadeh et al . ( 2018 ) and Medina et al . ( 2020 ) add more data augmentations , such as Auto Aug ( Cubuk et al. , 2018 ) and use MAML and ProtoNet , respectively , to learn a few-shot representation . To further improve few-shot performance , Ye et al . ( 2020 ) sample harder mixed support examples and apply a taskspecific projection head to help generalize to unseen classes . These methods focus on few-shot learning performance , which entails up to 50 training examples per class . In contrast , we focus on the unsupervised learning paradigm where large models are pre-trained on samples generated via data augmentation and are applied to downstream tasks such as ImageNet classification .

Summary: The paper first shows that the current popular contrastive learning for self-supervised visual representation learning shares some similarity with a meta-learning framework for few-shot learning. This inspires the authors to propose a meta-learning framework for self-supervised learning. In addition, the paper also shows that tools (data augmentation and gradient accumulation) developed in meta-learning can help enhance contrastive learners.   Experimental results show that the proposed meta-learning framework for self-supervised learning outperforms SimCLR, a state-of-the-art contrastive learning method, on multiple downstream tasks in a semi-supervised learning framework, though it performs not as well as SimCLR under the linear evaluation protocol on ImageNet.  Other results show tools developed in meta-learning can help enhance contrastive learners under the linear evaluation protocol.