Contrastive Learning is Just Meta-Learning

1 INTRODUCTION . Self-supervised visual representation learning ( SSL ) has recently gathered attention due to its ability to learn image features without manual supervision , thus allowing for efficient learning on downstream tasks such as detection and segmentation ( Noroozi & Favaro , 2016 ; Zhang et al. , 2016 ; Oord et al. , 2018 ; Hjelm et al. , 2018 ; Wu et al. , 2018 ; Gidaris et al. , 2018 ; He et al. , 2020 ; Misra & Maaten , 2020 ; Tian et al. , 2020a ; Chen et al. , 2020a ; Kim et al. , 2020 ; Grill et al. , 2020 ; Caron et al. , 2020 ; Kalantidis et al. , 2020 ; Shen et al. , 2020 ; Li et al. , 2020 ; Zbontar et al. , 2021 ; Chen & He , 2021 ; Caron et al. , 2021 ) . Among SSL approaches , contrastive learning based methods ( Chen et al. , 2020a ; He et al. , 2020 ; Chen et al. , 2020b ) show particularly strong potential and achieve promising results which are close to those of fully supervised methods on numerous computer vision benchmarks . These methods rely on applying various data augmentations such as random crops , flips , color distortion , and Gaussian blur on the same training sample to create different views of an image . Two such example methods , SimCLR ( Chen et al. , 2020a ) and MoCo ( He et al. , 2020 ) , involve reducing the distance between features corresponding to positive pairs ( different augmented views of the same image ) , and increasing the distance between features corresponding to negative pairs ( augmented views of different images ) . Meanwhile , meta-learning is an established popular framework for learning models that quickly adapt to on-the-fly tasks given a small number of examples ( Hochreiter et al. , 2001 ; Finn et al. , 2017 ; Nichol et al. , 2018 ; Bertinetto et al. , 2018 ; Lee et al. , 2019 ) . The training loop for metalearners typically involves ( i ) sampling a random batch of classes and ( ii ) updating a feature extractor to distinguish between these classes . This procedure mirrors that of contrastive learning , which proceeds by ( i ) sampling a batch of images and augmenting them to generate classes ( each “ class ” is an image plus all of its views ) , and ( ii ) updating a feature extractor to distinguish between these classes . Conceptually , this contrastive learning procedure resembles meta-learning where the training tasks are generated by computing multiple views of individual images . In this paper , we discuss the close relationship between contrastive learning and meta-learning . Concretely , we show that established meta-learning algorithms , originally designed for few-shot learning , can achieve the same performance as recent contrastive learning algorithms on standard SSL problems when paired with the same data sampling strategy . In addition , we explore techniques , originally designed for meta-learning , that can improve contrastive learning . Specifically , we first explore ways by which we can adapt data augmentation strategies inspired by recent work in metalearning ( Su et al. , 2020 ; Ni et al. , 2021 ) to SSL and find that this approach can yield significant performance boosts . We further investigate the relationship between the number of negative pairs in a batch and the number of training tasks from a meta-learning perspective , and we thus improve the performance of contrastive learning for small batch sizes and memory constraints . Our contributions can be summarized as follows : • First , we propose a meta-learning based framework for understanding self-supervised learning , and we show that meta-learners can achieve comparable self-supervised performance to contrastive learning methods . • Second , we propose a meta-specific task augmentation strategy which boosts the performance of self-supervised learning . This data augmentation method generalizes to methods with no negative pairs , such as BYOL ( Grill et al. , 2020 ) , as well . • Finally , inspired by meta-learning training tasks , we show that a simple technique can improve the performance of contrastive learning in the small batch regime by removing the reliance on massive numbers of negative pairs , thus allowing for gradient accumulation . 2 RELATED WORK AND BACKGROUND . 2.1 META-LEARNING FOR FEW-SHOT LEARNING . Meta-learning algorithms for few-shot learning aim to learn a model that can quickly adapt to new tasks with limited data and generalize to unseen examples . To achieve this , the adaptation and evaluation procedures are both simulated during meta-training . During each episode of meta-learning , we sample a task , Ti , from a distribution of tasks , often corresponding to combinations of training classes formed into classification problems . Each task consists of support data T si and query data T qi , so that Ti = { T si , T q i } . When applied to few-shot classification , this task is called a k-shot , N -way classification problem , where k denotes the number of training samples per category in the support data . Then , support data will be used to simulate few-shot training data , while query data will be used to simulate novel testing samples . A meta-learning model F , in this setting , contains a feature extractor and a classification strategy , A . This classification strategy can take various forms , such as adding a linear classifier on top of the feature extractor and fine-tuning either the linear layer or the whole network end-to-end , or this strategy may simply classify samples by selecting the nearest class prototype . Meta-learning training algorithms have an inner loop and an outer loop in each parameter update . In the inner loop , the model is first fine-tuned on support data T si . Then , in the outer loop , the updated model is used to predict on query data T qi , and a loss is minimized with respect to the model ’ s parameters before fine-tuning.1 Intuitively , we update parameters so that the feature extractor extracts better features for the classification strategy , often resulting in tightly clustered features corresponding to each class ( Goldblum et al. , 2020 ) . Existing works apply various methods for fine-tuning on support data during the inner loop . In a line of algorithms , such as MAML and Reptile ( Finn et al. , 2017 ; Nichol et al. , 2018 ) , all the parameters in the model are updated using gradient descent during fine-tuning on support data . Other algorithms , such as MetaOptNet and R2-D2 ( Lee et al. , 2019 ; Bertinetto et al. , 2018 ) , keep the feature extractor frozen during fine-tuning ; MetaOptNet uses SVM , and R2D2 uses ridge regression on top of the feature extractor . Similarly , metric learning approaches , such as ProtoNet ( Snell et al. , 2017 ; Kye et al. , 2020 ) , freeze the feature extractor as well , and create class centroids from the support data in the inner loop . In this paper , we primarily focus on the latter algorithms due to their efficiency and performance as well as the similarity of contrastive learning to metric learning . 2.2 SELF-SUPERVISED LEARNING . Contrastive SSL . Contrastive methods ( Oord et al. , 2018 ; Wu et al. , 2018 ; Tian et al. , 2020a ; Chen et al. , 2020a ; He et al. , 2020 ; Chen et al. , 2020b ) achieve promising performance in self-supervised 1Note that algorithms in the vein of Reptile ( Nichol et al. , 2018 ) do not split the tasks into support and query . learning . As mentioned previously , contrastive learning maximizes agreement on different augmented views of the same image ( called positive pairs ) while ensuring disagreement on samples generated by different base images ( called negative pairs ) . Given a batch of input images x , m random data augmentations are applied on the same batch , generating a set of training samples { x̃i } mi=1 . These samples are fed into a backbone network f ( · ) to obtain the feature representations { hi } mi=1 . Then , a small neural network g ( · ) , usually a non-linear MLP , is applied to project { hi } mi=1 to the latent representations { zi } mi=1 in the space where a contrastive loss l ( · ) is applied , ensuring that latent representations of positive pairs are similar while latent representations of negative pairs are different . In this paper , we mainly focus on this type of self-supervised learning and show its close relation with meta-learning . The batch sampling procedure of contrastive learning can be viewed as sampling a new classification problem with a number of classes equal to the number of base images used to generate augmented views . We will see that this on-the-fly sampling of classification problems closely mirrors a common meta-learning setup . Non-Contrastive SSL . Some non-contrastive methods are generative approaches , such as autoencoders ( Vincent et al. , 2008 ; 2010 ; Kingma & Welling , 2013 ) , and adversarial learning ( Goodfellow et al. , 2014 ) , where a distribution is learned over data and a latent embedding . These methods are typically computationally expensive as they require training a learned model which maps latent representations to pixel space . Other non-contrastive methods rely on using heuristic designed pretext tasks ( Doersch et al. , 2015 ; Zhang et al. , 2016 ; Noroozi & Favaro , 2016 ; Gidaris et al. , 2018 ) to learn the representation . More recently , BYOL ( Grill et al. , 2020 ) showed that by bootstrapping a target representation prediction , feature representations can be learned without negative pairs . However , BYOL still adopts the data augmentation procedure from contrastive learning , where different augmented views are used as training samples . In section 5 , we show that BYOL still benefits from our proposed task augmentation strategy . Data Augmentation in Meta-Learning and SSL . Data augmentations play an essential role in both meta-learning and self-supervised learning . In meta-learning , Su et al . ( 2020 ) ; Liu et al . ( 2020 ) ; Ni et al . ( 2021 ) show that proper data augmentation and meta-specific task augmentations dramatically improve few-shot learning performance by expanding the number of classes available for sampling . In self-supervised learning , Tian et al . ( 2020b ) show contrastive learners can find better feature representations when views contain less mutual information . In addition , Kim et al . ( 2020 ) ; Shen et al . ( 2020 ) ; Li et al . ( 2020 ) show that adding harder examples such as cut-mixed samples into the training pipeline can improve self-supervised performance . In this paper , we show that similarly to metalearning , self-supervised learners can benefit from carefully applied data augmentation techniques which mirror task augmentations from the meta-learning literature . 2.3 SELF-SUPERVISED LEARNING FOR META-LEARNING AND FEW-SHOT LEARNING . Another line of research ( Hsu et al. , 2018 ; Khodadadeh et al. , 2018 ; Ye et al. , 2020 ; Medina et al. , 2020 ) focuses on self-supervised learning for meta-learning and few-shot learning . Hsu et al . ( 2018 ) focuses on partitioning samples from a dataset to construct meta-learning tasks and using MAML or ProtoNet on 4-layer architectures to solve few-shot problems . Similarly , Khodadadeh et al . ( 2018 ) and Medina et al . ( 2020 ) add more data augmentations , such as Auto Aug ( Cubuk et al. , 2018 ) and use MAML and ProtoNet , respectively , to learn a few-shot representation . To further improve few-shot performance , Ye et al . ( 2020 ) sample harder mixed support examples and apply a taskspecific projection head to help generalize to unseen classes . These methods focus on few-shot learning performance , which entails up to 50 training examples per class . In contrast , we focus on the unsupervised learning paradigm where large models are pre-trained on samples generated via data augmentation and are applied to downstream tasks such as ImageNet classification .

Summary: The paper at hand shows relations from self-supervised-learning (e.g., contrastive learning) and meta-learning (e.g., fine-tuning). Whereas the shown relationship is interesting it's not that novel and I would even argue that several teams have implemented similar ideas already. However it has not made that explicit, especially the gradient accumulation for contrastive learning which is quite relevant for many  learning in real world applications.