Venue: ICLR 2022 Conference

Reviews:

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: The paper at hand shows relations from self-supervised-learning (e.g., contrastive learning) and meta-learning (e.g., fine-tuning). Whereas the shown relationship is interesting it's not that novel and I would even argue that several teams have implemented similar ideas already. However it has not made that explicit, especially the gradient accumulation for contrastive learning which is quite relevant for many  learning in real world applications.
main_review: Overall the paper is quite well written and easy to understand and follow. As said above, novelty might be a bit limited but I can see the benefit of highlighting the relationship of meta- and contrastive learning for many readers in the community. Experimental results are in favor of the approach, however a more compressive analysis (e.g., confidence intervals) would have been appreciated.
summary_of_the_review: A good summary of the state-of-the-art and highlighting some interesting relationships.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: This paper formalizes a connection between the training procedures of (few-shot) meta-learning and contrastive learning. It shows that meta-learning algorithms can be used to pretrain on image data and outperform standard contrastive learning on downstream tasks. The authors further use the connection to develop data augmentation procedures for contrastive learning.
main_review: Strengths:
1. Mathematical formalization of the connection between contrastive learning and meta-learning.
2. New data augmentation techniques for contrastive learning lead to improvements on standard evaluations.
3. Paper is clear and easy-to-read.
4. Code is provided in the supplement.

Weaknesses:
1. The main weakness with the paper is the lack of comparison to the large body of work comparing unsupervised learning and meta-learning. Most notably, Hsu et al. (2019) showed that meta-learning algorithms can be used for unsupervised learning, and many of the papers that cite it have explored transferring techniques in both directions. The novelty here seems to be just the connection to contrastive learning specifically.
2. It is unclear if any of the methods proposed here would extend to domains such as text, which is not examined.
3. [minor] The paper title is aggressive, suggesting that we can stop doing contrastive learning and focus on meta-learning, even though the only connection is in the data generation during training, not in the goals of the two learning paradigms.

Comments:
1. [*The training loop for meta-learners typically involves (i) sampling a random batch of classes and (ii) updating a feature extractor to distinguish between these classes.*] This describes the few-shot learning subset of meta-learning; meta-learning itself is much broader, including applications to supervised learning and RL that do not involve sampling classes.
2. [*In the inner loop, the model is first fine-tuned on support data T_s_i . Then, in the outer loop, the updated model is used to predict on query data T_q_i , and a loss is minimized with respect to the model’s parameters before fine-tuning.*] Algorithms such as Reptile do not separate task data into support and query data.
3. [*These methods usually require operations in pixel space, which is computationally expensive.*] What does this mean? Don’t most deep nets apply at least one operation to pixel space?

References:
Hsu, Finn, Levine. *Unsupervised Learning via Meta-Learning*. ICLR 2019.
summary_of_the_review: The main concern with this work is that the main insight seems to be a somewhat more specific variant of observations made in past work on the connection between modern unsupervised and meta-learning. While there are some interesting experimental results, I lean against accepting given the lack of any analysis concerning what the novelty is here compared to those papers.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: The paper first shows that the current popular contrastive learning for self-supervised visual representation learning shares some similarity with a meta-learning framework for few-shot learning. This inspires the authors to propose a meta-learning framework for self-supervised learning. In addition, the paper also shows that tools (data augmentation and gradient accumulation) developed in meta-learning can help enhance contrastive learners. 

Experimental results show that the proposed meta-learning framework for self-supervised learning outperforms SimCLR, a state-of-the-art contrastive learning method, on multiple downstream tasks in a semi-supervised learning framework, though it performs not as well as SimCLR under the linear evaluation protocol on ImageNet.  Other results show tools developed in meta-learning can help enhance contrastive learners under the linear evaluation protocol.
main_review: Strengths
1. Learning self-supervised visual representations via meta-learning is new and interesting.
2. The presented meta-learning framework for self-supervised learning improves over the SimCLR in downstream tasks. This shows the potential of using meta-learning for self-supervised visual representation learning.
3. The tools from meta-learning are proved to benefit SimCLR.

Weaknesses
1. A core claim and also the title of the paper is "contrastive learning is just meta-learning": contrastive learning can be interpreted as a special case of meta-learning with a certain task distribution. However, its main arguments are not convincing to support this statement. In particular, the paper only discusses similar features (and also differences) shared by contrastive learning and meta-learning, such as solving new tasks on-the-fly with each batch and learning invariances which generalize to novel problems at inference. These shared features are not unique or characteristics to either meta-learning or contrastive learning. For example, deep metric learning via a triplet loss also solves new tasks (different anchors, positive/negative examples in each batch) and aims to learn features generalize to new problems (e.g., new faces for face recognition). To claim "contrastive learning is just meta-learning", one should present a general meta-learning framework and show how the contrastive learning framework can fit in that framework as a special case. Also, one should think about the "core characteristics" of these two frameworks, and show one is a subset of the other. For example, I did not see anything "meta" in SimCLR. Otherwise (only showing some shared features), it is similar to proving that "elephants are just mice" because they both have four legs, a head and a tail.

2. The evaluation is inconsistent in different parts of the paper. Section 4 presents results from both linear protocol and downstream tasks while Section 5 and 6 only present the former. Some experiments are only on CIFAR-10 while some are only on ImageNet. It is unclear whether the data augmentation and gradient accumulation help meta-learning or SimCLR in downstream tasks.

3. The idea of Large Rotation as Auxiliary Loss reads similar to prior work on classifying random image rotations as a pretext task for self-supervised learning. What is the main difference?

4. The proposed meta-learning framework for self-supervised learning contains iterations in the inner loop. Prior self-supervised work indicates the number of training iterations/epochs can significantly affect the performance. An experiment is needed to the performance of meta-learning and SimCLR under different epochs/iterations. It is also interesting to see the extra training time added by the inner loop.

5. Is the loss l on page 5 a contrastive loss?

summary_of_the_review: Learning self-supervised visual representations via meta-learning is new and interesting. Experimental results also indicate its potential effectiveness. But the main claim "contrastive learning is just meta-learning" is not well supported, as detailed in Main Review. The choice of datasets and tasks in different experiments is heuristic, making the results less convincing. Experiments on the impact of iterations within the inner loop are needed to show more in-depth comparison between the meta-learning framework and SimCLR.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper proposed a framework to integrate contrastive/self-supervised learning into meta-learning literature. The authors demonstrate that contrastive learning principles implemented in meta-learning methods such as R2-D2 can achieve comparable results on various computer vision tasks. Next, the authors proposed two tricks in meta-learning literature to improve SSL models, including (1) rotation prediction (2) batch gradient accumulation. Both methods demonstrate incremental improvement over the standard SimCLR baseline.
main_review: 1. The idea of integrating self-supervised learning into meta-learning is novel. The insight of treating different augmentation as task augmentation is conceptually interesting. Even though the paper does not extend such a framework to the level of other meta-learning literature that datasets and tasks are largely different, I believe the concept proposed in the paper may be a good inspiration for future self-supervised learning research.

2. The first empirical idea proposed in the paper is exactly just rotNet (arXiv:1803.07728), and combining it with the joint-embedding approach is also not novel ("Self-Supervised Representation Learning by Rotation Feature Decoupling" CVPR'19). Therefore, it has limited novelty. I do not see how this setting is different when fitting into the meta-learning framework. It's still working as an auxiliary loss.

3. The second empirical idea seems new to the self-supervised learning framework. Even though the idea itself comes from meta-learning is not novel, I think that demonstrating such an idea can work in a self-supervised learning framework is interesting and can probably benefit the general self-supervised learning research community.

4. The authors provided all experimental details for reproducing the results. I understand that it's not possible to reproduce standard SimCLR due to computational budget but it seems that the comparison is not satisfactory. I do not see any hyperparameter search for the baseline SimCLR/BYOL model. These hyperparameters may not be optimal. For example, using a large learning rate of 4 and a high temperature of 0.5 may not be best for training small batch sizes such as 256. Also, all experiments improvements are incremental e.g. Table 6, 7, 8 only demonstrates +1%. Without an exhaustive hyperparameter search, this +1 % is not convincing enough.

5. Overall, the paper is well written.  
summary_of_the_review: The concept of combining self-supervised learning and meta-learning is interesting and may have a bigger impact in the future. The first trick of rotation prediction has limited novelty. But the second trick is interesting and demonstrated to be useful. Empirical results are all incremental and not convincing enough.
review_title: null

Meta-review: This paper was borderline, based on the reviews. The paper points out an interesting connection (somewhat known but not in this specific version) and good experimental results. However, numerous reviewers raised concerns that the paper was lacking a comparison to prior work connecting unsupervised learning and meta-learning, most notably, Hsu et al. (2019).

After reading the revised version of the paper, the authors address this issue and also all the other reviewer comments. In relation to prior work they clarify that they focus on the contrastive unsupervised case and also do a good job in answering other reviewer concerns relative to novelty and results. 

I would also like to point out, as reviewers also did that the previous title was a bit aggressive and provocative. Gladly the authors agree to change it to a more scientific `The Close Relationship Between Contrastive Learning and Meta-Learning”. 

Overall I think the authors have done a good effort on addressing the reviewer concerns and I think the paper would be interesting for ICLR readers.
Decision: Accept (Poster)