Venue: ICLR 2022 Conference

Reviews:

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: The paper analyzes MAML algorithms. Assuming in the inner loop the encoder is fixed and only last linear layer is updated, they analyze the gradient update and loss terms in the inner loop and outer loops (sec 2.3). Through this effort, the authors claim that there are noisy supervised contrastive term in the outer loop loss (eqn (7) and (8)). They further claim that there are additional interference terms which may degrade the performance of MAML at the beginning of training when the linear layer weights are largely random. To overcome this, they propose a simple zeroing trick by zeroing the initial linear layer weights after each outer loop update, essentially removing the interference terms (eqn (9) and (10)). They conduct experiments to support the contrastiveness in MAML and performance improvement using zeroing trick. 
main_review: Strengths 
=======

The analysis is quite interesting. Their efforts make it explicit the interaction between the support and query set in MAML, and how MAML learns feature encoding. They discover a noisy supervised contrastive loss term in the outer loop loss using the support features as positive and negative samples. Interestingly, they discover another interference term in the outer loop loss which may degrade performance of MAML if last layer linear weights are randomly initialized.  

To remove the interference term, and also to remove the noise in the supervised contrastive loss, they propose a simple zeroing trick to set the initial weights to be zero after each outer loop update. The trick is simple and reasonable given their analysis results.  

They conduct experiments to verify the contrastiveness in MAML and improvements using the zeroing trick.  

Overall, the paper is quite clear and easy to follow. 

Question and weakness 
===================

Comparison between eqn (7) and (8) is not clear. Can the authors discuss further the different range of stop gradient? 

The results in Figure 2 are nice but limited, not sure if these can be observed for other classes. What are the 5 classes in the experiment? Can the authors experiment other classes, especially some semantically similar classes? 

Further to the analysis in Figure 2, I am not sure if they really "verify the supervised contrastiveness". In particular, the results show that support and query features of same classes become similar as training progresses. I do not think this is particular for supervised contrastiveness. For example, authors can try standard transfer learning and fine-tuning and see if such pattern of consine similarities can be observed. 

I am ok with the assumption of freezing the encoder in the inner loop and update only linear layer. But with that, I thought some remarks of the paper sound trivial. For example, “In the inner loop, the features of support data are preserved in the linear layer via inner loop update. In the outer loop, the softmax output of the query data thus contains the inner products between the support features and the query feature.” If you can update only the last linear layer in the inner loop then of course support data feature can only reside there, and therefore in the outer loop it would be inner products between support features and query input as the last layer is a linear layer. Did I miss anything? 

The analysis focuses on classification. Inner product and softmax are critical components in their analysis. But MAML has been applied to other problems, e.g. regression. Perhaps the focus on classification should be reflected in the paper title. 

 
summary_of_the_review: Several concerns are listed above. I would be happy to re-consider my recommendation if the responses are reasonable.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: In this paper, a new view of MAML under few-shot learning is proposed. The main result is that under the assumption that the inner loop updates are only applied on the top linear layer, MAML actually performs supervised contrastive learning (SCL). SCL shows that MAML learns the feature transformation that makes the intra-class feature distances small, meanwhile the inter-class feature distances large. The zeroing trick is proposed based on this result, showing performance gain in the experiments. 


main_review: Strengths:
- The SCL view proposed in the paper shows the link between MAML and the metric-based approaches for few-shot learning, such as matching network and prototypical network. In my view, this is interesting and might inspire future improvement of MAML for few-shot learning.

Weaknesses:
- The analysis in the paper is restricted. The SCL view seems to be only valid for few-shot learning. It is also over-simplified to assume that the inner-loop update does not affect the feature backbone if the task is not few-shot learning. I think a better title for the paper is "MAML is approximately a noisy contrastive learner for few-shot learning". 

- There are no comparisons of MAML with the zeroing trick to metric-based few-shot learning methods in the experiments.  In my view, the SCL view of MAML indeed says that MAML is similar to metric-based approaches, especially when the zeroing trick is applied. The difference only lies in that metric-based approaches explicitly introduce metrics, while MAML with zeroing trick does perceptron-like learning on the top layer. I am surprised to see that the metric-based approaches are not even mentioned in the paper. It would make the results in the paper more insightful if this connection is explored in depth.
summary_of_the_review: To summarize, I think the SCL view proposed in the paper would be interesting for researchers of few-shot learning. While the result is restricted to few-shot learning, meanwhile no exploration is done to analyse the connection between MAML and metric-based approaches. These drawbacks make the paper less insightful as I expected.
review_title: null

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: This work shows that, in the setting of few-shot classification, MAML is a (noisy) contrastive learner. Complementary theoretical and experimental results are provided. The theoretical results also lead to a (to my knowledge) new proposal, namely to zero out the linear layer after each MAML outer loop update. In their experiments, the authors show that making this small change to the MAML algorithm can lead to meaningful improvements in performance.
main_review: I like the simplicity of the zeroing trick, and I found it useful to see it motivated via your theoretical arguments / perform well in your numerical experiments.

In my view, Section 2.1 could have done a better job of motivating why being a supervised contrastive learner is so appealing. This seems critical given that the rest of the paper provides conditions and theoretical/numerical results supporting that MAML is a (noisy) contrastive learner under certain conditions.

Your theoretical analysis focuses on a specific setting: few-shot classification tasks, softmax output, frozen convolutional encoder during inner loop To what extent do your findings generalize beyond these settings? I had been hoping that the discussion in Section 2.6 ("Generalization of our Analysis") would shed some light on this, but it only seemed to address one specific aspect of your analysis, namely the fact that you took $N_{batch}=N_{step}=1$.

Along the lines of the above, I think that the title and abstract are overly general (neither explicitly referencing your focus on few-shot classification problems, for example) given what you actually showed in the paper.

Sorry if I missed this, but did you freeze the encoder in the inner loop for your simulations? If so, did you also evaluate what happens when you didn't freeze it? This would help to give some experimental indication of how general your theoretical findings might be.
summary_of_the_review: I found the proposed zeroing trick (and the supporting theory/experiments) to be interesting. I would, however, encourage the authors to work to better motivate their study in the intro of the paper.
review_title: null

Meta-review: This paper connects MAML to contrastive learning under some simplifying assumptions and with slight modifications in the setting. 
Specifically, the authors show that if the inner loop updates are only applied on the top linear layer, MAML is equivalent to supervised contrastive learning (SCL). This means that MAML learns a feature transformation that brings in-class representations closer and representations across classes far away. The zeroing trick the authors propose seems to give some performance gain in the experiments and is an actionable insight from their theory. 

Overall the paper is very interesting and (as far as I know) novel. The proposed zeroing trick is supported by theory and experiments and is justifies the previous theoretical narrative. 

Some reviewers raised concerns on motivation and numerous clarification questions that the authors have addressed to a large extent in my opinion.
Decision: Accept (Poster)