Venue: ICLR 2021 Conference

Reviews:

final_score: rating: 3: Clear rejection
review_content: Summary: 

In this paper, the authors investigate the inner-loop optimization mechanism of meta-learning algorithms. The analysis shows the effectiveness of the multi-step adaptation and (1) the key of meta-learning is how to design a well-differentiated classifier. They then propose Random Decision Planes (RDP) and Meta Contrastive Learning (MCL) and achieve comparable performance with existing methods.


Pros:
1. Empirically investigating the performance w.r.t. the change of optimization mechanism on both head and body is very important in the meta-learning field.

Cons:
1. My major concern is about the contribution of this paper. 
 - The discussion is interesting. But most of the analysis results is widely accepted. For example, adapting the head layer improves performance. One claim  "as the model converges, the body accuracy even decreases in the first few adaptation steps" is not well-explained.
 - The final goal of this paper is to design a better metric for meta-learning (feature mapping function + differentiated metric design). I think the goal is the same as metric-based meta-learning, which is not new for me. 
 - In addition, the baselines and experiments are not sufficient to support the goal. Especially, the proposed methods (both MCL and RDP) only show comparable performance compared with metric-based methods (e.g., MetaOptNet). It would also be better if the authors can add more metric-based baselines for comparison (e.g., [1],[2]). Moreover, they should also involve metric-based methods in the efficiency comparison section (Appendix C.3).
Thus, I feel the overall contributions are not enough to be accepted. 

2. Besides the Cons 1, if the authors focus on gradient-based meta-learning. Similar to ANIL, more experiments on different types of applications are supposed to conduct. For example, the experiments on regression and reinforcement learning tasks.

3. The paper is not well-written. Here are some comments:
 - It would be more clear to formally formulate the MCL and RDL with more notations and equations.
 - Most figures are not clear. It is better to replot them with larger font size in legend, thicker lines, etc.

[1] Ye et al., Few-Shot Learning via Embedding Adaptation with Set-to-Set Functions. CVPR 2020.

[2] Wang, Yan, et al. "Simpleshot: Revisiting nearest-neighbor classification for few-shot learning." arXiv preprint arXiv:1911.04623 (2019).
review_title: Official Blind Review #1

final_score: rating: 5: Marginally below acceptance threshold
review_content: **Summary:** 
* This paper conducts analysis of the task-specific adaptation in the MAML algorithm, building on recent work analysing the effectiveness of MAML. The paper explores different variants of MAML, and provides empirical analysis to argue that the multi-step task-specific adaptation of a network head in the MAML algorithm is important in learning good representations and thus enabling effective few-shot learning performance.  
* The paper then proposes two algorithms: one that removes gradient-based updates for a network head (instead using random matrices for classification) and another one that uses contrastive learning (no inner loop adaptation). These are evaluated on benchmark datasets, and shown to have improved computational speed.


**Overall Comments:** This paper consists of some analysis on task-specific adaptation, and presentation of two related few-shot learning methods that build on some of the insights from the analysis. As specified in the detailed comments, I am unconvinced about some of the experimental evaluation in the analysis section, and I am not sure the comparisons are fair. 
Relatedly, one of the two algorithms presented is also unconvincing to me (with the presented evaluation, it doesn't appear to strengthen the claim made by the analysis). 
In contrast, the second algorithm presented is clear and interesting, with notable computational benefits and performance improvements over baselines. 
A stronger version of this paper could either focus on the analysis, making the claims clearer and strengthening the evaluation (see thoughts below), or present a shortened version of the analysis and then focus more on the MCL algorithm, stressing its performance improvements and computational benefits. In its current form, due to a weaker analysis section and limited analysis of MCL, I do not think this paper is strong enough for acceptance.


**Detailed Comments:** 
* The analysis about the importance of multi-step adaptation is a bit unclear at points, and the experimental evaluation may not support the claim that the multi-step task specific adaptation is important. If the claim is the multi-step adaptation in MAML is specifically the important aspect, then a fairer comparison would be MAML/ANIL/BOHI with 1 inner adaptation step for the head vs MAML/ANIL/BOHI with 5 inner adaptation steps for the head, and comparing how these affect feature learning.  
* As explored in Raghu et al (2019), there is an alignment problem that could make the multi task setting perform poorly, so this result is not as surprising.
* One aspect of the multi head setting is a bit unclear -- for every task, do we initialise a new 5-way head and then perform only 1 gradient step to update these parameters? It seems that a fairer comparison would be to perform more than 1 update step. From a randomly initialised head, I would expect > 1 step to be important to help classification performance. Again though, if the claim is specifically about multi-step adaptation being important for feature learning in MAML, then the 1 inner step vs 5 inner steps on MAML/ANIL/BOHI is a more relevant comparison, as the multi head situation is fundamentally different. 
* Section 4.2 is a more convincing argument (both the mathematical statement and the plots) of the important of task specific adaptation of the head. I think this analysis as a whole could be summarised as how good gradient signal for the network body is only obtained when the head of the network is (at least somewhat) effective at classification, and a bad head will lead to poorer learning signal. I also want to note that in Fig 2, the improvements in the body's representations is quite small overall (a delta of < 0.3 percent). Minor comment: it would be good to specify in the text/figure caption some more experimental details about Figs 1/2 (dataset, N way K shot, etc).
* Section 4.3 appears to extend the statement made in Section 6 of Raghu et al (2019) to newer methods, but is still an interesting observation. Minor comment: The method in Lee et al (2019b) is MetaOptNet, not MetaOpNet.
* The RDP algorithm is an interesting investigation, but I am not convinced of its utility or that it supports the overall argument of the paper. Even though FC100 performance is improved, the performance on Mini/TieredImageNet is appreciably worse than MAML, especially for 1 shot settings. Further evaluation would help this claim.
* The MCL algorithm section was interesting and I was impressed by the results, and the computational speedups that this method allows for. I think that this method itself is a good contribution. Table 8 in the appendix is in my opinion interesting enough that it would be good to put it in the main text. It is a nice idea to apply the insights from contrastive learning for good representations to FSL, especially as prior work has demonstrated good representation learning to be very important for good FSL performance.

review_title: Official Review #3 

final_score: rating: 4: Ok but not good enough - rejection
review_content: Strengths:

1.	Based on previous work (Raghu et al. 2019), this paper provides some analysis to explain why multi-step task-specific adaptation is important. 

2.	The idea of introducing contrastive learning is reasonable and leads to performance improvement in experiments.

Weaknesses:

1.	The proposed methods have little improvements over base models; Random Decision Planes (RDP) actually obtains worse results than the basic MAML. Although it is reasonable for the proposed method to use contrastive learning, its novelty is quite limited. Some questions are listed below, which would be great if they can be answered.

2.	Analyses in some section are of heuristic nature, and thus not very convincing (please see the detailed comments below).

3.	The presentation of the paper needs some improvement. Since the proposed algorithms are mainly based on the idea of learning better feature extraction, the analysis on why multi-step adaptation is important, thus, contributes little to the understanding of the proposed methods, which can be directly motivated by results in Raghu et al. 2019.

Detailed Comments & Questions:

1.	Would it be possible to provide some more details on the implementation of ‘Multi-Head’ in section 4.1? How to make the head parameters task-specific to randomly sampled tasks? Are the heads re-initialized at every iteration? Or simply use n heads for a batch of n tasks, and then apply the updated n heads to the next batch of n tasks? I think this might be important.

2.	I do agree that low-accuracy head may provide bad signals for the model body. But the heuristic nature of the analysis is not convincing enough. Firstly, the body is updated based on the head’s prediction accuracy.  Obviously, there will be a significant gap between the body accuracy and the head accuracy. Secondly, the parameters in the body might not change “efficiently” as the head parameters, because the gradient with respect to the body parameters (especially the low-level features) might be small. It will be helpful if we can know the averaged norm of elements changes (or gradients) in the body and the head, respectively, or the averaged spectral norm of the matrix changes (or gradients) of the body and the head, respectively. Also, what if different step sizes are used in the body and the head? Overall, question about the importance of multi-step adaptation is still not well-answered.

3.	In Table 2, results of RDP are shown, which are worse than  the basic MAML. If the analysis in previous sections are correct, why do not simply use ANIL or BOHI, which totally avoid the influence of the low-accuracy heads on the model body and  the results after adaptation, and obtain much better experimental results? What is the benefit of using RDP?
 
4.	Is it possible to provide more details on meta-testing for MCL? Is the classification based on RDP or using a new fully connected layer (as head)? Will the body also be updated during meta-testing? What will the results be if using cosine similarity as in Raghu et al. 2019, which might be better to illustrate the quality of features.

5.	Is it possible to visualize the features using algorithms like t-SNE? This will allow us to see how the contrastive learning leads to features that can better distinguish/discriminate samples from different classes.

6.	I agree that contrastive learning could benefit feature extraction, and consequently helps the final results after adaptation. It might be interesting to see whether multiple datasets can be used for pre-training, e.g. train the model by contrastive loss using FC100, CIFARFS, and ImageNet 32\times 32, and then test on only one of them.

review_title: This paper analyzes the impacts of task-specific head and model body in meta-learning. New algorithms are proposed based on the analysis. Some improvements are obtained using the proposed methods.

final_score: rating: 5: Marginally below acceptance threshold
review_content: This paper adds to a series of papers that look at understanding how and why MAML works as well as it does, and what exactly is going on in the adaptation process. There are many open questions in this regard, and so I think the topic of the paper is interesting and of interest to the community. However I found the paper somewhat difficult to read and I'm not entirely sure what to take away from the paper. It was often not clear to me why certain things were introduced or why certain experiments were conducted. Overall I think that the contribution is not strong enough for publication at ICLR.

I'll structure my questions according to the contributions listed in the introduction, and the corresponding main sections in the paper:

Contribution 1, Section 4:
- If I understand this point correctly, then you're arguing that the task-specific adaptation in the inner loop is *necessary* such that the outer loop can learn a good feature extractor, is that correct? But doesn't the fact that you can do MLP contradict this statement, because here you've basically removed the task-specific adaptation.
- Section 4.1.: You compare four methods here, to show that multi-step task adaptation is important. 
 - What do you mean with "Muti-step" adaptation? Are you focusing on the fact that you need multiple *gradient* updates in the inner loop and that one is not enough? 
 - (Model 1) How is the multi-task model trained? You say that all tasks use the same head, so how do you add the task-specific information to the classifier? How does this work in the few-shot learning setting? How do you adapt the model at test time? 
 - (Model 2) How is the multi-head model trained? I can see how you can do this for a small, fixed number of training tasks, where you just learn one head per task over the entire course of meta-training. But the what happens at test time? You say there is no inner loop, but the test accuracy is larger than random. So do you re-initialise a random new head and perform gradient updates on that?
 - (Model 3) Add a citation here to make clear that ANIL is from Raghul et al. 2019
 - (Model 4) Is this your own method, or is it taken from prior work? If I understand correctly a similar model is investigated in concurrent work: https://arxiv.org/abs/2008.08882 (which is not a problem; but you might be interested in what they find and it would be good to just add a brief mention of this concurrent work somewhere in the paper).
- Section 4.2: In this section you ask *why* multi-step adaptation is important.
 - In Figure 1 and 2, how come the accuracy before adaptation is significantly larger than random?
- What do you mean with "We present _sufficient_ experiments"? Sufficient in what sense? 

Contribution 2, Section 5:
- It's kind of cool that the suggested algorithm of using (random) pre-defined classifiers (last layer of the network) instead of adapting the last layer using gradient descent in the inner loop works well on few-shot image classification (on two datasets worse, on one dataset better than MAML). I think this illustrates a similar point as ANIL (Raghu et al. 2019), namely that only adapting the last layer is sufficient on some problems.
-  In Figure 3 you show the number of "decision planes" in the set of classifiers that you use. I don't understand how using 1 decision plane can give such high (~30-55%) accuracy, can you explain this? If I have only one decision plane to choose from in the inner loop, then I always have to choose the same one, so adaptation isn't really possible. Shouldn't I just get random performance, so 20% on the 5-way problem then?
- I think it would help to explain in this section why you are proposing this algorithm. Is it to drive home an observation about MAML? If so what does this example do that wasn't shown in ANIL (Raghu et al. 2019)? And/or is it to propose a new method that people should use - in which case it would be good to argue why. Current meta-learning methods perform much better than what RDP can achieve and evaluating the accuracy on all available heads might be a computational overhead. But maybe there are situations where we'd rather do that than gradient updates? 
- Do you have any intuition as to why RDP works better on the FC100 dataset compared to MAML, but not on the other datasets? Does that dataset have different properties? I think the reader could benefit from your insights here.
- Why does the accuracy drop slightly when 512 decision planes are used compared to using 128?

Contribution 3, Section 6:
- Do I understand correctly that this method basically does something like NIL (Raghu et al. 2019) but then explicitly trains the features with a contrastive loss such that features from different classes get pushed further apart? If that is the case, shouldn't you compare to NIL here? 
- There are a few other methods that use features to compare classes, or contrastive losses, and I'm surprised you don't compare to these. E.g. https://arxiv.org/1807.02872 https://arxiv.org/2008.09942 https://arxiv.org/abs/1606.04080
- You write that "out method outperforms almost previous [sic] well-designed methods and also achieves results comparable to MetaOpNet". But current methods SOTA that aren't listed in your comparison get much higher (+20 percentage points) accuracies (https://paperswithcode.com/sota/few-shot-image-classification-on-mini-1). Could you explain why you chose to compare to the methods in Table 3?
- I think I have the same question as my last point above for [2]. It's not clear to me what the purpose of the proposed method is - illustrating a point or proposing a useful new algorithm?

---------------------------------------------------
---------------------------------------------------
UPDATE:

I have read the other reviews and the author's response. 

Thank you for the thorough answers - this cleared a lot of things up and I understand much better how the multi-head, multi-task and random decision planes work. I'll increase my score to a 5 because I've gotten more insight now with the additional information, and think that the paper raises some interesting points. Overall, I still tend towards rejection - even with the updated version, I still find the contributions of the paper difficult to tease out and evaluate, and not all claims in the paper are sufficiently backed up / analysed by experiments. I would encourage the authors to try and centre the entire paper more clearly around *one single* central message in the future, and present all experiments in this light, making sure that every claim is sufficiently backed up empirically. 
review_title: review

Meta-review: MAML is a well-known gradient-based bi-level optimization to learn a good initialization over a set of relevant tasks. This paper investigate different variants of MAML, providing empirical analysis of two new algorithms (RDP and MCL). Reviewers agree that it is interesting to see what the change of optimization mechanism on both head and body brings to us in the MAML framework. This is done by only empirical analysis. However, all reviewers have concerns that the current version (or even revised one after the author responses) does not contain substantial contributions over existing work in the sense that: (1) experiments do not support well what's been claimed; (2) writing should be much improved to clearly explain the formulation of RDP and MCL, as well as figures in experiments; (3) the analysis about the importance of multi-step adaptation is not clear (Section 4); (4) the proposed method has little improvements over baseline methods.  Without any positive feedback from reviewers, I do not have choice but to suggest rejection. 




Decision: Reject