Venue: ICLR 2022 Conference

Reviews:

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: This paper analyses the well-known and well-studied MAML algorithm and raises two key observations. First one is requiring a high number of inner-loop updates, and the second one is the variation in the meta-test accuracy when permuting the indices of the classes. Then it proposes to simply meta-train a single vector and duplicate it as initialization for the classifier head to make MAML permutation-invariant and improve its generalization performance on meta-test tasks. 
main_review: This is an interesting study which is mainly inspired by the empirical results. However, I have some concerns that needs to be addressed: 

 
 

- In equation (1), the gradient is taken with respect to the $\theta^{‘}$ as we are in the inner-loop? 

 
 

- One main concern of this study is that when MAML pairs the learned classifier head with different permutations of a task, there could be an inconsistency in the performance. However, I believe this limitation does not exist in works like ProtoNet, as we have no $\omega$ there for pairing, and the prototype of each class does not depend on the class index but support set samples of that class. One major question is that, since ProtoNet has implicitly addressed this concern before, what is the *importance* of this study? 

 
 

- Considering good pairing between meta-test task and classifier head is not a new idea in few-shot learning. For example, in Dhillon et. al. [1], they implicitly consider this by initializing the classifier’s head using the normalized logits for each class, inspired by cosine distance. However, this work uses the simple idea of repeating a meta-learned vector as the classifier's weights. I think you need to add this work and similar works to related work, and emphasize on the differences. 

 
 

- My *major concern* is about your motivation for this study. Let’s have a quick look at the possibility of the permutation you discussed as motivation and illustrated in Figure 1.  

Considering the meta-test set of miniImageNet, there are 20 classes on this set. Then for 5-way classification as we need to sample 5 classes to construct a task (or episode), the number of permutations of 20 classes taken 5 at a time is 20!/(20-5)! = 1860480. Among these, 5!=120 could be the task that has the same classes. So, you are planning for an event with a chance of 120/1860480=0.000064 which is really negligible. This gets even worse for other datasets like tieredImageNet and CUB as they have larger numbers of classes in meta-test, or even for real scenarios with larger unseen classes. 

However, looking at your results, I think your solution is more general  and may be also considered as differences in the class attributes. So, you need to change your motivating example and make it more general. Although your solution answers, I think you have approached it with the wrong motivation. 

 
 

- The procedure to produce the results in figure 3 is not clear to me. Do you use meta-validation performance to select M? Or do you just test different values of M for training and testing? Please elaborate on this. 

 
 

- In section 5, when you propose simple solutions to make MAML permutation-invariant during meta-testing, what is the point of searching for the best permutation for a meta-training task? How is it going to affect the training of more powerful features and improving generalization? 

 
 

- Based on the results in Table 4, UNICORN-MAML (which has a lower number of parameters w.r.t MAML) improves its performance. However, I am concerned when authors mention that they have compared with state-of-the-art, because they have not included recent works in the field, like [2], which shows produced results fall behind current state-of-the-art. I think it is more reasonable to mention that UNICORN-MAML can be considered as a strong baseline for future works, and I am ok with this. 

 
 

- In section 6, I didn’t find the answer for the question “Why does UNICORN-MAML work?” informative enough. You may give more detail which backed up with results, or some facts from previous works, or simply omit this. 

 
 

- The explanation of Figure 5 is not relevant to the main purpose of the paper.  As one of key observations in your work, you have mentioned that MAML needs lots of inner-loop updates in Figure 3. However, when providing similar results (as you mentioned in the main text) for UNICORN-MAML, you conclude that it aligns with previous work that “we need to also adapt the feature extractor”? How do you relate this finding to your intention in this work?  

It needs to be related to the main context. Please elaborate on this. 

 
 

- Table 6 does not provide a fair comparison with current algorithms for cross-domain few-shot classification. As mentioned in [3], in the same scenario (miniImageNet $\rightarrow$ CUB), a metric-based algorithm like RelationNet, achieves 57.77% in 1-shot. Using FT proposed in [3], it can also achieve up to 59.94%. However, you have reported 51.80% as SOTA result in this case? 

 
 

References: 

[1] A Baseline for Few-Shot Image Classification, ICLR 2020. 

[2] MELR: Meta-Learning via Modeling Episode-Level Relationships for Few-shot Learning, ICLR 2021. 

[3] Cross-Domain Few-shot Classification via Learned Features-wise Transformation, ICLR 2020. 

 
 
summary_of_the_review: This work proposes a simple yet interesting algorithm which meta-trains just a single vector and duplicates it as initialization for the all classification head weights for various classes in MAML. It improves the performance of the MAML, however I have some serious concerns regarding the importance, the validity of the motivation and the fairness of the results for this study.
review_title: null

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: The authors study the MAML algorithm and propose two ways to improve the performance. First, they study the number of inner gradient steps. Second, they look at the permutation of labels when learning a new task which ideally should not make any difference. However, it seems that during testing this could lead to different test performances on a target task. Based on this observation, they propose a novel and very interesting solution to share the weights of the classification layer (which they call UNICORN MAML).
main_review: 
I really liked the way that the authors study the MAML algorithm. They performed very systematic and reasonable experiments. The second part of the paper that looks at the permutation of the labels is interesting. Ideally, this should not have happened since the task is the same and labels should not impact the performance. For example, for metric-based approaches such as ProtoNets[2], this never happens. Also, the proposed approach (UNICORN-MAML) just defines a weight vector that is shared for all classes such that permutation has no impact on the updates. It is a very interesting way to solve the problem.








[1] Hsu K, Levine S, Finn C. Unsupervised Learning via Meta-Learning. In Int'l Conf. on Learning Representations 2018.

[2] Snell J, Swersky K, Zemel R. Prototypical networks for few-shot learning. In Proc. of the 31st Int'l Conf. on Neural Information Processing Systems 2017.

summary_of_the_review: Finally, I think this is a thorough study of a very useful algorithm and can improve many papers that are based on this approach. It is a good improvement on MAML that is based on a very detailed systematic study. As a result, I vote for acceptance.
review_title: null

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: This paper is an interesting "re"search on a classical meta-learning method MAML. It shows that both the inner update iterations and task label assignment have a clear influence on the performance of MAML. It raises some questions and uses a set of in-depth experiments to draw conclusions. 
main_review: Advantages

1. The paper is interesting for me. I believe it is worth questioning the key settings in the benchmark or classical baseline methods. This encourages people to rethink why the model works or not in specific scenarios. Also, the idea of UNICORN-MAML is simple and easy to implement.

2. This paper offers a detailed exploration for two factors (in MAML based meta-learning benchmark) that are ignored by previous studies: one is the number of inner-updates in each training/testing task and the other is the task label assignment in each task. It is interesting to see the statistics in Fig4 that permutations make such a big difference to specific test tasks.

Disadvantages

1. From the results in Table 2,5,6, it is pity that there is little concrete performance boost by using the new MAML (UNICORN-MAML) compared to MAML (ours) --- where I assume ours means "our careful implementation of MAML".

2. Main conclusions or results are from arbitrarily designed experiments on existing benchmarks of few-shot learning---small tasks of object classification. It is hard to guarantee that the conclusion maintains in meta-learning larger-scale tasks or in another new benchmark. 

3. MAML was implemented originally not only on few-shot classification tasks but also on reinforcement learning tasks. While it is clear that this new UNICORN-MAML may not be straightforwardly used for RL, as it studies the specific problems in the image classification settings.
summary_of_the_review: I think it is interesting to research and rethink the possible issues in the existing baselines or benchmarks. I have a few concerns about the significance of this current version of the paper because of the poor performance or the arbitrary design of experiments. I would like to see the comments of other reviewers and the discussion with the authors to make a final decision.
review_title: null

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: The paper mainly investigates the effect of permutation in the class label assignment in the tasks for the MAML algorithm. First, the authors show that MAML requires a higher number of inner loop updates than what is commonly used. Then, they show that MAML is sensible to the permutation of the class labels in the tasks and experimented with diverse methods to alleviate this problem. Finally, they proposed Unicorn-MAML, a modification of the MAML algorithm that learns a single weight vector for the classifier layer, to make the model permutation invariant. They show through experiments on two different datasets (mini-ImageNet and tiered-ImageNet) that it achieves or outperforms state-of-the-art performance.
main_review: ### Strengths

- The paper is well written and easy to follow. The main problem at hand, the random label assignment during evaluation, is introduced through different experimentations, showing why and where is the problem in MAML.
- A good explanation and novel investigation of this problem. The results show that MAML can perform on-par with and even outperforms other state-of-the-art meta-learning methods.
- The experiments are detailed and multiple methods are tested to try to alleviate the permutation label assignment problem for MAML, before introducing their proposed modification to the algorithm.
- The experimental settings and hyperparameters are all mentioned and documented to help reproducibility.
- Even though the experiments focuses on the MAML algorithm, the authors compare their results with related works, but also discuss how other methods deal with the random label assignment.

### Weaknesses

- I am quite surprised by the performance they achieve with their vanilla MAML, even without considering the increased inner steps. I never saw performance as high in other papers using MAML with a ResNet12. Maybe the difference come from the pretrained weights ? I didn't find any information on how the backbone was pretrained though and I think this information would help for reproducibility.
summary_of_the_review: I found the paper clear and easy to read. All the experiments are detailed as well as the thought process behind them. I would appreciate more information on the full training process to improve reproducibility.
review_title: null

Meta-review: Three of four reviewers rated this paper as an 8. 
These positive reviewers felt that this paper provided a lot of value through extensive experimentation with MAML in the few-shot setting. It was felt that the detailed analysis of the inner and outer loop of MAML provided a lot of understanding to the reader regarding the behaviour of MAML. The fourth reviewer giving a score of 3 remains concerned about high variance on some experiments. However the strength of ratings from the other reviewers make the AC more than comfortable giving an accept recommendation for this work.
Decision: Accept (Poster)