Venue: ICLR 2020 Conference

Reviews:

final_score: rating: 8: Accept
review_content: After  rebuttal period: I recommend accepting  this  paper.
======================================
Summary:

This paper attempts to understand if the success of MAML is due to rapid learning or feature reuse. The analysis shows that MAML is performing better mainly due to feature reuse. Authors use this result to derive a simpler version of MAML called ANIL. ANIL does not update the non-final layers of the network during inner loop training and still has similar performance to MAML.

My comments:

Overall I think this is an interesting analysis paper which sheds some light on how MAML works, However, I see these analysis not just as a criticism towards MAML. I also see these analysis as a criticism against the meta-learning datasets that we use. All these datasets are artifically created from the same dataset and hence it might be very easy to reuse features to get good performance. I am not sure if the same analysis will hold if we consider a dataset where tasks are not this similar (like Meta-dataset, Triantafillou et al 2019). I encourage the authors to have this disclaimer in the end of the paper so that the community does not falsely conclude that MAML cannot do rapid learning.
review_title: Official Blind Review #4

final_score: rating: 3: Weak Reject
review_content: This paper is exploring the importance of the inner loop in MAML. It shows that using the inner loop only for the classifier head (ANIL) results are comparable to MAML. It also shows that using no inner loop at all (NIL) is okay for test time but not for training time.

It is indeed interesting to understand the effect of the inner loop. But, as the authors noted (“Our work is complementary to methods extending MAML, and our simplification and insights could be applied to such extensions also”), for it to be useful I’d like to see whether these insights can be extended to SOTA models. MAML is less than 50% accuracy on 1-shot mini-imagenet while current SOTS models achieve 60-65%.

The NIL experiment that shows low performance when no inner loop is used in training time doesn’t make sense. This is basically the same as the nearest-neighbour family of methods, e.g. ProtoNet (Snell et al., 2017), which have been shown to perform similarly to (or even better than) MAML.


After rebuttal:
I do think it's important to also have that kind of analysis works. My main concern is with how ANIL and NIL are introduced as new algorithms and not just an ablation of the MAML method. Presented as new algorithms I tend to compare them against the leaderboard where they are very far from the top. I am keeping my previous rating.  
review_title: Official Blind Review #1

final_score: rating: 8: Accept
review_content: The paper claims to examine the reasons for the success of MAML---an influential meta-learning algorithm to tackle few-shot learning. It thoroughly investigated the importance of the two optimization loops, and found that feature reuse is the dominant factor for MAML’s success. Moreover, the authors proposed new algorithms---ANIL and NIL---which spend much less computation on the inner loop of MAML. They also discussed their findings in a broader meta-learning context. 

I think the paper should be accepted for the following reasons: 

1. The experimental study is thorough. 

The experiments follow a rigorous design of hypothesis-checking style and the conclusions are supported by extensive results under various evaluations. 

The findings are potentially helpful for many future works in this field. 

2. The paper is clearly written. 

It is generally enjoyable to read, except for some minor things to improve: (1) Evaluation metrics in table-2, table-4 and table-5 had better be explicitly clarified in the captions (2) No subsection seems needed in section-6. 

review_title: Official Blind Review #2

Meta-review: Paper received mixed reviews: WR (R1), A (R2 and R3). AC has read reviews/rebuttal and examined paper. AC agrees that R1's concerns are misplaced and feels the paper should be accepted. 

Decision: Accept (Poster)