Venue: NeurIPS 2021 Conference

Reviews:

final_score: rating: 7: Good paper, accept
review_content: summary: The authors propose a novel algorithm for meta-learning, in which instead of just optimizing for the query loss, they also optimize the inner loop produced model to do as well as some larger ‘teacher’ model that is an actual expert on the task at hand. The data used to train those experts are the same data we normally use during the meta-training phase in a particular benchmark, therefore their process does not require more data, but its efficiency can either be better or worse than a model trained with the standard S/Q loss depending on how many ‘expert’ models are pretrained. 

main_review: Abstract: 

Overall a very good abstract. However, you never defined the term ‘Target Model’ and it is one of the key terms necessary to understand what is going on. I’d recommend an explicit definition somewhere in the abstract, to ensure the reader understands.

Writing Quality:

Overall the paper reads very well, with the occasional typo, but overall it is precise, concise and very clear. 

A few points I’d like to mention is, firstly, Figure 1 is not readable without zooming, and the text colour matches the surrounding box which makes it even harder to read. I recommend making the figure a bit larger, and the text kept dark.

Table 5 states the word Black in what appears to be bold, therefore confusing the reader, with the actual bold results being the best results. I recommend adapting the colours/formatting here to make it more intuitive.

Conclusion appears to be weirdly formatted for some reason. I assume the figure has some role to play there -- I recommend reformatting this bit to make it cleaner.

Novelty:

Very novel, and surprising work. Using the same data, but by producing such expert models and trying to basically distill their capabilities into the resulting adapted models in each task seems to be very powerful, generalization wise. This could be the beginning of a good meta learning penalty term! :)

I find the results very convincing as to the claims made, and I appreciate that multiple independent runs were done for each data point in the tables. 

Overall quite good, assuming the abstract issue is fixed, I recommend acceptance.

limitations_and_societal_impact: The impact is the same as any other few-shot learning work undetaken in the context of deep learning. Such concerns are better described in full papers in the ethics and deep learning subfield.
review_title: null

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: This paper investigated the S/T meta-learning framework which is not widely used due to the huge computational cost. The proposed method estimates the hardness of tasks and selects the target models of a small ratio of hard tasks. Then, the meta-model is learned with the S/Q meta-loss and the knowledge distillation loss between the selected target models and solvers. Experiments show that the proposed method outperforms the two classic meta-learning methods with the cost of constructing target models. 
main_review: 1.	Generally speaking, this paper is easy to read and understand. In the introduction, it will be clearer to match the solver, target model and meta-model to the corresponding modules or neural networks. Maybe the authors can give brief idea of how to construct the target models in the introduction. The description about S/Q and S/T redundantly appears several times in the paper and can be shorten. 
2.	The equivalence between the S/T meta-loss and the simplified one (Line 179-180) is interesting. Does it mean that adding random noise to training data is the key to meta-learning? Do the two losses lead to the same results? Can the authors empirically verify it?
3.	The study and analysis on the target models are sufficient and solid. 
4.	The experiments on miniImageNet and tieredImageNet show that the proposed method achieves obvious improvements over the two classic meta-learning methods with the cost of constructing target models.
5.	Overall, I think this paper can to some extent advance the meta-learning research.

limitations_and_societal_impact: It would be better to explicitly discuss the limitation in the main pages. Societal impact is N/A.
review_title: null

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: This paper proposes an efficient S/T protocol meta-learning algorithm. Specifically, in order to reduce the number of required target models and the high computational cost, only the target models on those hardest tasks are constructed by fine-tunning the pre-trained network, then the knowledge distillation is used to match the task-specific solvers to target models in the output space. Experiment results verify the effectiveness of the proposed algorithms.

main_review: This paper proposes an efficient S/T protocol meta-learning algorithm. Specifically, in order to reduce the number of required target models and the high computational cost, only the target models on those hardest tasks are constructed by fine-tunning the pre-trained network, then the knowledge distillation is used to match the task-specific solvers to target models in the output space. Experiment results verify the effectiveness of the proposed algorithms.

Strong points:
1.The motivation is clear.

2.An efficient S/T protocol meta learning algorithms is proposed by fine-tuning globally pre-trained model on several hardest tasks.

3.Experiment results illustrate that the classic meta learning algorithms trained via the proposed S/T protocol can achieve SOTA performance.

Weak points: 
1.The main contribution should be emphasized.  In this manuscript, the two contributions are introduced and verified respectively. It would be better to emphasize on the intro section.

2.The proposed algorithm is still under the condition of same task distribution, because the target models is always trained on a sufficiently informative dataset. However, in many real-world scenarios the training distribution differs from the test distribution. How the out of distribution will influence the algorithm performance should be further investigated in several benchmark datasets, such as [1].

3.Meta learning algorithms aim to achieve fast adaptive ability during training phase. Thus, when encountered new tasks, even those tasks have new label space, the learnt model can still achieve better performance with only few data. If the label space of new tasks is different from the training task one, will the proposed approach still effective?

4.The recognition of task hardness depends on the base model \phi^{pt}, which is pre-trained on training task dataset. Will the per-trained model influence the final performance should be further investigated.
5.It would be better to give in-depth analysis on why this algorithm can generalize better.


Overall, the topic is interesting, and experiments verified the proposed approach outperform than other algorithms. However, the contribution is still marginal, and this research lacks theoretical analysis. Authors didn’t provide insight into how the proposed method can achieve better generalizability.

If providing satisfying answers, I will accept the paper. 



[1] Koh, P. W., Sagawa, S., Marklund, H., Xie, S. M., Zhang, M., Balsubramani, A., ... & Liang, P. (2020). Wilds: A benchmark of in-the-wild distribution shifts. arXiv preprint arXiv:2012.07421.
limitations_and_societal_impact: No potential negative societal impact of their work
review_title: null

Meta-review: The reviewers agree that the paper has interesting ideas, and represents an exciting direction in meta learning domain. Hopefully, reviewer comments should help preparing the next draft. 
Decision: Accept (Poster)