Venue: NeurIPS 2022 Conference

Reviews:

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: This paper studies $\mathcal{S}$-$\mathcal{T}$ protocol in meta-learning, which means comparing a task-specific solver to a target model. The main contribution of this paper is proposing an efficient method to construct target models. The authors generate target models by adapting from the momentum network, which shows better adaptation performance than the meta-model itself. In the proposed method, the construction of target models and the training of meta-model are combined and iteratively performed. Experiment results demonstrate the effectiveness of the proposed method.
strengths_and_weaknesses: Strengths
1. This paper studies an important and interesting problem in meta-learning, i.e., $\mathcal{S}$-$\mathcal{T}$ protocol, which may bypass the limitations of query sets.
2. Experiments cover both few-shot learing and reinforcement learning, which are two main applications of meta-learning.
3. This paper is well written and easy to understand.

Weaknesses
1. **Imperfect target models**. In Line 7 of Algorithm 1, we can see that the authors generate a target model according to current momentum network $\phi_{\mathrm{moment}}$. However, the momentum network is updated during the meta-training procedure, and it is weak in early epochs. The definition of target model is a strong model (expert) on a specific task, but the generated target models by the proposed algorithm are not optimal.
2. **Insufficient interpretation**. Some technical details are not discussed adequately in the paper. For example, in Equation (3), the second terms stands for the loss of $\mathcal{S}$-$\mathcal{T}$ protocol. However, the authors evaluate the task-specific solver on the query set $\mathcal{Q}^{\tau_{i}}$ rather than the support set $\mathcal{S}^{\tau_{i}}$, which is different from [1]. In Line 146, the authors claim that the predictions of target models tend to generate soft predictions, which is not verified empirically or theoretically. Apart from this, the quality of generated target models are not evaluated. Since target models are experts on corresponding tasks, it is necessary to evaluate them.

References

[1] Lu S, Ye H J, Gan L, et al. Towards Enabling Meta-Learning from Target Models[J]. Advances in Neural Information Processing Systems, 2021, 34: 8060-8071.
questions: Questions
1. The target models are generated according to the momentum network. Could authors evaluate the quality of target models? Do these target models perform well on their tasks? 
2. Since the momentum network is updated during the meta-training process, early target models tend to be weak. Could authors give some explanations?
3. In Equation (3), the authors evaluate the task-specific solver on $\mathcal{Q}^{\tau_{i}}$, which is different from [1]. Could authors show the advantages of doing so?
4. In [1], the target models are constructed only for those hardest tasks (5% or 10%). It is interesting to see what will happen if only a part of meta-training tasks have target models. Do the number of target tasks have severe evident influences on the performance?
5. The test accuracies in Table 2 are not satisfying. For example, with ResNet-12 backbone, we often achieve 65% (or higher) accuray on 5-way 1-shot miniImageNet using ProtoNet [2,3]. Is it because the authors do not pre-train the backbone on the meta-training split?

References

[1] Lu S, Ye H J, Gan L, et al. Towards Enabling Meta-Learning from Target Models[J]. Advances in Neural Information Processing Systems, 2021, 34: 8060-8071.

[2] Ye H J, Hu H, Zhan D C, et al. Few-shot learning via embedding adaptation with set-to-set functions[C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2020: 8808-8817.

[3] Wang Y, Chao W L, Weinberger K Q, et al. Simpleshot: Revisiting nearest-neighbor classification for few-shot learning[J]. arXiv preprint arXiv:1911.04623, 2019.
limitations: The authors have included the limitations and future work in the paper, and some other limitations are listed in the *Strengths and Weaknesses* part.
review_title: null

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: This paper proposes a simple algorithm to use the benefits of the knowledge distillation (KD) in meta-learning. More specifically, since it is not possible to train an individual teacher for each meta-training tasks (which could be hundreds of thousands tasks in standard setup), this method obtains a momentum network using exponential moving average of the meta-model and then adapts this momentum network into each task using support set of that task. The adapted model is used as teacher to guide the main few-shot adaptation process.

strengths_and_weaknesses: Strengths: 


This work introduces a very interesting idea to improve the generalization of the meta-learning algorithms. From KD point of view, it is similar to Born Again Neural Networks where a model becomes its own teacher. However, instead of training from scratch for several times, it uses momentum network to train the teacher (obtained by adaptation from this momentum network and called target network). Even though the algorithm is very simple, it works very well and provided results show improvement on baseline for different backbones and in different applications including classification, reinforcement learning and regression. 

The writing is easy to follow and concepts are defined well, however, there are some concerns regarding this work.

Weaknesses:


Proposed SiMT algorithm can be considered as a regularizer which prevents the baseline meta-learning algorithm from overfitting to meta-train classes. Since the results proposed in this paper are way behind state-of-the-art (SOTA) performance, I expect to see at least the comparison with other regularizer-like methods for few-shot learning like:
- Cross-Domain Few-Shot Classification via Learned Feature-Wise Transformation, ICLR’20 


In addition, as the method can be considered in the line of works that uses KD for meta-learning, comparison with previous works needs to be included:
- Towards enabling meta-learning from target models, NeurIPS’21


For few-shot classification results, as we cannot generally evaluate the whole possible set of meta-test tasks, the results usually presented as the mean and standard deviation over a batch of tasks. However, this paper just reports the std on three trials which is not accurate enough.


For RL results in Figure 2, why the std of the proposed model is much higher than the baseline? As SiMT uses momentum network, I expect to see less std in the proposed method.


For computational efficiency, the analysis is very limited and is presented just for a specific configuration.


Regarding Table 4 (Ablation studies), what does it mean to use the momentum without distillation? Do you mean that the meta-learner is updated using momentum equation?


In Fig. 3(b) Configuration is not clear. If same config, why is this different from the 3(a)?



There are also some typos in the paper:

- In line 118, it should be \mathcal{L}(\phi^{\matchal{T}_i})

- same problem with line 126, and equation (6)

- in Fig 3(a) is should be 3X faster


questions: Please refer to the weaknesses part.

limitations: yes
review_title: null

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: An algorithm based on knowledge distillation together with a momentum network working as experts on the unseen fields to solve meta-learning problems. The proposed algorithm shows compatibility with multiple existing meta-models and improvement. The general idea is simple and easy to implement.  
strengths_and_weaknesses: Strengths: 
1. The experiment shows the good performance of the model on a variety of tasks. 
2. The model is analysed from different perspectives, loss landscape, and computational efficiency. 

Weakness: 
1. From my understanding, despite all the complicated presentation about all the algorithm components, the algorithm only forces the meta-model to get close to its momentum model which works as a regulariser. From this perspective, the novelty is quite limited. 
2. The proposed algorithm shows the improvement and compatibility on multiple MAML models but lacks comparison with other SOTA meta-models. 

questions: 1. It seems like the main idea of the proposed method is to stitch a momentum network and knowledge distillation. What is the motivation for doing this? I think one thing could be good to show is that how the momentum network works on each task after the adaptation, since the knowledge is distilled from it.
2. The adapted moving average parameter set, momentum, is adapted to the query set, to be an expert. Is it similar to say that the whole idea is to regularise the meta-model to its momentum model? 
3. Why the perturb (dropout) is important? Without dropout how does the model perform differently? 
limitations: The paper is presented in an unfriendly way for the readers who do not have an efficient background. In the beginning, especially in the introduction section, a lot of concepts, target model, original model and task-specific solvers, are introduced in the cause levels making the paper very confusing. Some components of the whole algorithm are explained very independently. It is not very clear why one relies on the other.
review_title: null

final_score: rating: 7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
review_content: summary: The paper introduces a new method for few-shot learning that utilises a self-improving momentum target. The idea is to learn from a meta-model that is more quickly adapted than the standard meta-model - the momentum target is updated using exponential moving average rather than SGD. The approach can be combined with a wide variety of few-shot learners and consistently leads to strong improvements in performance.
strengths_and_weaknesses: Strengths:
* The approach can be combined with various few-shot learning methods and consistently leads to strong improvements in performance.
* The idea itself is intuitive and novel, even if the idea has some similarities to bootstrapped meta-learning and momentum networks
* The paper provides excellent explanations and illustrations to show how the method works
* The empirical evaluation is extensive and covers few-shot regression, few-shot classification (both in domain and across domains) and also reinforcement learning - with strong results across all
* There is an ablation study evaluating the various components showing that all parts are important

Weaknesses:
* There are quite many additional components in the approach
* The approach introduces several additional hyperparameters for which we need to select values (but perhaps selecting them is not too difficult)

I’ve enjoyed reading the paper and the strengths significantly outweigh the weaknesses, so I recommend acceptance (the weaknesses are rather minor). The paper will likely be interesting and valuable for the ML community, especially for the researchers focused on meta-learning.

questions: * Based on Figure 3b, it seems that even just using momentum network instead of the meta-model leads to strong performance. Is it the case that if we simply replace the outer loop SGD update with exponential moving average in MAML it would significantly improve its performance? Is it something which any other few-shot learners already do?
* How easy is it to select the additional hyperparameters for SiMT?

limitations: Yes, the discussion of limitations and societal impacts is well-written.
review_title: null

Meta-review: metareview: This submission proposes a strategy to improve meta-learning that can be applied to many different base meta-learning methods. The base meta-learning method is used to independently adapt both the online network whose parameters are being optimized and a momentum network constructed by taking an exponential moving average of the online network's weights, and a distillation loss is used to encourage the adapted online network (with dropout on its parameters) to match the adapted momentum network. Extensive experiments demonstrate that the proposed method improves performance of several base meta-learning methods, and that each component of the method is necessary to attain optimal performance. Reviewers initially praised the idea and empirical evaluation, but noted that the results obtained were far from state-of-the-art, and asked for additional experiments with better-performing base meta-learning methods. The authors provided these experiments during the response period, and the reviewers now unanimously recommend acceptance. The AC agrees with the reviewers' assessment.
confidence: Certain
Decision: Accept