Venue: ICLR 2021 Conference

Reviews:

final_score: rating: 7: Good paper, accept
review_content: Summary

One problem of few-shot episodic learning is a poor sampling resulting in negative impacts on the learned model. 
The paper proposes a new episodic training by exploiting inter-episode relationships to deal with poor sampling problem and improve the learned model by enforcing consistency regularization. Cross Episode Attention Module (CEAM) is proposed to alleviate the effect of poorly-sampled shots and Cross-Episode Consistency Regularization (CECR) is proposed to enforce robustness of the classifiers.

Strength

- The paper proposes a novel idea of how to improve few-shot learning by exploiting inter-episode relationships. Using multiple episodes and exploiting inter-episode is a new attempt.
- There have been attempts to improve few-shot training by batch construction, but the proposed method outperforms the previous approaches with a sizable margin.
- The extensive ablative studies provide comprehensive comparisons among possible design choices. (including supplementary materials) 

Weakness

- I could not find a significant weakness of the paper.


Rating

I like the overall idea of using inter-episode relationships for few-shot training. The proposed approach shows strong performance and technically straightforward and easy to understand. Another strength of the method is that no additional hyper-parameter is used to tune the performance. The paper is clear and extensive experiments support the effectiveness of the paper including supplementary materials.

review_title: The contribution of the paper is clear and novel. My initial rating is accept.

final_score: rating: 6: Marginally above acceptance threshold
review_content: ### Summary
This paper proposes a way to exploit relationships across tasks in episodic training with the goal of improving the trained models who might be susceptible to poor sampling in for few-shot learning scenarios. The proposed model consists of two components: a cross-attention transformer (CEAM) which is used to observe details across two episodes, and a regularization term (CECR) which imposes that two different instances of the same task (which have the exact same classes) are consistent in terms of prediction. Cross-attention is computed via a scaled-attention transformer using both support and query set. The consistency loss is a knowledge distillation that imposes an agreement on the two episodes. The soft target is chosen among the two predictions selecting the classifier with the highest accuracy.
### Considerations: 
- I like the idea of exploiting the information across tasks to improve the performance of episodic meta training. This is an interesting direction that should might definitely help disambiguate in the case of poor sampling.
- The ablation study is accurately performed giving the impression of a careful examination of the components of the model proposed.
- I'm not sure the authors can claim sota results: here some of the latest models that perform best on mini-imagenet https://paperswithcode.com/sota/few-shot-image-classification-on-mini-1 I would prefer to restate the contribution as an improvement of x% over the baseline. It is obvious that sota performance requires higher capacity models such as dense-net. I think that other experiments are needed in order to make the claim of achieving sota, otherwise, if the claim is changed, I'm satisfied with the experiments.
- I suggest the authors moving algorithm 1 in the main paper, maybe replacing the verbose description of each step with actual formulas and pseudocode.
- I think there is still room for improvement on the manuscript.  The paper might be a good contribution to the scientific community, but I'll wait for the authors' response on my doubts before my final decision. 

### Questions:
- Q1: Why only considering tasks with the same classes for consistency? Why not considering also partial overlapping of classes across tasks? I guess it is only for simplicity, but it might be beneficial to consider other types of relationships.
- Q2: It is not exactly clear how the meta-test evaluation is performed. I understand that during training you always consider a pair of episodes that are used to transform the features, but how does this translate at meta-test time? Do you always need a pair of episodes? My guess is no, but not using the transformer should change the distribution of the features at test-time and I don't find it trivial to see how this is taken into account. Maybe I'm just misreading the paper. I suggest the authors clarifying this point in the paper.

review_title: Good paper, proper experimental evaluation.

final_score: rating: 6: Marginally above acceptance threshold
review_content: Summary:
- This paper proposes a meta-learning method (MELR) to alleviate the negative impact of poor sampling of support sets.
MLER consists of two main modules. The first module (CEAM) applies the attention mechanism to two sampled episodes and it alleviates the negative impact of badly-sampled instances.
The second module (CECR) enhances the consistency of classifiers obtained by using two episodes to deal with the sensitivity for the badly-sampled instances.
Specifically, to realize this, CECR utilizes a knowledge distillation framework. Experiments with two real-world datasets demonstrate that MELR works well.

Pros:
- This paper proposes a new meta-learning method to alleviate the negative effect of poor sampling of support sets.
- Experimental results show that MELR can improve the baseline method (Propnet).  These results show some evidence of the effectiveness of two proposed modules (CEAM and CECR).

Cons:
- It is not clear why CEAM can alleviate the negative effect of badly-sampled support instances.
- Hyper-parameter candidates used for MELR are not described.

Detailed comments and questions:
- Although empirical results seem to support the effectiveness of CEAM, I do not understand why CEAM alleviates the negative effect of badly-sampled few shots for a given query instance. Can the authors qualitatively explain the reason for this?
- In the last row of fig 4, it seems that CEAM creates obvious outlier instances. Why does this happen?
- How much does the proposed method depend on hyperparameters such as $T$ and $\lambda$?

Minor comments:
- In eqs. (1) and (9), although argmax is taken in the loss function $L$, it is not correct when using the cross-entropy loss.

review_title: Meta-learning method to alleviate the negative impact of poorly-sampled support sets is proposed. Explanation of why the observed improvements could be achieved will improve the paper's quality.

final_score: rating: 7: Good paper, accept
review_content: Summary of Paper:
This paper proposes to improve Prototypical Networks by a method MELR which aims to fix the problem caused by poorly represented classes in sampled episodes and to reap benefits from enforcing cross-episode consistency. The proposed method achieves State of Art results on commonly used benchmarks miniImageNet and tieredImageNet. 

Reasons for score:
Overall, I tend towards rejecting this paper. I think cross-episode relationship is an important topic of study in the context of few-shot learning, and the experimental results in this paper are strong. However, I am unconvinced that this ‘hammer’ is the right tool for the ‘nail’ that the authors claim to solve. A more thorough study of the motivating problem and how & why MELR works would greatly improve this paper.

Pros:
1.The proposed method performs well in standard benchmark datasets miniImageNet, tieredImageNet, and CUB200. Assuming correctness of experimental protocol, the improvement over previous methods is significant.
2. Visualization of embedding space by t-SNE lends further credibility to the strong performance. Samples from each class are well clustered yet still disperse.
3. Ablation of hyperparameters and algorithmic alternatives is mostly complete and honestly presented.

Cons:
1. The paper is motivated by the supposed “poorly sampled episodes” problem. While it intuitively makes sense that some data points are more representative than others, whether this has a disparate impact on episode few-shot learning is unclear. In my opinion, the small sample problem in few-shot episodes is no worse than that encountered in standard batch training. In standard supervised learning tasks, batch size as small as 1 has been used successfully to train deep networks given sufficient training epochs. Without empirical or theoretical illustration, I am not convinced that the problem the authors seek to address is a real problem.
2. The reasoning behind equation 2 is unclear. It is not clear why the attention module output in CEAM is added to the embedding F if the goal is to ignore bad examples and emphasize good examples. The choice to ignore class labels when doing attention is also surprising as it doesn’t use the fact that both episodes have the same classes.
3. The proposed method aims to use inter-episode information to stabilize representation learning, hence samples two episodes at a time and apply CEAM and CECR to the joint episode. I don’t see why the authors restrict the method to just two episodes. Classifier consistency should hold transductively across any number of episodes with the same base classes. Thus, you could make the number of episodes an hyperparameter and experimentally verify what is the best choice.
4. Grammar mistakes are common and writing generally lacks polish. 

Questions:
	Please address the points in cons above.

Minor points and additional feedback:
Introduction
“even may be impossible” -> may even be impossible
“reliance of deep neural networks on sufficient annotated training data”: you always need “sufficient” data, FSL aims to make fewer data be sufficient.
“Meta-training”, “episode” and “base class” used before definition
“Concretely, MELR consists of two key components: a Cross-Episode Attention Module (CEAM) and a Cross-Episode Consistency Regularization (CECR)” -> remove ‘a’
“two cross-episode components (i.e., CEAM and CECR) to explicitly enforcing” -> to explicitly enforce
Related Work
Very few model-based methods are mentioned but I guess that’s beyond the point here.
“Almost all existing meta-learning based FSL methods ignore the relationships across episodes” -> “Ignore” is probably not true since many works (incl. MAML) frame the problem of Meta-learning as an inter-task learning process (as presented in this review https://arxiv.org/pdf/2004.05439.pdf). There’s also this work (https://arxiv.org/abs/1909.11722) that looks at the role of shots when building episodes during and after meta-training.
Methodology
Too many inline equations. Even with a dedicated definition section there is still a new definition almost every paragraph. Important equations should be made standalone, definitions placed into its own section, and fluff math be removed.
Why does CEAM take S as argument twice? Should they be S_k and S_v instead?
Eqn 2: Is the softmax taken over rows or columns (or both) of F_qS_k^T?

[Post rebuttal] I have increased the score of my review to 7. Below is a copy-pasta of my comments post discussion:

While my original concern about how much sampling affects FSL is still not fully addressed, I think it is not a trivial question to answer and a full exploration of the topic could constitute its own paper. So although I'm not fully convinced about the motivation of this paper, I think the thorough experimental evaluation along with the strong empirical results together warrants publication. From my perspective, a particular important strength of this paper is its ablations. I'm fairly convinced that the presented implementation is likely the best way to implement the idea of cross-episode attention + distillation. I think the baseline proposed by the AC makes sense. It would be great if that could be incorporated into the final version of the paper. A possible explanation for the drop in performance when using 3 or more episodes is due to the relative decrease in episode diversity. Results on wider datasets could corroborate this hypothesis.
review_title: Review of MELR: META-LEARNING VIA MODELING EPISODELEVEL RELATIONSHIPS FOR FEW-SHOT LEARNING

Meta-review: This paper explores the effect of poorly sampled episodes in few-shot learning, and its effect on trained models. The improvements from the additional attention module (CEAM) and regularizer (CECR) are strong, and the ablations are thorough. The reviewers are not fully convinced that poor sampling is indeed the main issue. That is, it could be that CEAM and CECR improve performance for other reasons, but the hypothesis is sensible, and the reviewers believe a more thorough investigation is beyond the scope of this work.

During discussions, one note that came up is whether CEAM works because of cross-episode attention, or if the idea of an instance-level FEAT is itself a good one. One ablation to sort this out would be to apply FEAT and an instance-level FEAT on episodes that are twice as large as those seen by CEAM so that the effective episode size is the same. This would help answer: is it the reduced noise due to effectively larger episodes, a stronger attention mechanism using instance-level information, or is the idea of crossover episodes indeed the important factor? The reviewers agree that this baseline, or an analogous baseline, should be included in the final version.

Decision: Accept (Poster)