Venue: NeurIPS 2021 Conference

Reviews:

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: This submission studies episodic sampling, which is used to sample datasets (consisting of support and query sets) for training meta-learning models. The authors define a notion of episodic difficulty (the negative log-likelihood of the query set given the support set and model parameters) and propose using importance sampling to train using several different types of episode-sampling strategies for meta-training. By using importance sampling, there is no sampling required for the target distribution but just weighting the episode loss based on the proposal and target distribution. They conduct experiments both to confirm some of their assumptions that are critical for their proposal (that episodic difficulty is normally distributed) and to show how their episodic sampling strategies compare against random episodic sampling, which is the dominant strategy used by work in this area.

main_review: Originality: the paper considers an under-studied but important aspect of meta-learning that to my knowledge hasn't been studied in previous work. Rather than modify an existing architecture or method for meta-learning, the authors are studying an interesting aspect of the problem that most work in this area take for granted.

Quality: authors conducted experiments beyond simply comparing methods on benchmarks to confirm assumptions made in the paper. These include confirming the normal distribution of episodic difficulty or that episodic difficulty is independent from modeling choice. Additionally, benchmark experiments were conducted for a variety of networks, datasets, and meta-learning methods.

Clarity: the paper is well-written - the ideas are presented clearly and the description of the proposed method and experiments are easy to understand.

Significance: my main concern with the paper is the experimental results. I believe the sampling strategies proposed here would be used by the community if the benefits they offer are significant. I believe the online strategy is much more feasible for use in training but I don't believe it's clear it offers a big benefit in performance. For example, in 1/3 of the scenarios presented for mini-ImageNet and tiered-ImageNet, the baseline sampling strategy is better or equivalent in terms of performance. Additionally, for the FEAT comparison, the proposed strategy offers basically the same performance as the baseline one. Thus, because of the work involved in implementing the proposed sampling strategies, I'm not sure the benefit shown is large or consistent enough for this to become something standard that is used in this research subfield.
limitations_and_societal_impact: Though not discussed, the societal impact of this work is probably similar to that of any other few-shot method.
review_title: null

final_score: rating: 8: Top 50% of accepted NeurIPS papers, clear accept
review_content: summary: This paper investigates the effect of episode difficulty and sampling schemes on the performance of few-shot classification algorithms. It defines difficulty based on the negative log predictive likelihood of the few-shot learner and proposes to use importance sampling to mimic various sampling strategies. Experiments demonstrate that episodic difficulty is (a) approximately normally distributed, (b) tends to transfer across architectures and learning algorithms, (c) tends to be consistent over the course of training, and (d) that a uniform sampling scheme over difficulty tends to produce the best results. An online scheme for computing importance weights is also proposed.


=== Post-rebuttal ===

I have read the authors' response and other reviews and will maintain my original rating.
main_review: Episodic training underpins many few-shot learning algorithms and thus understanding it well is a key concern. For this reason, the results presented in this paper will be of broad interest to the few-shot learning community. The proposed online uniform difficulty sampling scheme is simple and therefore should be relatively easy to implement.

The paper is very well-written and easy to read. Experiments are systematic and easily interpreted. Though the performance gains from uniform difficulty sampling are not always very large, it does appear to be consistently better than other approaches.

I would have liked to see a discussion of future extensions, for example training a network to directly estimate sample difficulty or possibly a policy network that outputs the importance weight to be assigned to an episode. Also it would be good to understand what the authors tried that did not work. For example, were other measures of episode difficulty such as accuracy considered? Were other sampling schemes tried and discarded?

Overall I believe this is a strong paper and that others will benefit from reading it.
limitations_and_societal_impact: This work proposes a general purpose improvement to episodically trained few-shot learning algorithm and therefore the same societal implications that result from few-shot learning also apply here.
review_title: null

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: Meta-learning (few-shot learning) algorithms, in general, equally treat all tasks for the training process. Tasks are sampled uniformly at random. This paper proposes a novel meta-learning framework that controls the task sampling probabilities according to difficulties. This paper measures the difficulties of tasks using the negative log-likelihood and leverages the difficulties to introduce different weights to tasks. The main idea of the learning framework is that the prior of the difficulties follows a normal distribution while the target sampling distribution could be uniform or some other interesting forms. Then, important sampling provides different weights to different tasks. From experiments, the uniform target distribution shows good results. 
main_review: Originality: 
The paper proposes a novel task sampling framework for meta-learning, which is, indeed, a task weighting method considering task difficulties. The authors should discuss other existing task weighting methods. 

Quality: 
The proposed episodic training method is intuitive, and the performance looks good.

The proposed algorithm is evaluated with ProtoNet, MAML, and ANIL. 

Although the training algorithms are well-known algorithms, they are not the state-of-the-art. Since one of the claims is that the proposed method is algorithm agnostic, it is necessary to perform experiments with recent training algorithms. 

The authors only consider Mini-ImageNet and Tiered-ImageNet for the evaluation. In meta-learning research, specific data sets such as CUB and Cars are also used frequently, and cross-domain tasks are also important. More experiments with various data sets would be very useful.

Although the performance shows great results, it is not clear why uniform sampling outperforms other sampling distributions. The choice of the negative log-likelihood as the difficulty measure also is required to be justified. What will happen when we use other difficulty measures?

Clarity: 
This paper is easy to read.

Significance: 
This paper proposes a novel episodic learning framework. The idea could be used to build other research results.

limitations_and_societal_impact: This paper did not properly discuss their limitation. Please refer to the main review comments.
review_title: null

final_score: rating: 8: Top 50% of accepted NeurIPS papers, clear accept
review_content: summary: This paper explores episode/task difficulty in few-shot learning. The task difficulty is defined as the negative log-likelihood over the query set given the support set and model parameters. The authors find evidence that episode difficulty likely follows a normal distribution and proposes a new method, called UNIFORM, based on importance sampling. The new method achieves similar or better performance than other sampling procedures (EASY, HARD, and CURRICULUM). The paper also presents two variants of UNIFORM: offline and online. 
main_review: \+ means strength, \- means weakness

**Originality**

\+ *Interesting and novel insights into the task difficulty for meta-learners.* Although the idea of controlling episode difficulty is not very novel [47, 28], the authors go about it in a novel way and propose to use importance sampling for a faster and more efficient way of sampling episodes that is model agnostic. 

**Quality**

\+ *Solid evaluation and analysis.* Testing over multiple models, algorithms, and benchmarks makes for a convincing set of results.

\+ *Confirmed assumptions through experimentation.* The paper uses Q-Q plots and Shapiro-Wilk to verify the assumption that episode difficulty follows a normal distribution. 

\- *Advantage of UNIFORM over other procedures is not consistent.* The tables show that UNIFORM does not always offer a clear advantage over the results, especially in the 1-shot setting. Do the authors have a theory for why the method is not as effective on the 1 shot setting?

**Clarity**

\+ *Experiments are well designed, and the results are clear.*

\+ *Paper is organized and well-written.* 

**Significance**

\+ *Interesting solution to a relevant problem.* Episode difficulty is understudied in few-shot learning, and uniform presents a model agnostic solution to the problem of how to sample tasks during meta-learning.


_________
## Post-Rebuttal
_________

I have read the rebuttal and other reviews, and I increase my original rating. This paper addresses an interesting and relevant issue of episode difficulty in meta-learning and will be a particularly valuable contribution at the conference. The paper presents a solid set of evaluations and analyses, verifying assumptions (e.g. via Q-Q plots and Shapiro-Wilk tests) and reporting results across multiple benchmarks. The paper presents a novel, simple but effective method that, on the most part, boosts algorithm performance by a couple of additional accuracy points on average. Therefore, I recommend this work for the conference with a score of 8. 
limitations_and_societal_impact: Limitations not given.
review_title: null

Meta-review: The submission investigates the question of how to sample episodes for few-shot learning. By introducing a measure of episode difficulty based on the few-shot learner's NLL on the episode, the authors experiment with importance sampling to simulate different episode difficulty distributions. They observe that without any intervention, episode difficulty is normally distributed, and that a uniform distribution during meta-training yields better results.

Reviewers found the paper well-written and easy to read, and found the proposed approach original and easy to implement. Even though the improvement margins are not substantial, reviewers thought that the ideas and experiments have a broad enough appeal to the few-shot learning community to make the submission a valuable contribution. I therefore recommend acceptance.
Decision: Accept (Spotlight)