Venue: ICLR 2022 Conference

Reviews:

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: In this work, the authors propose various task-sampling strategies for an episodic meta-learning setup and compare their performances against a standard uniform sampling. Through experiments across diverse benchmark datasets, the authors empirically show which of these methods underperform the uniform sampling and which of those perform at-par. 
main_review: In this work, the authors provide an empirical evaluation of various sampling mechanisms which can be used to create few-shot tasks during an episodic training regime that's common with meta-learning training. The list of sampling strategies include simple methods like only using a fixed set of tasks in each episode and more complicated strategies like online-hard-negative-mining. Through experiments on multiple popular few-shot learning datasets, the authors show which sampling methods do work well and which of them do not.

Strengths
--------------
* Narrative is clear and the paper is enjoyable to read. Various pictorial representations of different sampling strategies make it easy to follow.
* Experimental setup is quite exhaustive, from both the sampling mechanism point of view and also the baseline datasets chosen. 

Weaknesses
------------------
* Lack of Novelty -- There is no clear novel contribution in this paper either from a theoretical or empirical perspective. The authors evaluated a series of existing task-sampling mechanisms using a bunch of existing meta-learning algorithms on few-shot benchmarks. It would have been good to see some novelty in terms of the sampling mechanism -- if uniform sampling is really the best one and that is what practitioners generally use as far as I know, then this analysis does not provide a lot of value.

* Lack of Explanation and Unfair Comparisons -- The paper does not do a good job in explaining why some of the sampling methods work better than others and in some cases, the explanations are trivial. For example, the NDT sampler performs poorly because it is only trained on a subset of data compared to the uniform sampler and therefore, is expected to provide weaker performance. However, the NDB sampler performs almost similar to uniform although it also works with limited amount of data. Why is that? What is inherently different between the subset of data selected by these two methods? One explanation I can think of is that for NDB, the initial batch contains some representations from a majority of classes in which case, its comparison to NDB is again unfair (which only contain images from a limited classes). Another similar topic is why convergence of SBU is slower than NDTB? Is it because in NDTB, each task is repeated K-times and the model has simply performed K-times more optimization steps compared to SBU? In that case, performance of SBU vs NDTB should be compared when SBU has run for k-times more epochs. 
summary_of_the_review: Although the empirical setup is well designed and the paper is easy to follow, the lack of novelty in the contribution and the apparent lack of clarity in the explanations/analysis led to my non-acceptance for this paper at this point.
review_title: null

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: In this paper, the authors investigate the effect of task diversity in the training process of meta-learning. The findings indicate that increasing task diversity during the meta-training process does not boost performance. They evaluate the performance on four few-shot image classification datasets. 
main_review: In this paper, the authors investigate the effect of task diversity in the training process of meta-learning. The findings indicate that increasing task diversity during the meta-training process does not boost performance. They evaluate the performance on four few-shot image classification datasets. 

Though the authors investigate several samplers, they only conduct the experiments on N-way K-shot few-shot image classification tasks. The findings do not surprise me since the meta-training process of N-way K-shot image classification essentially learns a better representation using all meta-training data samples (see the discussions in Baseline++, Meta-Baseline). Thus, using a less diverse sampler does not change the number of training samples, and thus it may not hurt the performance. 

To make the findings more convincing and exciting, I suggest that the authors conduct the experiments on much more diverse tasks, e.g., regression tasks and noisy data. It would also be interesting to conduct qualitative analysis on toy tasks to understand why task diversity does not benefit the performance.

[1] Dhillon, Guneet S., Pratik Chaudhari, Avinash Ravichandran, and Stefano Soatto. "A baseline for few-shot image classification." arXiv preprint arXiv:1909.02729 (2019).

[2] Chen, Yinbo, Zhuang Liu, Huijuan Xu, Trevor Darrell, and Xiaolong Wang. "Meta-Baseline: Exploring Simple Meta-Learning for Few-Shot Learning." In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 9062-9071. 2021.
summary_of_the_review: I would recommend rejection since the authors only investigate few-shot image classification tasks. More analysis and experiments are needed for such an analysis paper.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: The paper studies how the diversity of tasks in the training phase affects the performance of meta-learning algorithms. The paper finds negative evidence, which is consistent with Setlur et al. (2021). Compared with the existing work, the paper performs more extensive experiments with different meta-learning algorithms, different task samplers, and different datasets.
main_review: Significance: The paper follows an existing line of work that empirically shows task diversity (in the training phase) does not help with meta-learning. This is an important step towards a better understanding of meta-learning. 

Novelty: The key finding that task diversity may not be beneficial for meta-learning has been proposed and studied by Setlur et al. (2021). The paper extends the same findings to (a) different meta-learning algorithms, (b) different task samplers, and (c) different datasets. For this reason, I find the novelty of the paper limited.

There are some technical ambiguities in the paper, as discussed below.
1. The explanation of the empirical results is not insightful and convincing enough. In section 5, the authors attribute the poor (or good) performance of various diversity samplers to noise (or lack of thereof) introduced when training on multiple sub-datasets. This reason is not particularly insightful or useful to other researchers and needs more detail. For example:
(a) what noise the authors are referring to? There is no noise introduced in the problem formulation in section 3.1.
(b) how does diversity in tasks introduce noise?
(c) how does this noise affect the performance of meta-learning?
(d) can we perform ablation studies to confirm that the samplers’ poor performance is indeed caused by noise?
2. Some choice of task samplers in section 3.3 lacks motivation. For some task samplers, e.g., Online Hard Task Mining and Static DPP, the authors provide citations to various works that propose the use of these samplers. However, for the remaining samplers, the paper does not discuss why these samplers are good candidates to compare with the uniform sampler.
summary_of_the_review: The paper provides an extensive set of empirical evidence to demonstrate that task diversity (during training) is not beneficial for meta-learning. The insights and conclusions drawn from these empirical experiments, however, are not so convincing and helpful, i.e., it does not tell other researchers how to rectify the problem or how to design better meta-learning algorithms.
review_title: null

Meta-review: This paper set out to show that increasing task diversity during meta-training process does not boost performance. The reviewers mostly  agreed (only reviewer wVFn dissented) that the empirical set up of the paper was convincing, but they also felt it over-emphasized empirics over a deeper understanding of the phenomena observed. In turn, this resulted in discussions around how the experiments and the explanations didn't fully prove that increasing task diversity does not help. Overall, the discussion and the additional analysis tools provided by the authors (such as the diversity metric) will greatly improve the paper.
Decision: Reject