Venue: NeurIPS 2021 Conference

Reviews:

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: In this paper, the authors propose an adaptive task scheduler to pick the set of tasks that a meta-learner should solve at a given iteration which will improve its generalizability/transferability to downstream tasks rather than picking a set of tasks at random. Instead of choosing a metric/heuristic based approach, the authors propose to use a second network that learns-to-learn the right set of tasks to sample and the overall training progresses via an interleaved (bi-level) optimization procedure. Authors provide empirical evidence based on two datasets that their proposed scheduler ATS performs better than similar existing approaches. The paper also contains some theoretical justifications on why ATS performs better than random sampling.
main_review: In this paper, the authors propose an adaptive task scheduler to pick the set of tasks that a meta-learner should solve at a given iteration (rather than picking a set of tasks at random) with an aim to improve its generalizability/transferability for downstream tasks. The authors propose to use a learning based approach by using a second network to automatically determine the weightage of each task from the task pool and then picking the top K tasks from that pool. For the meta-learning part, the authors use a standard gradient based learner (ANIL) and for the scheduling-learner part, they use an LSTM based network and use REINFORCE to train the scheduler network. The authors perform experiments on one classification and one regression task and show that their proposed task scheduler performs better than existing ones when it comes to the final test performance. Authors also provide some theoretical justifications as to why their method provides better generalization compared to random selection.

Overall, the work is good and the paper was easy to follow - the narrative and the notations were all clear. The idea also has enough novelty, especially using another network to select the "hardest" tasks (based on support-query difference and query loss) and then using REINFORCE to train that in a DARTS-style (NAS) bi-level setup.

I have a few concerns that I would like authors to address:
*  The authors mention using two LSTMs for the task selector network. Why was LSTM chosen? What is the temporal piece of information that you wanted to capture?  
*  The authors perform some ablation studies but there is one that I think is missing - what if, instead of using another network to learn the right task to choose, you just heuristically determine the best tasks by performing some combination of the two metrics selected -- the  query loss and the gradient similarity between the support and query? For example, what happens if you just pick the top K tasks by taking a simple average of these two metrics instead of having another network that takes these as inputs and then learns to pick the top K tasks? Will there be a significant performance difference between these two?
*  To me, it looks like there will be non-trivial computational overhead in adopting this approach vs random sampling. I'd like to see some training time comparison between the methods which will be more informative for a user to decide if the extra computation gain is worth it.
* For MiniImageNet, the authors mention that number of tasks is $4368$ when $16$ classes are chosen because $16  \choose 5$ is $4368$. However, while picking $5$-way tasks, you also need to pick $5$-shots i.e. choosing $5$ images per class from $600$ images. Therefore, number of distinct tasks is way more than $4368$. I would like the authors to clarify why they mention $4368$ as the total number of tasks in this case. 
* Finally, pardon my naivety but I do not follow the derivation of proposition 1 (given in appendix). How did you go from line 1 to line 2 in the proof (starting at line 6 in the appendix)? Also, in the manuscript, it should be mentioned that the proof is given in the appendix. 
limitations_and_societal_impact: There are no potential negative societal impact of their work as far as I understand.
review_title: null

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: Meta-learning is a popular strategy to learn a well-generalized model from different meta-training tasks. Classical meta-learning assumes equal importance for all the meta-training tasks. Recent work has proposed sampling of the tasks based on various factors such as difficulty, amount of information, etc. The paper presents an extension to these ideas by proposing an adaptive task scheduler. In contrast to the prior approaches where the task distribution is hard-crafted and fixed, the authors propose a learnable task scheduler that outputs the sampling probability score of each task in a task pool to directly minimize the generalization error. The inputs to the task scheduler - the query loss and the similarity between the gradients on the query and support set are theoretically grounded and well-motivated. The meta-model and the tasks scheduler are learned using a bi-level optimization; in particular, REINFORCE is used to update the task scheduler. The authors conduct experiments on the mini-imagenet (classification) and drug discovery (regression) datasets to validate the effectiveness of the task scheduler. The results indicate the robustness of the task scheduler on both the datasets under noisy conditions and limited task budget.

main_review: Meta-learning is among the popular approaches for learning generalizable models. An aspect of this learning paradigm that could be further optimized was task selection. While the recent approaches along this direction use a fixed or hand-crafted method to select the tasks, this paper proposes a novel method to learn the distribution automatically. This is a valuable contribution. I liked the theory motivating the two factors input to the automatic task scheduler. This is the strongest contribution of the paper. I am sure that this will spawn new focussed research on improving the task scheduler for meta-learning. Having said that, there are a couple of issues with the methodology as discussed below.

Randomly selecting $N^{pool}$ of tasks is understandable due to computational constraints. However, I do not see the need for a second round of task selection from this candidate pool based on the estimated weights. Why cannot the meta-model be trained using all the $N^{pool}$ tasks? According to Section C.1, the meta-batch size is 2 and $N^{pool}=10$. Is there any issue with $N^{pool}=4$ and the meta-batch containing all the tasks in the $N^{pool}$? I ask this as the theory seems to be more aligned with weighted task loss rather than sampled task loss.

Experiments on the noisy mini-imagenet and drug discovery datasets validate the effectiveness of the ATS. The ablation studies are also adequate. However, there are two issues with the experimental setup, as elaborated below.

- The authors restrict the evaluation to only the noisy version of the mini-imagenet. While ATS might work best on noisy data, an optimal ATS module should work on any generic meta-learning task (even ones without noise). The bare minimum would be to validate ATS on the original mini-imagenet, tiered-imagenet, and CIFAR-FS benchmarks widely adopted in the literature. Table 3 suggests that ATS does not yield significant improvements with reduced noise levels. Thus, it is natural to gauge the performance on a ‘noiseless’ dataset. No noise in the dataset does not mean the tasks are easily learnable. The suggested benchmark datasets are extremely challenging and are perhaps hard learning tasks. This will test the performance of ATS on truly hard tasks. I agree that ATS might perform better on highly noisy datasets (noise level of 60% and above), but such settings as illustrated in the paper are quite unrealistic.

- ATS is compared only against other sampling strategies. In my opinion, this is an unfair comparison. Liu et al. eccv 2020 [16] - GCP; compares the performance of various meta-learning approaches on the standard benchmark datasets using different backbones. I believe it is necessary to compare the performance of the ATS with some of these approaches. This will address a fundamental question - is task selection critical for meta-learning? The paper implicitly assumes it and does not present compelling evidence. It would be worthwhile to adopt and compare the sampling strategies on meta-learning approaches that allow for it, instead of restricting only to the ANIL framework. 

In my humble opinion, addressing the following comments will further strengthen the contribution.

1. The results on all the datasets require additional significance tests. This is especially true with the drugs dataset where the difference in the mean $\mathcal{R}^2$ is in the second decimal place.
2. Line 268 - Are the weights for the 1000 meta-training tasks estimated at the same time by the ATS? If so, at what stage in the training process is these weights estimated?
3. The authors mention in the supplementary material that warm-start initialization refers to the meta-model that is pre-trained using the original datasets (without noise). I think this is unfair, as the meta-model has already seen the clean meta-training tasks. The proposed training process is only trying to maintain the performance. This also negates the primary assumption of the paper - training with noisy and hard tasks. I believe a fair comparison warrants training from scratch on the noisy datasets directly.
4. Line 231 mentions a set of clean validation tasks for ATS. What is the role of this validation set? This can be further added to Algorithm 1.
5. Would you please explain the pre-training process of a neural scheduler? Is the meta-model $\theta_0$ used during pre-training randomly initialized or pre-trained? If $\theta_0$ is pre-trained, is it pre-trained along the lines of supervised learning on meta-train set or episodic regimen? Does pre-training of neural scheduler $\phi$ follow episodic regimen? Does pre-training of $\phi$ and $\theta_0$ both occur on clean data? Is pre-trained $\theta_0$ used as such (warm start), or $\theta_0$ is reinitialized during the iterative optimization phase?

The paper is well structured and written. The ideas are easy to understand and well-articulated.

Corrections to the theory in the supplementary material----

- In the proof of proposition 1, we are adding and subtracting two terms $\mathcal{L(D_i^q;\theta_i})$ and $\sum_i^{N^{pool}}w_i\mathcal{L}(D_i^q;\theta_i)$. The second term in the first step of the proof should be $-\sum_{i=1}^{N^{pool}}w_i(\mathcal{L}(D_i^q;\theta_i)-\mathcal{L}(D_i^q;\theta_i))$

- Also, the second order term in equation 3 of the supplementary material is incorrectly stated (Though it does not impact the rest of the analysis). It should be $\alpha^2 \nabla_{\theta_0}\mathcal{L}(D_i^s;\theta_0)^T\nabla_{\theta_0}^2\mathcal{L}(D_i^q;\theta_0)\nabla_{\theta_0}\mathcal{L}(D_i^s;\theta_0)$

Typos

Line 141 - differentiate - differentiable

Line 203 - follow - following

Post response************
I thank the authors for providing a detailed response to my queries.

I am satisfied with the authors' response to most of the questions. I still have a few lingering questions/suggestions that will further add clarity to the results

Q2 and Q3 I am not convinced that the results in Table 5 represent the full budget setting. The accuracies for MAML in the 5.1 and 5.5 settings stand at 48.7% and 63.1% respectively as reported in the literature. However, the table reports only 45.7% accuracy. Furthermore, in the response to Q3, the authors state MAML performance as 47.09% and 63.1% for the two settings. So, why are the accuracies for the classical MAML different at different places within the paper?

Q4 - I am assuming that the term after $\pm$ is the standard deviation. Significance tests do not mean only reporting the standard deviation, but checking the p-value for this the results are significant. At the moment, no such test has been performed.

Overall, I am inclined to raise my rating of the paper.
limitations_and_societal_impact: yes
review_title: null

final_score: rating: 5: Marginally below the acceptance threshold
review_content: summary: The author proposed a dynamic sampling approach to form meta learning tasks, with the sampling distribution determined by 1) loss of the tasks, 2) the similarity between gradient of support and query data. Experimental results on the setting of noisy meta learning and limited budgets demonstrate the effectiveness of the proposed approach.
main_review: 1)   I agree on the author’s intuition which is to sample out tasks that  are informative and not outliers. The author used loss to determine on the difficulty and used the alignment of gradient between support and query data to determine outlier. However, this alignment of gradient can’t infer the task is not an outlier, i.e., when the tasks are from more than one domain, task of another domain can align perfectly by itself but still of little value to the current domain. The algorithm design needs improvement to cover such cases.

2)   The author used LSTM stacks to process on the two factors. Given meta learning models are usually light weight by itself, it would be beneficial to know on the computational cost from this added component and how it impacts the training efficiency.


3)    Better to add a little description on motivation of adding label noise in the data. i.e. will this make it more close to real world scenarios? Or it is for a fair comparison with previous works and less bias prone? etc.

4)  The paper works on noisy meta learning setting and demonstrate the effectiveness of the proposed approach. I would like to know what is the performance on the normal meta learning setting without noise. 



limitations_and_societal_impact: The authors discussed limitations.
review_title: null

final_score: rating: 7: Good paper, accept
review_content: summary: This paper proposed Adaptive Task Scheduler (ATS) for meta-learning, which trains a neural network to adaptively select tasks. A bi-level optimization strategy is used to optimize the the meta-model and neural scheduler. Experiments are conducted on miniImagenet  and drug compounds, which beats the baseline methods. 
main_review: The paper is a good attempt on the adaptive task scheduler. The proposed algorithm totally make sense to me. In general, the paper is well-written and easy to follow. The experimental results are nice. 


Typos:
Line 140: computationally
Equation (6): the definition of $\theta$ should be $\theta_{i}^{(k)}$
Equation (7): should be $\theta_{i}^{(k)}$ in the bottom equation.
Equation (9): should be $\theta_{i}=\theta_{0}-\alpha \nabla_{\theta} \mathcal{L}\left(\mathcal{D}_{i}^{s} ; \theta_0\right)$
limitations_and_societal_impact: I think the story brought by Proposition 1 is questionable. Indeed, certain sampling probability $w$ could make the meta-training loss lower. However, meta-training loss is not our ultimate goal. Our goal is the performance on the test set. In some extreme cases where some tasks available in the training phase have very low noise, such that one would give them high sampling probability to reduce the meta-training loss. However, they might be unrelated to some tasks in the test phase. So I am not sure how meaningful is Proposition 1.
review_title: null

Meta-review: The idea of the tasks scheduler for meta-learning is interesting. The solution is technically sound and intuitive. The theoretical analysis is also interesting, although it requires more discussions on when the assumption can hold. The experiments are quite exhaustive. The discussion on the computational cost is non-trivial, and more detailed analysis is needed
Decision: Accept (Poster)