Venue: ICLR 2021 Conference

Reviews:

final_score: rating: 4: Ok but not good enough - rejection
review_content: This paper proposes a data allocation scheme for meta-learning. The authors argue that it is important to consider the number of total tasks versus the number of datapoints per task given a fixed budget of the total number of datapoints since labeling is expensive for large datasets. The paper presents an algorithm that does the data allocation using a sequential decision process (SDM) and models the problem as a two-armed contextual bandit where for each fixed number of iterations, the agent can choose to add 1 task or 1 datapoint per task. There are some theoretical results on the optimal data allocation in the linear regression setting. The authors also empirically show that the number of the fixed budget and the data allocation can affect performance fairly a bit in a sinusoid regression dataset and the CIFAR-FS dataset. The proposed SDM algorithm can also recover the optimal data allocation in the sinusoid regression setting.

For the pros of this paper, I think the idea of better data allocation for meta-learning is novel since, in standard meta-learning settings, we usually assume some predefined number of data per task and number of tasks. I also like the analysis of the data allocation scheme in the linear regression setting, which sheds some light on how we should balance the number of tasks and the number of data per task.

Meanwhile, for the cons of this paper, I can't really see the significance of the problem of data allocation, particularly the data allocation set-up in this paper where each task has the same number of datapoints. Why is the problem important? I don't find the argument about the limited budget in the paper convincing enough since it seems that some naive, hand-designed data allocation schemes (e.g. having as many tasks as possible) work just well as shown in the experiments regardless of the value of the budgets. Is this problem a bit contrived and not very important in the field of meta-learning research? I believe the setting where we do not have uniform data allocation is much more important since some tasks might be much harder to get labels, but this is not explored in detail in this paper.

Moreover, it seems that the initialization of the data allocation of the proposed SDM algorithm matters a lot. If the algorithm is so brittle w.r.t. the initialization, why can't we just do grid search instead? Also, in the CIFAR-FS results, the optimal data allocation discovered by the grid search is simply using as many tasks as possible, which is what people are doing in practice. Such results make me doubt the significance of the problem setting again.

Finally, the paper is not evaluated on some widely used meta-learning benchmarks such as Omniglot, miniImagenet and etc.. Getting more empirical evidence on those benchmarks would be important.

Given the above comments, I would vote for a reject for this paper for now.
review_title: Review #2

final_score: rating: 4: Ok but not good enough - rejection
review_content: The authors study the problem of finding the optimal allocation of labels across training tasks given a fixed budget. The authors mainly want to answer the question that "if the total number of labels across training tasks is limited, it is better to have a large number of tasks  with very small data in each or a relatively smaller number of highly labeled tasks?" Although the authors prove that the optimal scaling of the number of tasks in synthetic tasks, the conclusions remain unclear while handling real-world challenging tasks. More specifically, the authors get the result on CIFAR-FS that the optimum allocation is close to the maximal number of tasks, which is weird and not consistent with previous studies in the paper. 

Although the authors provide a possible interpretation that the model has enough capacity to perfectly cluster 5 classes with a very small number of examples for each class, such an interpretation is not convincing considering that meta-learning models still have a big room to improve. Moreover,  most meta-learning models adopt a similar or even more complex model, the key research question proposed in the paper about optimal allocation between tasks and labels will be less interesting if the proposed interpretation is correct.  

Overall, the question proposed in the paper is interesting and important. I will suggest that the authors should further study the weird outcome on the real-world datasets.


review_title: The problem needs to be further studied

final_score: rating: 4: Ok but not good enough - rejection
review_content: Summary
----------

This paper presents an active learning approach to allocation of labels across tasks in meta-learning. The approach is based on a contextual bandit setting, and the approach is tested on linear regression, sinusoidal regression, and a version of CIFAR. The authors also present a theoretical analysis for the linear setting.

Comments
----------

The direction considered in this work is interesting, but the paper feels largely incomplete. Overall, the writing is quite unclear throughout, and the presentation of the SDM algorithm/approach is hard to follow. 

The largest shortcoming of the paper is the apparent gap between the the linear results and the nonlinear results. The theoretical analysis for the linear setting is relatively interesting, and the linear experiments appear to match closely. The sinusoidal results also match closely, but this problem can be solved by linear regression on learned nonlinear features, so it is unclear how much of a jump this setting is from the purely linear setting. The proposed approach appears to perform poorly for CIFAR. The authors should include evaluation on standard meta-learning benchmarks such as miniImageNet to show the utility of their approach.

The presentation of the algorithm is quite unclear, and it is unclear how practical the proposed approach is. Formulating the problem as a sequential decision-making problem is logical. However, given the relatively small number of labeled examples considered in this work, the validity of an assumption of a labeled meta-validation set is somewhat questionable. Indeed, given the sparsity of labeled data, it seems unlikely that a large amount of validation data should be withheld from training. I could not find a discussion of the size of this validation set. Indeed, a much more interesting approach for estimating error would be the prediction error associated with adding a new label, as opposed to an external validation set. Moreover, the authors discuss training for 100 epochs after each decision-making step, which seems prohibitively expensive. 

Finally, the discussion of existing literature was quite shallow and should be explored much more deeply. In particular, there are likely many connections in active learning and the design of experiments that are highly relevant to this setting. 

review_title: Review 3

final_score: rating: 6: Marginally above acceptance threshold
review_content: ********Summary

In most popular meta-learning approaches, there are usually a pre-defined number of data per task. For example, 1-shot or 5-shot learning. It has shown as the number of data increases in such methods, model performance improves. In this paper, they try to analyze the effect of having different number of tasks with different number of data points in meta-learning benchmarks. Specifically, given a fixed number of data across different tasks as a budget, they want to see if having a large number of tasks with small data in each works better (or worse) than having small number of tasks with more data in each. They focus on MAML as a meta-learning method, and analyzed the results on mixed linear regression, sinusoid regression, and CIFAR. For mixed linear regression, they showed that the optimal number of tasks is \sqrt(b). They also provide an online algorithm for finding optimal number of tasks in meta-learning.

********Positives
- The paper is well-written and well-organized. They explained their approach clearly.

- They tried to answer a very interesting question, that I have always in my mind (and I am sure many other researchers have). It can be generalized to multi-task learning when we have several source tasks for a target task. In multi-tasking there are several sources with various number of data points. The question in this paper can be applied to multi-tasking, but we should consider another important factor, which is task relatedness. Question in that setting might be what is the best combination of tasks with different number of data considering task-relatedness.

********Notes
- There is one confusing part for me. In abstract, they mentioned that "Given a fixed budget b of labels to
distribute across tasks, should we use a small number of highly labelled tasks, or
many tasks with few labels each?", However, 'uniform data allocation' assumption does not align with the original question they wanted to answer! It seems to me that they answer the optimal number of tasks in meta-learning approaches (MAML specifically) when each task has the same number of data.


- I would recommend spending more time on explaining section 4.1. (SEQUENTIAL DECISION MAKING FOR THE OPTIMAL DATA ALLOCATION)

- Where was their theoretical guide as they mentioned in the abstract and introduction, section 5.1? 

- Even though the experiments results are really interesting, I would recommend adding one or two more experiments (for example image classification), and also validate their analysis on other meta-learning benchmarks (rather than MAML)

********Reason to accept or reject

They are pointing to very important problem in meta-learning methods. This problem would be interesting to see for other researcher in this area, and it could be generalized to other related research topics such multi-task learning. However, I think they simplified the problem (by uniform assumption), and they could add more experiments to prove their claim. In general, I liked this work, and it could be useful for others to continue since it's not answered yet (at least as far as I know). I would be happier with more experiments or even other meta-learning approaches. So, my score to this paper is marginally above acceptance threshold.
review_title: address an interesting question in meta-learning; uniform simplified assumption; more experiments would be helpful

Meta-review: This paper studies the problem of how data should be balanced among a set of tasks within meta-learning. This problem is interesting, and largely hasn't been studied before. However, the reviewers raised several shortcomings of the current version of the paper, including the significance of the problem setting, the limited experimental study (i.e. the only experiment with real data is CIFAR-FS), the depth of the related work section, and the clarity/impreciseness of the writing. Further, the paper has not been revised to address any of these shortcomings. As such, the paper is not ready for publication at ICLR.
Decision: Reject