Optimal allocation of data across training tasks in meta-learning
Authors: Georgios Batzolis
Alberto Bernacchia
Da-shan Shiu
Michael Bromberg
Alexandru Cioba
Venue:  (2021)

Abstract: Meta-learning models transfer the knowledge acquired from previous tasks to quickly learn new ones. They are tested on benchmarks with a fixed number of data-points for each training task, and this number is usually arbitrary, for example, 5 instances per class in few-shot classification. It is unknown how the performance of meta-learning is affected by the distribution of data across training tasks. Since labelling of data is expensive, finding the optimal allocation of labels across training tasks may reduce costs. 
Given a fixed budget b of labels to distribute across tasks, should we use a small number of highly labelled tasks, or many tasks with few labels each? In MAML applied to mixed linear regression, we prove that the optimal number of tasks follows the scaling law sqrt{b}. We develop an online algorithm for data allocation across tasks, and show that the same scaling law applies to nonlinear regression. We also show preliminary experiments on few-shot image classification. Our work provides a theoretical guide for allocating labels across tasks in meta-learning, which we believe will prove useful in a large number of applications.


Under review as a conference paper at ICLR 2021
OPTIMAL ALLOCATION OF DATA ACROSS TRAINING TASKS IN META-LEARNING
Anonymous authors Paper under double-blind review
ABSTRACT
Meta-learning models transfer the knowledge acquired from previous tasks to quickly learn new ones. They are tested on benchmarks with fixed number data-points for each training task, and this number is usually arbitrary, for ex- ample, 5 instances per class in few-shot classification. It is unknown how the performance of meta-learning is affected by the distribution of data across train- ing tasks. Since labelling of data is expensive, finding the optimal allocation of labels across training tasks may reduce costs. Given fixed budget of labels to distribute across tasks, should many tasks with few labels each? In MAMI applied to mixed linear regression, we prove that the optimal number of tasks follows the scaling law vb. We develop online algorithm for data allocation across tasks, and show that the same scal- ing law applies to nonlinear regression. We also show preliminary experiments on few-shot image classification. Our work provides a theoretical guide for allocat- ing labels across tasks in meta-learning, which we believe will prove useful in a large number of applications.
a small number of highly labelled tasks,
INTRODUCTION
Deep learning (DL) models require large amount of data in order to perform well, when trained from scratch, but labeling data is expensive and time consuming. An effective approach to avoid the costs of collecting and labeling large amount of data is transfer learning: train a model on one big dataset, or a few related datasets that are already available, and then fine-tune the model on the target dataset, which can be of much smaller size (Donahue et al. (2014)). In this context, there has been recent surge of interest in the field of meta-learning, which inspired by the ability of humans to learn how to learn Hospedales et al. (2020). A model is meta-trained on a large number of tasks, each characterized by a small dataset, and meta-tested the target dataset.
The number of data points per task is usually set to an arbitrary number in standard meta-learning benchmarks. For example, in few-shot image classification benchmarks, such (Vinyals et al. (017)), Ravi & Larochelle (2017) and CIFAR-FS (Bertinetto et al. (2019)). ) this number is usually set to 1 or 5. So far, there has not been any reason to optimize this number, in most circumstances the performance
Imageee
a model will improve with the number of data points (see Nakkiran al. (2019) for exceptions). However, if the total number of labels across training tasks is limited, is it better to have a large number of tasks with very small data in each, or relatively smaller number of highly labelled tasks? Since data-labeling is costly, the answer to this question may inform the design of new meta-learning datasets and benchmarks.
In this work, to our knowledge, we answer this question for the first time, for specific meta-learning algorithm: MAML (Finn et al. (207))) We study the problem of optimizing the number of meta- training tasks, with fixed budget b of total data-points to distribute across tasks. We study the application of MAML to three datasets: mixed linear regression, sinusoid regression, and CIFAR. In the case of mixed linear regression, we derive an approximation for the meta-test loss, and according to which the optimal number of tasks follows the scaling rule vb. In order to optimize the number design an algorithm for online allocation of data across training tasks, and
of tasks empirically, we validate the algorithm by performing grid search over a large set of possible allocations. In summary, our contributions are:

---

Under review as a conference paper at ICLR 2021
We introduce and formalize the problem of optimizing data allocation with fixed budget b in meta-learning.
We prove that the optimal scaling of the number of tasks is vb in mixed linear regression, and confirm this scaling empirically in nolinear regression.
We introduce an algorithm for online allocation of data across tasks, to find the optimal number of tasks during meta-training, and validate the algorithm by grid search.
We perform preliminary experiments on few-shot image classification.
2 RELATED WORK
A couple of recent papers investigated a problem similar ours. In the context of meta-learning and mixed linear regression, Kong et al. (2020) asks whether many tasks with small data can compensate for a lack of tasks with big data. However, they do not address the problem of finding the optimal number of tasks within a fixed budget. The work Shekhar et al. (2020) studies exactly the problem of allocating fixed budget of data points, but to the problem estimating a finite set of discrete distributions, therefore they do not study the meta-learning problem and their data has no labels.
An alternative approach to avoid labelling a large amount of data active learning, where a model learns with fewer labels by accurately selecting which data to learn from (Settles) (2010). In the context of meta-learning, the option of implementing active learning has been considered in a few recent studies (Bachman et al. (2017), Garcia & Bruna (2018), Kimm et al. (2018), FFinn et al. (2019), Requeima et al. (2020)). However, they considered the active labeling of data within a given task, for the purpose of improving performance in that task only. Instead, ask how data should be distributed across tasks.
In the context of recommender systems and text classification, few studies considered whether labeling a data point, within a given task, may increase performance not only in that task but also in all other tasks. This problem has been referred to as multi-task active learning (Reichart et al. (2008)/Zhang
(2008). Zhang , Saha et al. (2011), 1). Harpale 2012). Fang et al. (2017), or multi-domant active learning (Li al. (2112), Zhang et al. (2016)). However, none of these studies consider the problem of meta-learning with a fixed budget. A few studies have looked into actively choosing the next task in a sequence of tasks (Ruvolo & Eaton (2013). Pentina et al. (2015), Pentina & Lampert 2001
Sun et al. (2018)), but they do not look at how to distribute data across tasks.
3 THE PROBLEM OF DATA ALLOCATION FOR META-LEARNING
In the cross-task setting, we are presented with hierarchically structured dataset, with task param- eters (r()))m sampled from T p(T) and data (ær (= sampled from D == (DIT) p(
r). Our problem is minimizing the following loss function with respect a parameter w:
L [
= E L (w:x")
(1)
D.
The empirical risk mininization principle (see Vapnik (1988) ensures that the optimum of the empirical risk converges that of the true risk with an increase samples from the joint distribution of (D, T)
3.1 META-LEARNING ACROSS TASKS
In the meta-learning problem, we are given the opportunity to adjust the objective function to each task. This adjustment is given by the adaptation step of meta-leaming (Hospedales et al. (2020), which represents a transformation on the parameters w, which is task-dependent and which we refer to as 07 (w))
The loss function metta is defined as an average across both distribution of tasks and data points. The goal of meta-learning is to minimize the loss function with respect to vector of meta- parameters w
mmeta (w)
E E L (0 (w):x²)
T D
2

---

Under review as a conference paper at ICLR 2021
Different meta-learning algorithms correspond to different choice of 0 (w) and allow for each task r to have its own specific loss function. The dependence on T built in through the composition with or (w) makes the loss function a sample from random function field, when considered deterministic function of the data and model parameter w.
3.2 MODEL-AGNOSTIE META- LETA-LEARNING
Our case-study for this paper is the meta-learning algorithm MAML developed in Finn al. (017)) MAML employs base learner, which parametrizes an estimator family by w. The algorithm's adaptation step, inspired from fine-Iuning, performs a fixed number of SGD steps with respect the data for each task. Thus, the adaptation step maps into the same parameter space as w.
In MAML with single gradient step. if we denote the data for task 7 by (az) (z))))) and the SGD learning rate by a, this transformation is equal to:
θ g (w)
(9)
(3)
14
This formula corresponds to full-batch update, employing all the data for task 7(0, but mini- batch gradient updates can be performed as well. During meta-training, loss evaluated on sample of m tasks, and a sample of validation data points n, for each task, leading to the following optimization objective:
mmeoa (w)
L (w):
During meta-testing, new (target) task is given and the parameters 0 are learned by set of target data points following the same equation 3 The final performance the model is computed on test data of the target task.
3.3 DATA ALLOCATION
In this work, we study the problem of finding values of m and T4, such that, under some constraint, the meta-learning loss i minimized given the available data. In this section make the terms allocation and budget explicit. The budgel of a meta-learning problem is defined as the value b = Em 745 i.e. the total number of data points available for meta-training. A meta-training set which respects budget ó is a collection of m tasks sampled independently from the task distribution, each composed of a set of ny data points sampled independently from their respective data- settragnnnaaingg distributions, such that b = Ei= T4:
Conversely, given value of the budget b, and set of tasks y (m)) data-allocation of the budget 6 to these tasks is a partition of b into n, such that = >m n2. If n = n, for all i, this definition is independent of the task samples drawn, and call this the uniform allocation m tasks for the budget b. In this work, we only consider the family of uniform allocations, and we leave the study of non-uniform allocations for future work.
We denote a meta- dataset which respects the uniform allocation with m tasks and n datapoints per task, drawn independently from task distributions T and data distributions Dr by .M (T, Dr; m, n). The optimal uniform data allocation is given by the values of m and n which minimize the expected test meta-loss (2), after optimizing the meta-paie parameter w on meta- trata-rrining set M (T, DF; m, n). Formally. expanding notation to make the dependence of Lmota on M explicit in its arguments:
w' (M (T, DJ m, n)) = arg min Cieta (ws; M (T, (T; n))
(5)
Then the optimal data allocation is
(m" (b), n (b) = arg min E E meta (M'(M M m-n=b D
(6)
Remark. Notice that in equation (6) metta is already an expectation taken over the data distribution which generated the test set. In the definition, this expectation is taken again over the distribution of meta training sets M (T, m, n). It will be useful write Emeta (m, n; T, DT) for the expression
minimized in equation 6
3

---

Under review as a conference paper at ICLR 2021
4 METHODOLOGY
We develop an experimental framework to recover the optimal data allocation (b)()),, (b),,((b)) for a variety of different meta-learning problems.
We consider 3 meta-learning problems in increasing orders of complexity of the data involved:
A synthetic dataset where each task is linear regression problem, and the tasks are parametrized by a multidimensional Gaussian. We analyze this setting theoretically, de- riving the asymptoties of the optimal allocation with respect to the budget and the learning rate a. We then validate our theoretical findings with experiments. The full results are presented in 45.1
A synthetic non non-linear regression dataset, comprised of 1-dimensional sinusoid regression tasks. Each task parametrized by amplitude, phase and noise. We perform synthetic experiments meant to validate A, and report results in section 5.2 We compute the allo- cation curves for multiple budgets, effectively exhausting a portion of the search space for the optimal allocation. We further attempt to recover the optimal allocations by controlling the MAML training regimen via a greedy sequential data allocation. We present results of this method across multiple adaptation step strengths. Details of this procedure in $4.1
The CIFAR-FS dataset introduced in Bertinetto et al. (2019). While few-shot classification is not our main focus, we employ this dataset validate the greedy search procedure in a more realistic context. We rerun the allocation grid search across fewer budgets and compare the results of the data allocation procedure. We repeat the experiments across multiple adaptation step strengths. Full results can be found in 45.
Further details of all computations and experiments are presented in the appendix.
The model family for the final two problems is parametrized by neural network, whereas for the mixed linear regression problem, exact solutions to each optimization problem are given. The meta- learning algorithm is always MAMI.. Our first method for exploring the optimal data allocation problem is to compute ( (m, n; T, DT) as a function of m for various fixed budgets 6. con- junction with this, we investigate the possibility of recovering the optimal allocation sequentially, by making incremental data allocation decisions during MAML training time. We proceed to describe this algorithm.
4.1 SEQUENTIAL DECISION MAKING FOR THE OPTIMAL DATA ALLOCATION
In this section we present the data allocation problem as a sequential decision making (SDM) process and draw parallels Shekhar et al. (2020), Lu et al. (2100). Formally, we treat the learning dynamics of the meta-learning algorithm as a Markov Decision Process (MDP), with states given by tuples (t, (t). M (T, D" m, wt) is the meta-parant parameter learned at time t. An algorithm A is said to perform sequential data allocation if it acts on the state space of the MDP as
mm, n)), where denotes the number of gradient updates performed thus far and
A (I), M (T, DJ; m, n)) = (1 + Al, + AI), (T, DT m + Am, n + An))
where At, An, Am are positive integers, M' is an extension of M by an i.i.d. sampled quantum of data. Implicit in this formula is the dependence of w on t, where after the algorithm A has acted, the parameter w() has been updated by At iterations of training on M' (T, D"; + Am, n + An).
For our implementation of this algorithm treat the problem
contextual bandit with two arms (Lu et al.] (2010)). In each state, the action space admits a fixed At, which in our experiments will be an integer between 2 and 100, and (Am, An) E {(0, 1), (1, O))-
Our procedure for evaluating the payoff of each decision is as follows:
Save the meta- meta- parameter w at the start of the evaluation procedure.
Sample Ko extensions (M to the meta-dataset M and according to the distributions T and D
Compute the loss on each extension after training from the saved meta-parameter w, for Ki meta-updates, and evaluating the loss the meta-validation set Mea!
4

---

Under review as a conference paper at ICLR 2021
0.035
notimaal rntemallumooe number of tankss
301
0.045
0.029
0.030
107
10
10'
10'
10'
10'
10
number tasks
budget
(a)
(b)
Figure 1: The optimal number of tasks increases as the square root the budget in mixed linear regression. a) Average test loss as function of number of tasks for different values of the budget: magenta- 256, cyan 512, green 1024, red 2048. Circles: average over 100 experiments; Lines: theoretical predictions from equation g Diamonds: minima of the loss predicted by theory. b) Optimal number tasks as a function of the budget, the four minima from panel a are replotted here (diamonds). Full line: thcoretical prediction from equation 9 Dashed line: square root law.
Hyperparameter values: p = 32, a = v = 0.2, a 0.3
As Ko, K2 + 0o, the estimation of the payoff from the correct decision converges to the true expected loss of training in the chosen allocation. However, to avoid introducing prohibitively long runtimes, and to respect the budget constraint, in practice, we use small values for Ko and Ki.
5 RESULTS
5.1 SYNTHETIC ANALYSIS: MIXED LINEAR REGRESSION
In mixed linear regression, each task characterized by different linear function, and the loss the mean squared error. The input has dimension p and the output is scalar, therefore the number of parameters of a given task is p. The generative model for the parameters is a multivariate Gaussian distribution with 0 mean and covariance vl, which output has Gaussian noise added, of width a, as in standard linear regression, In the appendix, we calculate the average test loss function of m. b, a, p, V. The number of data per task recovered by the substitution = nm. Our result valid only for a large number of meta-training tasks m. and do derive error bounds on the approximation. We also simplified the final expression by neglecting powers of a higher than two (see the Appendix for details). The result is
refer to as task variability. Furthermore, the
be
I lest
2² [a 2 (
+
m +
o2
P a²pm
P b (
1
1
m
Intuitively, the loss always increases with the output noise and task variability u, and decreases with the total budget of data points b. Our main focus is to find the optimal number of tasks m, for the fixed budget b. We find this optimum by differentiating with respect to m and setting the derivative to zero. However, note that errors the approximation of carry over the calculation of the optimum. The optimal number of tasks is approximated by
1- a
vb
m
p1 (1+
We note that the optimal number of tasks order Vb. Furthermore, tasks are all equal (u - 0). then the optimal choice is to allocate all data a single task (here mn* - 0), because single task is representative all tasks. On the other hand, the optimal number of tasks large when the learning rate is small. In the limit - O, the model ignores the existence of separate tasks and learns single set of parameters for all tasks. Intuitively, the best choice in that case is to see as many tasks possible get better average over tasks.
5

---

Under review as a conference paper at ICLR 2021
65 1000
05-5009
G5 10020
15P
3000
SDK 5000
optinca numbor of ofaads
102
toxt lost
10a1
SEN fit esp: 6.6 son tuck allocations
301
10'
10' 10a
10
10#
101
10
budget
Figure 2: Experimental results on the sinusoid regression dataset. Left: The loss has a unique optimum for the number of tasks, and the optimum increases with the budget size. The SDM al- gorithm recovers the optimum. Full lines show the loss evaluated by grid search, crosses show the corresponding SDM clusters. Grid search curves an average over multiple runs. SDM cluster centroids are marked by diamonds. Right: Power law fits for the SDM cluster centroids, optimal number of tasks function of budget. The exponent of the power law is 0.466, close to the predicted square root law.
Figure shows the test loss function of the number of tasks, for four different values of budget b and the square root law of the optimal number of tasks as a function of the budget. A good agreement observed between theory and experiment. Do these results extend to nonlinear problems? In the next sections we test the following two predictions in nonlinear problems: does the optimal number of tasks scales as the square root of the budget? Does the optimal number of tasks increases when decreasing the learning rate a??
5.2 SYNTHETIC ANALYSIS: SINUSOID REGRESSION
The next experiment involves non-linear regression on I-dimensional sinusoid wave dataset. The task distribution is joint over the distributions of parameters (A, Φ) where each of A, Φ, follow predefined distributions (see Appendix) and serve to parametrize the data (x" "e""y) by:
A sin o)
(10)
MAML is applied to a simple Multi-layer Perceptron base learner, following the architecture Finn al. 2017). Details of the MAML training regime for this experiment can be found in the appendix.
Grid search applied to this problem reveals the loss-allocation landscape. Results for various budgets and allocations depicted in Figure 2 Curves computed after 1e4-2.5e4 meta-updates, the exact numbers depend on the stopping criterion (see Appendix). To reduce variance in the estimates, up to 10 independent runs from uncorrelated initial conditions are performed for each allocation, and each budget. In the same panel we present the final allocation/performaace clusters for multiple runs of the SDM algorithm. The total amount of meta-updates for SDM is commensurate with the grid-search. However, because at the start of training gradient estimates are much noisier, the final performance suffers slightly. Further details of the configuration and training regime for SDM can be found in 6
We observe good agreement between the minima of the grid search and the centroid of the SDM allocation cluster. As predicted, found that the optimal number of tasks increases with the total budget of data points. In Figure we show the optimal number of tasks found by the SDM algorithm function of the budget. As predicted by the theory for linear regression, find power law with an exponent close to 1/2 (0.466).
In Figure we test the SDM algorithm 2 different budgets and 3 different learning rates for the adaptation step MAML. Theoretically, as a - 0. the optimal allocation should contain an increasing number of tasks, because the meta-learning loss reduces the cross-task loss 1 As

---

Under review as a conference paper at ICLR 2021
Budget: 5000
Budget: 10000
100-
10a-
loss
loss
Ir:
10
Ir o-
In n-d
e-5 centroids
Ir. e-6 centroids
50
70
80
90
100
110
120
130
number of tasks
number of tasks
(a)
(b)
The optimal allocation favours more tasks the learning rate a, which controls the strength of the adaptation step, decreases. Correspondingly, performance of the algorithm worsens. Legend: the learning rate values and the coordinate values the corresponding cluster centroid.
Figure 3:
predicted, we observe an increase in the number of tasks recovered by the SDM as a decreases, despite an increase in the variance of the SDM process.
5.3 CIFAR-FS RESULTS
In this section we apply our methods to an image classification problem based on the CIFAR-100 dataset. We use the same train / test validation splits as the CIFAR-FS dataset Bertinetto et al. (2019). but in order to formulate the data allocation problem and give meaningful budget constraints we pre-sample large set of training and test tasks. The classification problem is 5-way, in that each task contains 5 classes. In CIFAR-FS, the number of data points is fixed to either "1-shot" or "5-shots", meaning or 5 data points per class. In our model, The number of shots during training depends the budget allocation. However, we use a large number of data points for adaptation at testing time order to provide minimal variance MC estimate of the meta-loss. Note that this dataset, the same image may appear in multiple tasks, therefore the total budget does not equal the total number of labels.
In Figure 4a we show the behaviour of Lmmet n; T, DT) for b = 6e4 and various values of m, as well as the results of the SDM algorithm for the same budgets. From the grid search curve, we surmise MAML favours tasks in the optimal allocation, effectively placing the optimum of the curve close to the maximal number of tasks, which is b/5 as each task has at least 5 labelled datapoints, one from each class. One possible interpretation of this result is that the model has enough capacity to be able to perfectly cluster 5 classes with very small number of examples for each class. However, future work necessary to shed more light on this observation.
The SDM algorithm is unable to recover the optimal allocation set at this extreme point of the curve partly because the process starts higher number of datapoints per task than the optimum (to reduce variance in the reward estimates) and partly because the SDM process is likely not given sufficient time between decision (At) and payoff estimation time (K1) to match the large budget (see note in the Appendix). For smaller budgets, however, it becomes more clear that the SDM process favours adding tasks rather than data per task.
In Figure 4b we present the results of the SDM allocation across various smaller budgets for the CIFAR-FS data. A power law in this scenario superfluous, due to the fact that at this end of the budget curves, small changes in elicit large changes in However, we present the decision spectrum of the algorithm, which effectively favours adding tasks in proportion of 0.8 to 0.95, as opposed to the same scenario in the sinusoid dataset, where the ratios in between 0.4 and 0.6. Note that the initialization of the SDM in this case is at 50 tasks and 50 datapoints per task (10 points for each class). The final column presents the action spectrum of the process for the two extremal

---

Under review as a conference paper at ICLR 2021
8x10-
Budget
Accuracy
Allocations
Decisions
7x15
10000
0.4363
(200,50)
0/15 0/15
test accuracy
6x13-1
15000
0.4485
(252,60)
0/25 2/16
20000
0.4634
(260,76)
2/23 3/20
5x
30000
0.483
(306,98)
4/28- 6/22
(b) Table of SDM allocations for smaller budgets averaged over multiple runs. Aver- age final accuracy for each run. Allocations start at (50/50): first coordinate is number of tasks, second coordinate number of data- points. The decision column describes the action spectrum for the two extremal SDM allocations at each budget.
4x13
SUN
10 number tasks
30
(a) Test accuracy versus number of tasks for a budget of 6e4. Corresponding SDM final allocation cluster.
Figure 4: Grid search for optimal allocation across a budget of 6e4 on the CIFAR-FS training classes. Various allocations were subjected to multiple runs. We note steep increase in performance as the number of tasks increases followed by plateau and second increase as the number of tasks approaches the maximum. The SDM allocations (orange) cluster at the left hand side of the plateau. Table of SDM allocations for smaller budgets.
runs: the one which most favours the decision (An, 0) and the one which most favours (0, Am). For example, 2/23 signifies that the SDM chose (An, 0) twice and (0, Am) 23 times.
DISCUSSION AND FUTURE WORK
In this paper we have analysed the notion optimal data allocation in meta-learning, by confirming hypothesized properties empirically and proposing methods to efficiently bring meta-learning algo- rithms close to optimal allocation. In particular, we empirically show that the problem of uniform data allocation has unique local minimum for each budget. This optimal number of tasks increases with the budget, and we investigate the rate increase in some specific examples. While it may be that these properties do not hold in general, evidence shows that they do apply in the very simple scenario linear and non- non-linear regression, when applied to MAML, where there is more control over the training dynamics. For the CIFAR-FS dataset, very sparse run of grid search raises ques- tions as to the validity of this assumption, but the variance in the individual meta-training runs is too high to conclusively confirm or disprove this hypothesis.
For the purpose of solving the data allocation problem via sequential decision making, more refined methods can be applied. The SDM algorithm presented in this paper is merely instance of a policy acting on the state space of the MDP. Designing more complex state space which takes into account parameters of the MAML training process such as information about previously seen tasks and the model's 's performance on these tasks, together with augmented action space can lend solutions of the problem to the framework of reinforcement learning. While preliminary experiments have been performed in this direction, stable enough solutions have not been found, and we defer further exploration.
Overall, our studies exemplify the importance of correct data allocation in meta-learning well as give a series of preliminary empirical and theoretical insights on the relation between model performance and data allocation for MAML. While the behaviour of other meta-learnen need not be in any way similar, we surmise that the problem training models close to optimal allocation remains meaningful, and leave much space for empirical study in variety of contexts, well as for the development of a more general theoretical framework.
REFERENCES
Theodore Wilbur Anderson. An introduction to multivariate statistical analysis. Technical report, Wiley New York, 1962.
8

---

Under review as a conference paper at ICLR 2021
Philip Bachman, Alessandro Sordoni, and Adam Trischler. Learning Algorithms for Active Learn ing. arXiv: 7000:0000808888 [cs], July 2017. URL arXiv: 1708.00088.
Luca Bertinetto, João Henriques. Philip H. Torr, and Andrea Vedaldi. Meta-learning with differentiable closed-form solvers. vrXivv/805.068 366 [cs, stat], July 2019. URL http:// arXiv: 1805.08136
arxiv.org/aaa www.i.oor/abb//B/000 08136
Jeff Donahue, Yangqing Jia, Oriol Vinyals, Judy Hoffman, Ning Zhang, Eric Tzeng, and Trevor Dar- rell. DeCAF: A Deep Convolutional Activation Feature for Generic Visual Recognition. ICML,
pp. 9, 2014.
Meng Fang. Jie Yin, Lawrence O. Hall, and Dacheng Tao. Active Multitask Learning With Trace Norm Regularization Based on Excess Risk. IEEE Transactions on Cybernetics, 47(11):3906 3915, November 2017. ISSN 2168- 2267, 2165-2275. doi: 10.1 10.1197TCYB.2226.559000. URL htttp://ieeep) ore. eee /rcaee.rrrrcccmmme///533331
Chelsea Finn, Pieter Abbeel, and Sergey Levine. Model-Agnostic Meta-Learning for Fast Adap- tation Deep Networks. arXiv: aee::/7033444 [cx]. March 2017. URL http:://axxvv.o///
abs/1703 03400
arXiv: 1703.03400.
Chelsea Finn, Kelvin Xu, and Sergey Levine.
Probabilistic Model-Agnostic Meta-Learning.
arXiv:/806.02817 [cs, stat), October 2019.
arXiv: 1806.02817.
Victor Garcia and Joan Bruna. Few-Shot Learning with Graph Neural Networks. arX0v:1711.04443 [cs, stat), February 2018.
URL http:://aaxxv...//aaa/////////// 1101p//11110. 04043
arXiv:
1711.04043.
Abhay Harpale. Multi-Tass Active Learning. PhD thesis, pp. 2012.
2012.
Timothy Hospedales, Antreas Antoniou, Paul Micaelli, and Amos Storkey. Meta-Learning in Neural Networks: Survey. arXiv:2004.0544 [cs, stat], April 2020. URL http:::/axxii..rr/
abs/2004. 05439
arXiv: 2004.05439.
Taesup Kim, Jaesik Yoon, Ousmane Dia, Sungwoong Kim, Yoshua Bengio, and Sungjin Ahn. Bayesian Model-Agnostic Meta-Learnnng, arXiv: 1506:08800883 [cs, stat), November 2018. URL
http:://rxiiii wttp://....ggb/b//aaaeoooo
03836
arXiv: 1806.03836.
Diederik P. Kingma and Jimmy Ba. Adam: A Method for Stochastic Optimization. 1112.6940 arXiv: 1412.6980.
[cs], December 2014. URL http::/ttt/att http:/o///abbaoooeeee 6980
Weihao Kong, Raghav Somani, Zhao Song, Sham Kakade, and Sewoong Oh. Meta-learning for mixed linear regression. arXiv:2002.08893 [cs, stat), February 2020. URL http::/arxai org/abs/20022 08936
arXiv: 2002.08936.
Lianghao Li, Xiaoming Jin, Sinno Jialin Pan, and Jian-Tao Sun. Multi-domain active learning for text classificatio. KDD, pp. 9, 2012.
Tyler Lu, David Pal, and Martin Pal. Contextual multi-armed bandits. 9:485-492, 13-15 May 2010.
Preetum Nakkiran, Gal Kaplun, Yamini Bansal, Tristan Yang, Boaz Barak, and Ilya Sutskever. Deep Double Descent: Where Bigger Models and More Data Hurt. arXiv: 19/2.02292 [cs, stat], De- cember 2019. http:t/t://aaae..
org/ abs 9912 02292 arXiv: 1912.02292.
Anastasia Pentina and Christoph H. Lampert. Multi-Task Learning with Labeled and Unlabeled Tasks. urXiv:1602 06518 [cs, stat), June 2017. URL 06518
arXiv: 1602.06518
version: 2.
Anastasia Pentina, Viktoriaa Sharmanska, and Christoph H. Lampert. Curriculum learning of mul- tiple tasks. In 2015 IEEE Conference Computer Vision and Pattern Recognition (CVPR), pp. 5492-5500, Boston, MA, USA. June 2015. IEEE. ISBN 978-1-4673-6964
doi: 10.1 109/
URL http: httphht:p//eeeeplrreeeeeee wweeeeeepllloo ieee. /ew..e..g///cccmmee////3533

---

Under review as a conference paper at ICLR 2021
Sachin Ravi and Hugo Larochelle. OPTIMIZATION AS MODEL FOR NEW-SHOT LEARN. ING. ICLR, pp.
2017.
Roi Reichart, Katrin Tomanek, Udo Hahn, and Ari Rappoport. Multi-Task Active Learning for Linguistic Annotations. ACL, pp. 2008.
James Requeima, Jonathan Gordon, John Bronskill, Sebastian Nowozin, and Richard Turner. Fast and Flexible Multi-Task Classification Using Conditional Neural Adaptive Processes.
arXiv:/906.000 1906.00697 [es, stat], January 2020. URL
arXiv: 1906.07697.
Paul Ruvolo and Eric Eaton. Active Task Selection for Lifelong Machine Learning. AAAI, pp. 7, 2013.
Avishek Saha, Piyush Rai, Hal Daume li,, and Suresh Venkatasubramannin. Online Learning of Multiple Tasks and Their Relationships. AISTATS, pp. 9, 2011.
Ozan Sener and Silvio Savarese. Active learning for convolutional neural networks: A core-set approach. 2018.
Burr Settles. Active Learning Literature Survey. Technical Report, pp- 67, 2010.
Shubhanshu Shekhar, Tara Javidi, and Mohammad Ghavamzadeh. Adaptive Sampling for Estimat- ing Probability Distributions. ICML, pp. 10, 2020.
Gan Sun, Yang Cong, and Xiaowei Xu. Active Lifelong Learning with "Watchdog". AAAI, pp. 8, 2018.
Vladimir Vapnik. Statistical Learning Theory. Wiley-Interscience, 1998.
Oriol Vinyals, Charles Blundell, Timothy Lillicrap, Koray Kavukcuoglu, and Daan Wierstra. Match- ing Networks for One Shot Learning. arXiv: (6t6:6606.000 04080 [cs, stat), December 2017. URL
http:///arsvv.org/abbb/ 00000000000000000
arXiv: 1606.04080.
Yi Zhang. Multi Task Active Learning with Output Constraints. AAAI, 6, 2010.
Zihan Zhang, Xiaoming Jin, Lianghao Li, Guiguang Ding, and Qiang Yang. Multi-Domain Active Learning for Recommendation. AAAI, pp. 2016.
10


Reviews:
[{"review_id": "XHLpUegZPyb", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "********Summary\n\nIn most popular meta-learning approaches, there are usually a pre-defined number of data per task. For example, 1-shot or 5-shot learning. It has shown as the number of data increases in such methods, model performance improves. In this paper, they try to analyze the effect of having different number of tasks with different number of data points in meta-learning benchmarks. Specifically, given a fixed number of data across different tasks as a budget, they want to see if having a large number of tasks with small data in each works better (or worse) than having small number of tasks with more data in each. They focus on MAML as a meta-learning method, and analyzed the results on mixed linear regression, sinusoid regression, and CIFAR. For mixed linear regression, they showed that the optimal number of tasks is \\sqrt(b). They also provide an online algorithm for finding optimal number of tasks in meta-learning.\n\n********Positives\n- The paper is well-written and well-organized. They explained their approach clearly.\n\n- They tried to answer a very interesting question, that I have always in my mind (and I am sure many other researchers have). It can be generalized to multi-task learning when we have several source tasks for a target task. In multi-tasking there are several sources with various number of data points. The question in this paper can be applied to multi-tasking, but we should consider another important factor, which is task relatedness. Question in that setting might be what is the best combination of tasks with different number of data considering task-relatedness.\n\n********Notes\n- There is one confusing part for me. In abstract, they mentioned that \"Given a fixed budget b of labels to\ndistribute across tasks, should we use a small number of highly labelled tasks, or\nmany tasks with few labels each?\", However, 'uniform data allocation' assumption does not align with the original question they wanted to answer! It seems to me that they answer the optimal number of tasks in meta-learning approaches (MAML specifically) when each task has the same number of data.\n\n\n- I would recommend spending more time on explaining section 4.1. (SEQUENTIAL DECISION MAKING FOR THE OPTIMAL DATA ALLOCATION)\n\n- Where was their theoretical guide as they mentioned in the abstract and introduction, section 5.1? \n\n- Even though the experiments results are really interesting, I would recommend adding one or two more experiments (for example image classification), and also validate their analysis on other meta-learning benchmarks (rather than MAML)\n\n********Reason to accept or reject\n\nThey are pointing to very important problem in meta-learning methods. This problem would be interesting to see for other researcher in this area, and it could be generalized to other related research topics such multi-task learning. However, I think they simplified the problem (by uniform assumption), and they could add more experiments to prove their claim. In general, I liked this work, and it could be useful for others to continue since it's not answered yet (at least as far as I know). I would be happier with more experiments or even other meta-learning approaches. So, my score to this paper is marginally above acceptance threshold.", "extra_scores": {}, "extra_text": {}}, {"review_id": "KfPSNvz5RQA", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper proposes a data allocation scheme for meta-learning. The authors argue that it is important to consider the number of total tasks versus the number of datapoints per task given a fixed budget of the total number of datapoints since labeling is expensive for large datasets. The paper presents an algorithm that does the data allocation using a sequential decision process (SDM) and models the problem as a two-armed contextual bandit where for each fixed number of iterations, the agent can choose to add 1 task or 1 datapoint per task. There are some theoretical results on the optimal data allocation in the linear regression setting. The authors also empirically show that the number of the fixed budget and the data allocation can affect performance fairly a bit in a sinusoid regression dataset and the CIFAR-FS dataset. The proposed SDM algorithm can also recover the optimal data allocation in the sinusoid regression setting.\n\nFor the pros of this paper, I think the idea of better data allocation for meta-learning is novel since, in standard meta-learning settings, we usually assume some predefined number of data per task and number of tasks. I also like the analysis of the data allocation scheme in the linear regression setting, which sheds some light on how we should balance the number of tasks and the number of data per task.\n\nMeanwhile, for the cons of this paper, I can't really see the significance of the problem of data allocation, particularly the data allocation set-up in this paper where each task has the same number of datapoints. Why is the problem important? I don't find the argument about the limited budget in the paper convincing enough since it seems that some naive, hand-designed data allocation schemes (e.g. having as many tasks as possible) work just well as shown in the experiments regardless of the value of the budgets. Is this problem a bit contrived and not very important in the field of meta-learning research? I believe the setting where we do not have uniform data allocation is much more important since some tasks might be much harder to get labels, but this is not explored in detail in this paper.\n\nMoreover, it seems that the initialization of the data allocation of the proposed SDM algorithm matters a lot. If the algorithm is so brittle w.r.t. the initialization, why can't we just do grid search instead? Also, in the CIFAR-FS results, the optimal data allocation discovered by the grid search is simply using as many tasks as possible, which is what people are doing in practice. Such results make me doubt the significance of the problem setting again.\n\nFinally, the paper is not evaluated on some widely used meta-learning benchmarks such as Omniglot, miniImagenet and etc.. Getting more empirical evidence on those benchmarks would be important.\n\nGiven the above comments, I would vote for a reject for this paper for now.", "extra_scores": {}, "extra_text": {}}, {"review_id": "ErkiIc7Bz9w", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The authors study the problem of finding the optimal allocation of labels across training tasks given a fixed budget. The authors mainly want to answer the question that \"if the total number of labels across training tasks is limited, it is better to have a large number of tasks  with very small data in each or a relatively smaller number of highly labeled tasks?\" Although the authors prove that the optimal scaling of the number of tasks in synthetic tasks, the conclusions remain unclear while handling real-world challenging tasks. More specifically, the authors get the result on CIFAR-FS that the optimum allocation is close to the maximal number of tasks, which is weird and not consistent with previous studies in the paper. \n\nAlthough the authors provide a possible interpretation that the model has enough capacity to perfectly cluster 5 classes with a very small number of examples for each class, such an interpretation is not convincing considering that meta-learning models still have a big room to improve. Moreover,  most meta-learning models adopt a similar or even more complex model, the key research question proposed in the paper about optimal allocation between tasks and labels will be less interesting if the proposed interpretation is correct.  \n\nOverall, the question proposed in the paper is interesting and important. I will suggest that the authors should further study the weird outcome on the real-world datasets.\n\n", "extra_scores": {}, "extra_text": {}}, {"review_id": "B_CF_50qPPm", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Summary\n----------\n\nThis paper presents an active learning approach to allocation of labels across tasks in meta-learning. The approach is based on a contextual bandit setting, and the approach is tested on linear regression, sinusoidal regression, and a version of CIFAR. The authors also present a theoretical analysis for the linear setting.\n\nComments\n----------\n\nThe direction considered in this work is interesting, but the paper feels largely incomplete. Overall, the writing is quite unclear throughout, and the presentation of the SDM algorithm/approach is hard to follow. \n\nThe largest shortcoming of the paper is the apparent gap between the the linear results and the nonlinear results. The theoretical analysis for the linear setting is relatively interesting, and the linear experiments appear to match closely. The sinusoidal results also match closely, but this problem can be solved by linear regression on learned nonlinear features, so it is unclear how much of a jump this setting is from the purely linear setting. The proposed approach appears to perform poorly for CIFAR. The authors should include evaluation on standard meta-learning benchmarks such as miniImageNet to show the utility of their approach.\n\nThe presentation of the algorithm is quite unclear, and it is unclear how practical the proposed approach is. Formulating the problem as a sequential decision-making problem is logical. However, given the relatively small number of labeled examples considered in this work, the validity of an assumption of a labeled meta-validation set is somewhat questionable. Indeed, given the sparsity of labeled data, it seems unlikely that a large amount of validation data should be withheld from training. I could not find a discussion of the size of this validation set. Indeed, a much more interesting approach for estimating error would be the prediction error associated with adding a new label, as opposed to an external validation set. Moreover, the authors discuss training for 100 epochs after each decision-making step, which seems prohibitively expensive. \n\nFinally, the discussion of existing literature was quite shallow and should be explored much more deeply. In particular, there are likely many connections in active learning and the design of experiments that are highly relevant to this setting. \n", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper studies the problem of how data should be balanced among a set of tasks within meta-learning. This problem is interesting, and largely hasn't been studied before. However, the reviewers raised several shortcomings of the current version of the paper, including the significance of the problem setting, the limited experimental study (i.e. the only experiment with real data is CIFAR-FS), the depth of the related work section, and the clarity/impreciseness of the writing. Further, the paper has not been revised to address any of these shortcomings. As such, the paper is not ready for publication at ICLR.
Decision: Reject