Optimal allocation of data across training tasks in meta-learning

Meta-learning models transfer the knowledge acquired from previous tasks to quickly learn new ones . They are tested on benchmarks with a fixed number of data-points for each training task , and this number is usually arbitrary , for example , 5 instances per class in few-shot classification . It is unknown how the performance of meta-learning is affected by the distribution of data across training tasks . Since labelling of data is expensive , finding the optimal allocation of labels across training tasks may reduce costs . Given a fixed budget b of labels to distribute across tasks , should we use a small number of highly labelled tasks , or many tasks with few labels each ? In MAML applied to mixed linear regression , we prove that the optimal number of tasks follows the scaling law √ b . We develop an online algorithm for data allocation across tasks , and show that the same scaling law applies to nonlinear regression . We also show preliminary experiments on few-shot image classification . Our work provides a theoretical guide for allocating labels across tasks in meta-learning , which we believe will prove useful in a large number of applications . 1 INTRODUCTION . Deep learning ( DL ) models require a large amount of data in order to perform well , when trained from scratch , but labeling data is expensive and time consuming . An effective approach to avoid the costs of collecting and labeling large amount of data is transfer learning : train a model on one big dataset , or a few related datasets that are already available , and then fine-tune the model on the target dataset , which can be of much smaller size ( Donahue et al . ( 2014 ) ) . In this context , there has been a recent surge of interest in the field of meta-learning , which is inspired by the ability of humans to learn how to learn Hospedales et al . ( 2020 ) . A model is meta-trained on a large number of tasks , each characterized by a small dataset , and meta-tested on the target dataset . The number of data points per task is usually set to an arbitrary number in standard meta-learning benchmarks . For example , in few-shot image classification benchmarks , such as mini-ImageNet ( Vinyals et al . ( 2017 ) , Ravi & Larochelle ( 2017 ) ) and CIFAR-FS ( Bertinetto et al . ( 2019 ) ) , this number is usually set to 1 or 5 . So far , there has not been any reason to optimize this number , as in most circumstances the performance of a model will improve with the number of data points ( see Nakkiran et al . ( 2019 ) for exceptions ) . However , if the total number of labels across training tasks is limited , is it better to have a large number of tasks with very small data in each , or a relatively smaller number of highly labelled tasks ? Since data-labeling is costly , the answer to this question may inform the design of new meta-learning datasets and benchmarks . In this work , to our knowledge , we answer this question for the first time , for a specific meta-learning algorithm : MAML ( Finn et al . ( 2017 ) ) . We study the problem of optimizing the number of metatraining tasks , with a fixed budget b of total data-points to distribute across tasks . We study the application of MAML to three datasets : mixed linear regression , sinusoid regression , and CIFAR . In the case of mixed linear regression , we derive an approximation for the meta-test loss , and according to which the optimal number of tasks follows the scaling rule √ b . In order to optimize the number of tasks empirically , we design an algorithm for online allocation of data across training tasks , and we validate the algorithm by performing a grid search over a large set of possible allocations . In summary , our contributions are : • We introduce and formalize the problem of optimizing data allocation with a fixed budget b in meta-learning . • We prove that the optimal scaling of the number of tasks is √ b in mixed linear regression , and confirm this scaling empirically in nolinear regression . • We introduce an algorithm for online allocation of data across tasks , to find the optimal number of tasks during meta-training , and validate the algorithm by grid search . • We perform preliminary experiments on few-shot image classification . 2 RELATED WORK . A couple of recent papers investigated a problem similar to ours . In the context of meta-learning and mixed linear regression , Kong et al . ( 2020 ) asks whether many tasks with small data can compensate for a lack of tasks with big data . However , they do not address the problem of finding the optimal number of tasks within a fixed budget . The work of Shekhar et al . ( 2020 ) studies exactly the problem of allocating a fixed budget of data points , but to the problem of estimating a finite set of discrete distributions , therefore they do not study the meta-learning problem and their data has no labels . An alternative approach to avoid labelling a large amount of data is active learning , where a model learns with fewer labels by accurately selecting which data to learn from ( Settles ( 2010 ) ) . In the context of meta-learning , the option of implementing active learning has been considered in a few recent studies ( Bachman et al . ( 2017 ) , Garcia & Bruna ( 2018 ) , Kim et al . ( 2018 ) , Finn et al . ( 2019 ) , Requeima et al . ( 2020 ) ) . However , they considered the active labeling of data within a given task , for the purpose of improving performance in that task only . Instead , we ask how data should be distributed across tasks . In the context of recommender systems and text classification , a few studies considered whether labeling a data point , within a given task , may increase performance not only in that task but also in all other tasks . This problem has been referred to as multi-task active learning ( Reichart et al . ( 2008 ) , Zhang ( 2010 ) , Saha et al . ( 2011 ) , Harpale ( 2012 ) , Fang et al . ( 2017 ) ) , or multi-domain active learning ( Li et al . ( 2012 ) , Zhang et al . ( 2016 ) ) . However , none of these studies consider the problem of meta-learning with a fixed budget . A few studies have looked into actively choosing the next task in a sequence of tasks ( Ruvolo & Eaton ( 2013 ) , Pentina et al . ( 2015 ) , Pentina & Lampert ( 2017 ) , Sun et al . ( 2018 ) ) , but they do not look at how to distribute data across tasks . 3 THE PROBLEM OF DATA ALLOCATION FOR META-LEARNING . In the cross-task setting , we are presented with a hierarchically structured dataset , with task parameters ( τ ( i ) ) mi=1 sampled from T ∼ p ( τ ) and data ( xτj ) nτ j=1 sampled from Dτ : = ( D|T ) ∼ p ( x|T = τ ) . Our problem is minimizing the following loss function with respect to a parameter ω : L ( ω ) = E T E Dτ L ( ω ; xτ ) ( 1 ) The empirical risk minimization principle ( see Vapnik ( 1998 ) ) ensures that the optimum of the empirical risk converges to that of the true risk with an increase in samples from the joint distribution of ( D , T ) . 3.1 META-LEARNING ACROSS TASKS . In the meta-learning problem , we are given the opportunity to adjust the objective function to each task . This adjustment is given by the adaptation step of meta-learning ( Hospedales et al . ( 2020 ) ) , which represents a transformation on the parameters ω , which is task-dependent and which we refer to as θτ ( ω ) . The loss function Lmeta is defined as an average across both distribution of tasks and data points . The goal of meta-learning is to minimize the loss function with respect to a vector of metaparameters ω Lmeta ( ω ) = E T E Dτ Lτ ( θτ ( ω ) ; xτ ) ( 2 ) Different meta-learning algorithms correspond to a different choice of θτ ( ω ) and we allow for each task τ to have its own specific loss function . The dependence on τ built in through the composition with θτ ( ω ) makes the loss function a sample from a random function field , when considered as a deterministic function of the data and model parameter ω . 3.2 MODEL-AGNOSTIC META-LEARNING . Our case-study for this paper is the meta-learning algorithm MAML developed in Finn et al . ( 2017 ) . MAML employs a base learner , which parametrizes an estimator family by ω . The algorithm ’ s adaptation step , inspired from fine-tuning , performs a fixed number of SGD steps with respect to the data for each task . Thus , the adaptation step maps into the same parameter space as ω . In MAML with a single gradient step , if we denote the data for task τ ( i ) by ( x ( i ) j ) ni j=1 , and the SGD learning rate by α , this transformation is equal to : θ ( i ) ( ω ) = ω − α ni ni∑ j=1 ∇ωL|ω ; x ( i ) j ( 3 ) This formula corresponds to a full-batch update , employing all the data for task τ ( i ) , but minibatch gradient updates can be performed as well . During meta-training , the loss is evaluated on a sample of m tasks , and a sample of validation data points ni for each task , leading to the following optimization objective : Lmeta ( ω ) = 1 m m∑ i=1 1 ni ni∑ j=1 L ( θ ( i ) ( ω ) ; x ( i ) j ) ( 4 ) During meta-testing , a new ( target ) task is given and the parameters θ are learned by a set of target data points following the same equation 3 . The final performance of the model is computed on test data of the target task . 3.3 DATA ALLOCATION . In this work , we study the problem of finding values of m and ni , such that , under some constraint , the meta-learning loss 2 is minimized given the available data . In this section we make the terms allocation and budget explicit . The budget of a meta-learning problem is defined as the value b =∑m i=1 ni , i.e . the total number of data points available for meta-training . A meta-training set which respects a budget b is a collection of m tasks sampled independently from the task distribution , each composed of a set of ni data points sampled independently from their respective data-generating distributions , such that b = ∑m i=1 ni . Conversely , given a value of the budget b , and a set of tasks τ ( 1 ) , ... τ ( m ) , a data-allocation of the budget b to these tasks is a partition of b into ni such that b = ∑m i=1 ni . If n = ni for all i , this definition is independent of the task samples drawn , and we call this the uniform allocation on m tasks for the budget b . In this work , we only consider the family of uniform allocations , and we leave the study of non-uniform allocations for future work . We denote a meta-dataset which respects the uniform allocation with m tasks and n datapoints per task , drawn independently from task distributions T and data distributionsDτ byM ( T , Dτ ; m , n ) . The optimal uniform data allocation is given by the values ofm and n which minimize the expected test meta-loss ( 2 ) , after optimizing the meta-parameter ω on a meta-training setM ( T , Dτ ; m , n ) . Formally , expanding notation to make the dependence of Lmeta onM explicit in its arguments : ω∗ ( M ( T , Dτ ; m , n ) ) = arg min ω Lmeta ( ω ; M ( T , Dτ ; m , n ) ) ( 5 ) Then the optimal data allocation is ( m∗ ( b ) , n∗ ( b ) ) = arg min m·n=b E T E Dτ Lmeta ( ω∗ ( M ( T , Dτ ; m , n ) ) ; Mtest ) ( 6 ) Remark . Notice that in equation ( 6 ) , Lmeta is already an expectation taken over the data distribution which generated the test set . In the definition , this expectation is taken again over the distribution of meta training setsM ( T , Dτ ; m , n ) . It will be useful to writeLmeta ( m , n ; T , Dτ ) for the expression minimized in equation 6 .

Summary: The authors study the problem of finding the optimal allocation of labels across training tasks given a fixed budget. The authors mainly want to answer the question that "if the total number of labels across training tasks is limited, it is better to have a large number of tasks  with very small data in each or a relatively smaller number of highly labeled tasks?" Although the authors prove that the optimal scaling of the number of tasks in synthetic tasks, the conclusions remain unclear while handling real-world challenging tasks. More specifically, the authors get the result on CIFAR-FS that the optimum allocation is close to the maximal number of tasks, which is weird and not consistent with previous studies in the paper.