Theoretical bounds on estimation error for meta-learning

1 INTRODUCTION . Many practical machine learning applications deal with distributional shift from training to testing . One example is few-shot classification ( Ravi & Larochelle , 2016 ; Vinyals et al. , 2016 ) , where new classes need to be learned at test time based on only a few examples for each novel class . Recently , few-shot classification has seen increased success ; however , theoretical properties of this problem remain poorly understood . In this paper we analyze the meta-learning setting , where the learner is given access to samples from a set of meta-training distributions , or tasks . At test-time , the learner is exposed to only a small number of samples from some novel task . The meta-learner aims to uncover a useful inductive bias from the original samples , which allows them to learn a new task more efficiently.1 While some progress has been made towards understanding the generalization performance of specific meta-learning algorithms ( Amit & Meir , 2017 ; Khodak et al. , 2019 ; Bullins et al. , 2019 ; Denevi et al. , 2019 ; Cao et al. , 2019 ) , little is known about the difficulty of the meta-learning problem in general . Existing work has studied generalization upper-bounds for novel data distributions ( Ben-David et al. , 2010 ; Amit & Meir , 2017 ) , yet to our knowledge , the inherent difficulty of these tasks relative to the i.i.d case has not been characterized . In this work , we derive novel bounds for meta learners . We first present a general information theoretic lower bound , Theorem 1 , that we use to derive bounds in particular settings . Using this result , we derive lower bounds in terms of the number of training tasks , data per training task , and data available in a novel target task . Additionally , we provide a specialized analysis for the case where the space of learning tasks is only partially observed , proving that infinite training tasks or data per training task are insufficient to achieve zero minimax risk ( Corollary 2 ) . We then derive upper and lower bounds for a particular meta-learning setting . In recent work , Grant et al . ( 2018 ) recast the popular meta-learning algorithm MAML ( Finn et al. , 2017 ) in terms of inference in a Bayesian hierarchical model . Following this , we provide a theoretical analysis of a hierarchical Bayesian model for meta-linear-regression . We compute sample complexity bounds for posterior inference under Empirical Bayes ( Robbins , 1956 ) in this model and compare them to our predicted lower-bounds in the minimax framework . Furthermore , through asymptotic analysis of the error rate of the MAP estimator , we identify crucial features of the meta-learning environment which are necessary for novel task generalization . 1Note that this definition encompasses few-shot learning . Our primary contributions can be summarized as follows : • We introduce novel lower bounds on minimax risk of parameter estimation in meta-learning . • Through these bounds , we compare the relative utility of samples from meta-training tasks and the novel task and emphasize the importance of the relationship between the tasks . • We provide novel upper bounds on the error rate for estimation in a hierarchical meta-linearregression problem , which we verify through an empirical evaluation . 2 RELATED WORK . An early version of this work ( Lucas et al. , 2019 ) presented a restricted version of Theorem 1 . The current version includes significantly more content , including more general lower bounds and corresponding upper bounds in a hierarchical Bayesian model of meta-learning ( Section 5 ) . Baxter ( 2000 ) introduced a formulation for inductive bias learning where the learner is embedded in an environment of multiple tasks . The learner must find a hypothesis space which enables good generalization on average tasks within the environment , using finite samples . In our setting , the learner is not explicitly tasked with finding a reduced hypothesis space but instead learns using a general two-stage approach , which matches the standard meta-learning paradigm ( Vilalta & Drissi , 2002 ) . In the first stage an inductive bias is extracted from the data , and in the second stage the learner estimates using data from a novel task distribution . Further , we focus on bounding minimax risk of meta learners . Under minimax risk , an optimal learner achieves minimum error on the hardest learning problem in the environment . While average case risk of meta learners is more commonly studied , recent work has turned attention towards the minimax setting ( Kpotufe & Martinet , 2018 ; Hanneke & Kpotufe , 2019 ; 2020 ; Mousavi Kalan et al. , 2020 ; Mehta et al. , 2012 ) . The worst-case error in meta-learning is particularly important in safety-critical systems , for example in medical diagnosis . Mousavi Kalan et al . ( 2020 ) study the minimax risk of transfer learning . In their setting , the learner is provided with a large amount of data from a single source task and is tasked with generalizing to a target task with a limited amount of data . They assume relatedness between tasks by imposing closeness in parameter-space ( whereas in our setting , we assume closeness in distribution via KL divergence ) . They prove only lower bounds , but notably generalize beyond the linear setting towards single layer neural networks . There is a large volume of prior work studying upper-bounds on generalization error in multi-task environments ( Ben-David & Borbely , 2008 ; Ben-David et al. , 2010 ; Pentina & Lampert , 2014 ; Amit & Meir , 2017 ; Mehta et al. , 2012 ) . While the approaches in these works vary , one common factor is the need to characterize task-relatedness . Broadly , these approaches either assume a shared distribution for sampling tasks ( Baxter , 2000 ; Pentina & Lampert , 2014 ; Amit & Meir , 2017 ) , or a measure of distance between distributions ( Ben-David & Borbely , 2008 ; Ben-David et al. , 2010 ; Mohri & Medina , 2012 ) . Our lower-bounds utilize a weak form of task relatedness , assuming that the environment contains a finite set that is suitably separated in parameter space but close in KL divergence—this set of assumptions also arises often when computing i.i.d minimax lower bounds ( Loh , 2017 ) . One practical approach to meta-learning is learning a linear mapping on top of a learned feature space . Prototypical Networks ( Snell et al. , 2017 ) effectively learn a discriminative embedding function and performs linear classification on top using the novel task data . Analyzing these approaches is challenging due to metric-learning inspired objectives ( that require non-i.i.d sampling ) and the simultaneous learning of feature mappings and top-level linear functions . Though some progress has been made ( Jin et al. , 2009 ; Saunshi et al. , 2019 ; Wang et al. , 2019 ; Du et al. , 2020 ) . Maurer ( 2009 ) , for example , explores linear models fitted over a shared linear feature map in a Hilbert space . Our results can be applied in these settings if a suitable packing of the representation space is defined . Other approaches to meta-learning aim to parameterize learning algorithms themselves . Traditionally , this has been achieved by hyper-parameter tuning ( Rasmussen & Nickisch , 2010 ; MacKay et al. , 2019 ) but recent fully parameterized optimizers also show promising performance in deep neural network optimization ( Andrychowicz et al. , 2016 ) , few-shot learning ( Ravi & Larochelle , 2016 ) , unsupervised learning ( Metz et al. , 2019 ) , and reinforcement learning ( Duan et al. , 2016 ) . Yet another approach learns the initialization of task-specific parameters , that are further adapted through regular gradient descent . Model-Agnostic Meta-Learning ( Finn et al. , 2017 ) , or MAML , augments the global parameters with a meta-initialization of the weight parameters . Grant et al . ( 2018 ) recast MAML in terms of inference in a Bayesian hierarchical model . In Section 5 , we consider learning in a hierarchical environment of linear models and provide both lower and upper bounds on the error of estimating the parameters of a novel linear regression problem . Lower bounding estimation error is a critical component of understanding learning problems ( and algorithms ) . Accordingly , there is a large body of literature producing such lower bounds ( Khas ’ minskii , 1979 ; Yang & Barron , 1999 ; Loh , 2017 ) . We focus on producing lower-bounds for parameter estimation using local packing sets , but expect that extending these results to density estimation or non-parametric estimation is feasible . 3 NOVEL TASK ENVIRONMENT RISK . Most existing theoretical work studying out-of-distribution generalization focuses on providing upper-bounds on generalization performance ( Ben-David et al. , 2010 ; Pentina & Lampert , 2014 ; Amit & Meir , 2017 ) . We begin by instead exploring the converse : what is the best performance we can hope to achieve on any given task in the environment ? After introducing notation and minimax risks , we then show how these ideas can be applied , using meta linear regression as an example . A full reference table for notation can be found in Appendix A and a short summary is given here . We consider algorithms that learn in an environment ( Z , P ) , with data domain Z = X × Y and P a space of distributions with support Z . In the typical i.i.d setting , the algorithm is provided training data S ∈ Zk , consisting of k i.i.d samples from P ∈ P . In the standard multi-task setting , we sample training data from a set of training tasks { P1 , . . . , PM+1 } ⊂ P . We extend this to a meta-learning , or novel-task setting by first drawing S1 : M : n training data points from the first M distributions , for a total of nM samples . We call this the meta-training set . We then draw a small sample of novel data , called a support set , SM+1 ∈ Zk , from PM+1 . Consider a symmetric loss function ` ( a , b ) = ψ ( ρ ( a , b ) ) for non-decreasing ψ and arbitrary metric ρ . We seek to estimate the output of θ : P → Ω , a functional that maps distributions to a metric space Ω . For example , θ ( P ) may describe the coefficient vector of a high-dimensional hyperplane when P is a space of linear models , and ρ may be the Euclidean distance . The i.i.d minimax risk Before studying the meta-learning setting , we first begin with a definition of the i.i.d minimax risk that measures the worst-case error of the best possible estimator , R∗ = inf θ̂ sup P∈P ES∼Pk [ ` ( θ̂ ( S ) , θP ) ] . ( 1 ) For notational convenience , we denote the output of θ ( P ) by θP . The estimator for θ is denoted , θ̂ : Zk → Ω , and maps k samples from P to an estimate of θP . Novel-task minimax risk In the novel-task setting , we wish to estimate θPM+1 , the parameters of the novel task distribution PM+1 . We consider two-stage estimators for θPM+1 . In the first stage , the meta-learner uses a learning algorithm f : S1 : M 7→ θ̂S1 : M , that maps the meta-training set to an estimation algorithm , θ̂S1 : M : Zk → Ω . In the second stage , the learner computes θ̂S1 : M ( SM+1 ) , the estimate of θPM+1 . The novel-task minimax risk is given by , R∗P ( β ) = inf f sup P1 , ... , PM+1∈QβP E S1 : M∼Pn1 : M SM+1∼PkM+1 [ ` ( ( f ( S1 : M ) ) ( SM+1 ) , θPM+1 ) ] , ( 2 ) where QβP = { ( P1 , . . . , PM+1 ) ∈ P : DKL ( PM+1‖Pi ) ≤ β , for i = 1 , . . . , M } . This ensures a degree of relatedness between the novel and meta-training tasks . The estimator for θM+1 now depends additionally on theMn samples in S1 : M , where only k Mn samples from PM+1 are available to the learner . Thus , R∗P addresses the domain shift expected at test-time in the meta-learning setting and allows the learner to use data from multiple tasks . The goal of f is to learn an inductive bias from S1 : M such that a good estimate is possible with only k data points from PM+1 . In this setting , k is equivalent to the number of shots in few-shot learning . An example with meta-linear regression We present here a short summary based on meta linear regression , which we will analyze in more detail in Section 5 . In Figure 1 , we show observed data samples from a family of polynomial regression models . Our aim is to output an algorithm which recovers the parameters of a new polynomial function from limited observations–we choose a MAP estimator which is described fully in Section 5 . In the bottom right , we are given only 5 data points from a novel task distribution and estimate the parameters of the model with both the MLE and MAP estimators — the MLE overfits the support set while the MAP estimator is close to the true function . In terms of the terminology used above , the set , P = { pθ ( y ) = N ( x > θ , σ2 ) : θ ∈ Rd , x = [ 1 , x , . . . , xd−1 ] } , is the space of polynomial regression models , parameterized by θ . For this problem , we take ` ( θ̂ , θ ) = ‖θ̂ − θ‖22 . In Figure 1 , tasks are generated with p ( θ ) = N ( τ , σ2θ ) , for unknown , sparse , τ ∈ Rd . Thus , each model is a polynomial function with few large coefficients . The algorithm f , first takes samples from P1 , P2 , P3 and computes an estimate , τ̂ . This estimate of τ is then used to compute θ̂ ( SM+1 ; τ̂ ) = argmaxθ4 p ( θ4|τ̂ , SM+1 ) . Note that this approach is able to learn the correct inductive bias from the data , without requiring a carefully designed regularizer . The lower bounds we derive in Section 4 can be applied to problems of this general type , and the upper and lower bounds in Section 5 apply specifically to meta-learning linear regression .

Summary: The paper studies the information-theoretic lower bounds in the minimax setting of meta-learning. The paper also discusses upper and lower bounds in the hierarchical Bayesian framework of meta linear regression. The novelty of the paper is two-fold: a) it proves a novel meta-learning local packing result to compute the conditional information between training task samples and the novel task data distribution and b) it compares the lower bound of the risk to the risk of posterior estimate in meta linear regression. In addition, the authors verify the dependence of risk on various parameters in 2 different experiments.