Theoretical bounds on estimation error for meta-learning

1 INTRODUCTION . Many practical machine learning applications deal with distributional shift from training to testing . One example is few-shot classification ( Ravi & Larochelle , 2016 ; Vinyals et al. , 2016 ) , where new classes need to be learned at test time based on only a few examples for each novel class . Recently , few-shot classification has seen increased success ; however , theoretical properties of this problem remain poorly understood . In this paper we analyze the meta-learning setting , where the learner is given access to samples from a set of meta-training distributions , or tasks . At test-time , the learner is exposed to only a small number of samples from some novel task . The meta-learner aims to uncover a useful inductive bias from the original samples , which allows them to learn a new task more efficiently.1 While some progress has been made towards understanding the generalization performance of specific meta-learning algorithms ( Amit & Meir , 2017 ; Khodak et al. , 2019 ; Bullins et al. , 2019 ; Denevi et al. , 2019 ; Cao et al. , 2019 ) , little is known about the difficulty of the meta-learning problem in general . Existing work has studied generalization upper-bounds for novel data distributions ( Ben-David et al. , 2010 ; Amit & Meir , 2017 ) , yet to our knowledge , the inherent difficulty of these tasks relative to the i.i.d case has not been characterized . In this work , we derive novel bounds for meta learners . We first present a general information theoretic lower bound , Theorem 1 , that we use to derive bounds in particular settings . Using this result , we derive lower bounds in terms of the number of training tasks , data per training task , and data available in a novel target task . Additionally , we provide a specialized analysis for the case where the space of learning tasks is only partially observed , proving that infinite training tasks or data per training task are insufficient to achieve zero minimax risk ( Corollary 2 ) . We then derive upper and lower bounds for a particular meta-learning setting . In recent work , Grant et al . ( 2018 ) recast the popular meta-learning algorithm MAML ( Finn et al. , 2017 ) in terms of inference in a Bayesian hierarchical model . Following this , we provide a theoretical analysis of a hierarchical Bayesian model for meta-linear-regression . We compute sample complexity bounds for posterior inference under Empirical Bayes ( Robbins , 1956 ) in this model and compare them to our predicted lower-bounds in the minimax framework . Furthermore , through asymptotic analysis of the error rate of the MAP estimator , we identify crucial features of the meta-learning environment which are necessary for novel task generalization . 1Note that this definition encompasses few-shot learning . Our primary contributions can be summarized as follows : • We introduce novel lower bounds on minimax risk of parameter estimation in meta-learning . • Through these bounds , we compare the relative utility of samples from meta-training tasks and the novel task and emphasize the importance of the relationship between the tasks . • We provide novel upper bounds on the error rate for estimation in a hierarchical meta-linearregression problem , which we verify through an empirical evaluation . 2 RELATED WORK . An early version of this work ( Lucas et al. , 2019 ) presented a restricted version of Theorem 1 . The current version includes significantly more content , including more general lower bounds and corresponding upper bounds in a hierarchical Bayesian model of meta-learning ( Section 5 ) . Baxter ( 2000 ) introduced a formulation for inductive bias learning where the learner is embedded in an environment of multiple tasks . The learner must find a hypothesis space which enables good generalization on average tasks within the environment , using finite samples . In our setting , the learner is not explicitly tasked with finding a reduced hypothesis space but instead learns using a general two-stage approach , which matches the standard meta-learning paradigm ( Vilalta & Drissi , 2002 ) . In the first stage an inductive bias is extracted from the data , and in the second stage the learner estimates using data from a novel task distribution . Further , we focus on bounding minimax risk of meta learners . Under minimax risk , an optimal learner achieves minimum error on the hardest learning problem in the environment . While average case risk of meta learners is more commonly studied , recent work has turned attention towards the minimax setting ( Kpotufe & Martinet , 2018 ; Hanneke & Kpotufe , 2019 ; 2020 ; Mousavi Kalan et al. , 2020 ; Mehta et al. , 2012 ) . The worst-case error in meta-learning is particularly important in safety-critical systems , for example in medical diagnosis . Mousavi Kalan et al . ( 2020 ) study the minimax risk of transfer learning . In their setting , the learner is provided with a large amount of data from a single source task and is tasked with generalizing to a target task with a limited amount of data . They assume relatedness between tasks by imposing closeness in parameter-space ( whereas in our setting , we assume closeness in distribution via KL divergence ) . They prove only lower bounds , but notably generalize beyond the linear setting towards single layer neural networks . There is a large volume of prior work studying upper-bounds on generalization error in multi-task environments ( Ben-David & Borbely , 2008 ; Ben-David et al. , 2010 ; Pentina & Lampert , 2014 ; Amit & Meir , 2017 ; Mehta et al. , 2012 ) . While the approaches in these works vary , one common factor is the need to characterize task-relatedness . Broadly , these approaches either assume a shared distribution for sampling tasks ( Baxter , 2000 ; Pentina & Lampert , 2014 ; Amit & Meir , 2017 ) , or a measure of distance between distributions ( Ben-David & Borbely , 2008 ; Ben-David et al. , 2010 ; Mohri & Medina , 2012 ) . Our lower-bounds utilize a weak form of task relatedness , assuming that the environment contains a finite set that is suitably separated in parameter space but close in KL divergence—this set of assumptions also arises often when computing i.i.d minimax lower bounds ( Loh , 2017 ) . One practical approach to meta-learning is learning a linear mapping on top of a learned feature space . Prototypical Networks ( Snell et al. , 2017 ) effectively learn a discriminative embedding function and performs linear classification on top using the novel task data . Analyzing these approaches is challenging due to metric-learning inspired objectives ( that require non-i.i.d sampling ) and the simultaneous learning of feature mappings and top-level linear functions . Though some progress has been made ( Jin et al. , 2009 ; Saunshi et al. , 2019 ; Wang et al. , 2019 ; Du et al. , 2020 ) . Maurer ( 2009 ) , for example , explores linear models fitted over a shared linear feature map in a Hilbert space . Our results can be applied in these settings if a suitable packing of the representation space is defined . Other approaches to meta-learning aim to parameterize learning algorithms themselves . Traditionally , this has been achieved by hyper-parameter tuning ( Rasmussen & Nickisch , 2010 ; MacKay et al. , 2019 ) but recent fully parameterized optimizers also show promising performance in deep neural network optimization ( Andrychowicz et al. , 2016 ) , few-shot learning ( Ravi & Larochelle , 2016 ) , unsupervised learning ( Metz et al. , 2019 ) , and reinforcement learning ( Duan et al. , 2016 ) . Yet another approach learns the initialization of task-specific parameters , that are further adapted through regular gradient descent . Model-Agnostic Meta-Learning ( Finn et al. , 2017 ) , or MAML , augments the global parameters with a meta-initialization of the weight parameters . Grant et al . ( 2018 ) recast MAML in terms of inference in a Bayesian hierarchical model . In Section 5 , we consider learning in a hierarchical environment of linear models and provide both lower and upper bounds on the error of estimating the parameters of a novel linear regression problem . Lower bounding estimation error is a critical component of understanding learning problems ( and algorithms ) . Accordingly , there is a large body of literature producing such lower bounds ( Khas ’ minskii , 1979 ; Yang & Barron , 1999 ; Loh , 2017 ) . We focus on producing lower-bounds for parameter estimation using local packing sets , but expect that extending these results to density estimation or non-parametric estimation is feasible . 3 NOVEL TASK ENVIRONMENT RISK . Most existing theoretical work studying out-of-distribution generalization focuses on providing upper-bounds on generalization performance ( Ben-David et al. , 2010 ; Pentina & Lampert , 2014 ; Amit & Meir , 2017 ) . We begin by instead exploring the converse : what is the best performance we can hope to achieve on any given task in the environment ? After introducing notation and minimax risks , we then show how these ideas can be applied , using meta linear regression as an example . A full reference table for notation can be found in Appendix A and a short summary is given here . We consider algorithms that learn in an environment ( Z , P ) , with data domain Z = X × Y and P a space of distributions with support Z . In the typical i.i.d setting , the algorithm is provided training data S ∈ Zk , consisting of k i.i.d samples from P ∈ P . In the standard multi-task setting , we sample training data from a set of training tasks { P1 , . . . , PM+1 } ⊂ P . We extend this to a meta-learning , or novel-task setting by first drawing S1 : M : n training data points from the first M distributions , for a total of nM samples . We call this the meta-training set . We then draw a small sample of novel data , called a support set , SM+1 ∈ Zk , from PM+1 . Consider a symmetric loss function ` ( a , b ) = ψ ( ρ ( a , b ) ) for non-decreasing ψ and arbitrary metric ρ . We seek to estimate the output of θ : P → Ω , a functional that maps distributions to a metric space Ω . For example , θ ( P ) may describe the coefficient vector of a high-dimensional hyperplane when P is a space of linear models , and ρ may be the Euclidean distance . The i.i.d minimax risk Before studying the meta-learning setting , we first begin with a definition of the i.i.d minimax risk that measures the worst-case error of the best possible estimator , R∗ = inf θ̂ sup P∈P ES∼Pk [ ` ( θ̂ ( S ) , θP ) ] . ( 1 ) For notational convenience , we denote the output of θ ( P ) by θP . The estimator for θ is denoted , θ̂ : Zk → Ω , and maps k samples from P to an estimate of θP . Novel-task minimax risk In the novel-task setting , we wish to estimate θPM+1 , the parameters of the novel task distribution PM+1 . We consider two-stage estimators for θPM+1 . In the first stage , the meta-learner uses a learning algorithm f : S1 : M 7→ θ̂S1 : M , that maps the meta-training set to an estimation algorithm , θ̂S1 : M : Zk → Ω . In the second stage , the learner computes θ̂S1 : M ( SM+1 ) , the estimate of θPM+1 . The novel-task minimax risk is given by , R∗P ( β ) = inf f sup P1 , ... , PM+1∈QβP E S1 : M∼Pn1 : M SM+1∼PkM+1 [ ` ( ( f ( S1 : M ) ) ( SM+1 ) , θPM+1 ) ] , ( 2 ) where QβP = { ( P1 , . . . , PM+1 ) ∈ P : DKL ( PM+1‖Pi ) ≤ β , for i = 1 , . . . , M } . This ensures a degree of relatedness between the novel and meta-training tasks . The estimator for θM+1 now depends additionally on theMn samples in S1 : M , where only k Mn samples from PM+1 are available to the learner . Thus , R∗P addresses the domain shift expected at test-time in the meta-learning setting and allows the learner to use data from multiple tasks . The goal of f is to learn an inductive bias from S1 : M such that a good estimate is possible with only k data points from PM+1 . In this setting , k is equivalent to the number of shots in few-shot learning . An example with meta-linear regression We present here a short summary based on meta linear regression , which we will analyze in more detail in Section 5 . In Figure 1 , we show observed data samples from a family of polynomial regression models . Our aim is to output an algorithm which recovers the parameters of a new polynomial function from limited observations–we choose a MAP estimator which is described fully in Section 5 . In the bottom right , we are given only 5 data points from a novel task distribution and estimate the parameters of the model with both the MLE and MAP estimators — the MLE overfits the support set while the MAP estimator is close to the true function . In terms of the terminology used above , the set , P = { pθ ( y ) = N ( x > θ , σ2 ) : θ ∈ Rd , x = [ 1 , x , . . . , xd−1 ] } , is the space of polynomial regression models , parameterized by θ . For this problem , we take ` ( θ̂ , θ ) = ‖θ̂ − θ‖22 . In Figure 1 , tasks are generated with p ( θ ) = N ( τ , σ2θ ) , for unknown , sparse , τ ∈ Rd . Thus , each model is a polynomial function with few large coefficients . The algorithm f , first takes samples from P1 , P2 , P3 and computes an estimate , τ̂ . This estimate of τ is then used to compute θ̂ ( SM+1 ; τ̂ ) = argmaxθ4 p ( θ4|τ̂ , SM+1 ) . Note that this approach is able to learn the correct inductive bias from the data , without requiring a carefully designed regularizer . The lower bounds we derive in Section 4 can be applied to problems of this general type , and the upper and lower bounds in Section 5 apply specifically to meta-learning linear regression .

Summary: This paper provides a minimax novel-task risk lower bound for meta learning via information-theoretical techniques, showing the fundamental limits of meta learning. The novel-task minimax risk depends on the number of samples from the meta-training set and novel task, as well as the task similarity. The authors further investigate the meta learning problem on a hierarchical Bayesian model, discuss the lower bound and upper bound with maximum-a-posterior estimator.