Meta-Learning Reliable Priors in the Function Space

1 Introduction . Learning new concepts and skills from a small number of examples as well as adapting them quickly in face of changing circumstances is a key aspect of human intelligence . Unfortunately , our machine learning algorithms lack such adaptive capabilities . Meta-Learning [ 1 , 2 ] has emerged as a promising avenue towards enabling systems to learn much more efficiently by harnessing experience from previous related learning tasks [ 3–8 ] . By meta-learning probabilistic prior beliefs , we not only make more accurate predictions when given a small amount of training data , but also improve the self-assessment machine learning algorithm in the form of epistemic uncertainty estimates [ 9–12 ] . Such uncertainty estimates are critical for sequential decision problems such as Bayesian optimization ( BO ) [ 13 , 14 ] and reinforcement learning [ 15 , 16 ] which require efficient information gathering and exploration . However , in most practical settings , there are only few tasks available for meta-training . Hence we face the risk of overfitting to these few tasks [ 17 ] and , consequently impairing the performance on unseen tasks . To prevent this , recent work has proposed various forms of regularization on the meta-level [ 18 , 12 , 8 ] . While these methods are effective in preventing meta-overfitting for the mean predictions , they fail to do so for the associated uncertainty estimates , manifested in gross overconfidence . Such overconfidence is highly detrimental for downstream sequential decision tasks that rely on calibrated uncertainty estimates [ 19 , 20 ] to perform sufficient exploration . For instance , if the global optimum of the true target function in BO lies outside the model ’ s 95 % confidence bounds the acquisition algorithm may never query points close to the optimum and get stuck in sub-optimal solutions . We hypothesize that previous methods yield overconfident predictions since they do not meta-regularize the predictive distribution directly . Instead , they perform meta-regularization in some latent space , for example the method parameters , which is non-trivially associated with the resulting predictive distribution and thus may not have the indented regularization effects . 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . To overcome the issue of overconfident predictions in meta-learning , we develop a novel approach that regularizes meta-learned priors directly in the function space . We build on the PAC-Bayesian PACOH framework [ 12 ] which uses a hyper-prior over the latent prior parameters for meta-regularization . However , we propose to define the hyper-prior as stochastic process , characterized by its marginal distributions in the function space , and make the associated meta-learning problem tractable by using an approximation of the KL-divergence between stochastic processes [ 21 ] . The functional KL allows us to directly steer the meta-learned prior towards high epistemic uncertainty in regions of insufficient meta-training data and , thus , obtain reliable uncertainty estimates . When instantiating our functional meta-learning framework , referred to as F-PACOH , with Gaussian Processes ( GPs ) , we obtain a simple algorithm that can be seamlessly integrated into sequential decision algorithms . In our experiments , we showcase how F-PACOH can facilitate transfer and life-long learning in the context of BO , and unlike previous meta-learning methods , consistently yields well-calibrated uncertainty estimates . In our benchmark study on meta-learning for BO and hyper-parameter tuning , F-PACOH significantly outperforms all other meta-learners and standard baselines . Finally , we consider lifelong BO , where the meta-BO algorithm faces a sequence of BO tasks and needs build-up prior knowledge iteratively . In this challenging setting , F-PACOH is the only method that is able to significantly improve its optimization performance as it gathers more experience . This paves the way for exciting new applications for meta-learning and transfer such as the recurring re-optimization and calibration of complex machines and systems under changing external conditions . 2 Background . In this section , we formally introduce PAC-Bayesian meta-learning , the foundation of the functional meta-learning framework that we develop in Section 4 . Moreover , we provide a brief description of Bayesian Optimization , which serves as the main testing ground for our proposed method . PAC-Bayesian Meta-Learning . Meta-learning extracts prior knowledge ( i.e. , inductive bias ) from a set of related learning tasks to accelerate inference in light of a new task . In the context of supervised learning , the meta-learner is given n datasets D1 , T1 , ... , Dn , Tn . Each dataset Di , Ti = ( XDi , yDi ) consists of Ti noisy function evaluations yi , t = fi ( xi , t ) + corresponding to a function fi : X 7→ Y and additive noise . In short , we write XDi = ( xi,1 , ... , xi , Ti ) > for the matrix of function inputs and yDi = ( yi,1 , ... , yi , Ti ) > for the vector of corresponding observations . The functions fi ∼ T are sampled from a task distribution T , which can be thought of as a stochastic process that governs a random function f : X 7→ Y . In standard Bayesian inference , we exogenously presume an – often carefully manually designed – prior distribution P ( h ) over learning hypotheses h : X 7→ Y , h ∈ H and combine it with empirical data to form a posterior Q ( h ) = P ( h|D ) . In contrast , in meta-learning we endogenously infer the prior P ( h ) in a data-driven manner , by using the provided meta-training data . While there exist different approaches , we focus on PAC-Bayesian meta-learning [ 22 , 23 , 12 ] due to its principled foundation in statistical learning theory . The PACOH framework by Rothfuss et al . [ 12 ] presumes a loss function l ( h , x , y ) and a parametric family { Pφ|φ ∈ Φ } of priors Pφ ( h ) with hyperparameter space Φ . In a probabilistic setting , one typically uses the negative log-likelihood as the loss function , i.e. , l ( h , x , y ) = − ln p ( y|h ( x ) ) . Given the meta-training data D1 , ... , Dn and a hyper-prior distribution P ( φ ) over Φ , PACOH aims to infer a hyper-posterior distribution Q ( φ ) over the parameters φ of the prior . Using PAC-Bayesian learning theory [ 24 ] , they derive a high-probability bound on the transfer error , i.e. , the generalization error for posterior inference on an unseen task f ∼ T with priors sampled from the hyper-posterior Q ( φ ) . This meta-level generalization bound also serves as an objective for the meta-learner , minimized w.r.t . Q : J ( Q ) = − 1 n n∑ i=1 1 Ti Eφ∼Q [ lnZβ ( Di , Ti , Pφ ) ] + ( 1 λ + 1 nT̃ ) KL [ Q||P ] + const . ( 1 ) Here , λ > 0 , T̃ = ( T−11 + ... + T −1 n ) −1 is the geometric mean of the dataset sizes , lnZ ( Di , Ti , P ) : = ∫ H P ( h ) exp ( − ∑Ti t=1 l ( h , xi , t , yi , t ) ) dh the generalized marginal loglikelihood and KL ( Q||P ) the Kullback–Leibler ( KL ) divergence between hyper-posterior and hyper-prior . To obtain asymptotically consistent bounds , λ is typically chosen as λ = √ n. Bayesian Optimization . Bayesian Optimization ( BO ) aims to find the global maximizer x∗ = arg maxx∈X f ( x ) of a function f : X → R over a bounded domain X . To obtain an estimate of x∗ , the BO algorithm iteratively chooses points x1 , ... , xT ∈ X at which to query f , and observes noisy feedback y1 , ... , yT ∈ R , e.g. , via yt = f ( xt ) + , ∼ N ( 0 , σ2 ) with σ2 ≥ [ 13 , 14 ] . BO methods form a Bayesian surrogate model of the function f based on previous observations Dt = { ( xt , yt ) } . Typically , a Gaussian Process ( GP ) GP ( m ( x ) , k ( x , x′ ) ) with mean m ( x ) and kernel function k ( x , x′ ) is employed to form a posterior belief p ( f ( x ) |Dt ) over function values [ 25 ] . In each iteration , the next query point xt+1 : = arg maxx∈X αt ( x ) is chosen as maximizer of an acquisition function αt ( x ) that is typically based on the p ( f ( x ) |Dt ) and trades-off exploration and exploitation [ 26–29 ] . 3 Related Work . Meta-Learning . Common approaches in meta-learning attempt to learn a shared embedding space [ 3 , 30–32 ] amortize inference by a meta-learned recurrent model [ 33–35 ] or learn the initialization of a NN so it can be quickly adapted to new tasks [ 6 , 7 , 36 ] . A growing body of work also uses probabilistic modeling to enable uncertainty quantification in meta-learning [ 9 , 11 , 37 ] . However , when only given a small number of meta-training tasks such approaches tend to overfit and fail to provide reliable uncertainty estimates . The problem of overfitting to the meta-training tasks has been brought to attention by [ 17 , 38 ] , followed by work that discusses potential solutions in the form of meta-regularization [ 8 , 12 , 18 ] . Our meta-learning framework builds on a sequence of work on PAC-Bayesian meta-learning [ 22 , 23 , 12 ] . However , instead of using a hyper-prior over the latent parameters of the learnable prior , we define the hyper-prior as stochastic process in the function space which gives us better control over the behavior of the predictive distributions of our meta-learned model in the absence of data . Learning hypotheses in the function space . Recent work bridges the gap between stochastic processes and complex parametric models by developing a variational inference in the function space [ 21 , 39 ] . Our framework in the function space heavily builds on these ideas ; in particular , the proposed approximations of the KL-divergence between stochastic processes [ 21 , 40 , 41 ] . Meta-Learning and Transfer for Bayesian Optimization . To improve the sample efficiency of BO based on previous experience , a basic approach is to warm-start BO by initially evaluating configurations that have performed well on similar tasks in the past [ 42–44 ] . Another common theme is to form a global GP based model across tasks [ 45–48 ] . However , such global GPs quickly become computationally infeasible as they scale cubically in the number of tasks times the number samples per task or require hand-designed task features . Conceptually , the most similar to our work is the approach of learning a neural-network based feature map that is shared across tasks and used for BO based on a Bayesian linear regression model in the shared feature space [ 49 , 50 ] . While these methods are highly scalable , they lack any form meta-level regularization and thus , in face of data-scarcity , are prone to overfitting and over-overconfident uncertainty estimates . In contrast , our method overcomes these issues thanks to its principled meta-regularization in the function space .

Summary: This work introdcues a new way to regularize priors in meta-learning through the use of stochastic processes as hyper-priors. The authors rely on sampling-based measurements to estiamte the KL divergence between the meta-learned prior and a stochastic process hyper-prior.   Through experiments on toy datasets, the algorithm, F-PACOH, demonstrates principled uncertainty estimates. It provides higher levels of uncertainty far from training data, which is in contrast to previous methods for imposing meta-learning priors and hyper-priors. Finally, the authors apply their method to a Bayesian Optimization meta-learning problem and demonstrate decreased regret compared to other methods benchmarked.