Meta-Learning Reliable Priors in the Function Space

1 Introduction . Learning new concepts and skills from a small number of examples as well as adapting them quickly in face of changing circumstances is a key aspect of human intelligence . Unfortunately , our machine learning algorithms lack such adaptive capabilities . Meta-Learning [ 1 , 2 ] has emerged as a promising avenue towards enabling systems to learn much more efficiently by harnessing experience from previous related learning tasks [ 3–8 ] . By meta-learning probabilistic prior beliefs , we not only make more accurate predictions when given a small amount of training data , but also improve the self-assessment machine learning algorithm in the form of epistemic uncertainty estimates [ 9–12 ] . Such uncertainty estimates are critical for sequential decision problems such as Bayesian optimization ( BO ) [ 13 , 14 ] and reinforcement learning [ 15 , 16 ] which require efficient information gathering and exploration . However , in most practical settings , there are only few tasks available for meta-training . Hence we face the risk of overfitting to these few tasks [ 17 ] and , consequently impairing the performance on unseen tasks . To prevent this , recent work has proposed various forms of regularization on the meta-level [ 18 , 12 , 8 ] . While these methods are effective in preventing meta-overfitting for the mean predictions , they fail to do so for the associated uncertainty estimates , manifested in gross overconfidence . Such overconfidence is highly detrimental for downstream sequential decision tasks that rely on calibrated uncertainty estimates [ 19 , 20 ] to perform sufficient exploration . For instance , if the global optimum of the true target function in BO lies outside the model ’ s 95 % confidence bounds the acquisition algorithm may never query points close to the optimum and get stuck in sub-optimal solutions . We hypothesize that previous methods yield overconfident predictions since they do not meta-regularize the predictive distribution directly . Instead , they perform meta-regularization in some latent space , for example the method parameters , which is non-trivially associated with the resulting predictive distribution and thus may not have the indented regularization effects . 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . To overcome the issue of overconfident predictions in meta-learning , we develop a novel approach that regularizes meta-learned priors directly in the function space . We build on the PAC-Bayesian PACOH framework [ 12 ] which uses a hyper-prior over the latent prior parameters for meta-regularization . However , we propose to define the hyper-prior as stochastic process , characterized by its marginal distributions in the function space , and make the associated meta-learning problem tractable by using an approximation of the KL-divergence between stochastic processes [ 21 ] . The functional KL allows us to directly steer the meta-learned prior towards high epistemic uncertainty in regions of insufficient meta-training data and , thus , obtain reliable uncertainty estimates . When instantiating our functional meta-learning framework , referred to as F-PACOH , with Gaussian Processes ( GPs ) , we obtain a simple algorithm that can be seamlessly integrated into sequential decision algorithms . In our experiments , we showcase how F-PACOH can facilitate transfer and life-long learning in the context of BO , and unlike previous meta-learning methods , consistently yields well-calibrated uncertainty estimates . In our benchmark study on meta-learning for BO and hyper-parameter tuning , F-PACOH significantly outperforms all other meta-learners and standard baselines . Finally , we consider lifelong BO , where the meta-BO algorithm faces a sequence of BO tasks and needs build-up prior knowledge iteratively . In this challenging setting , F-PACOH is the only method that is able to significantly improve its optimization performance as it gathers more experience . This paves the way for exciting new applications for meta-learning and transfer such as the recurring re-optimization and calibration of complex machines and systems under changing external conditions . 2 Background . In this section , we formally introduce PAC-Bayesian meta-learning , the foundation of the functional meta-learning framework that we develop in Section 4 . Moreover , we provide a brief description of Bayesian Optimization , which serves as the main testing ground for our proposed method . PAC-Bayesian Meta-Learning . Meta-learning extracts prior knowledge ( i.e. , inductive bias ) from a set of related learning tasks to accelerate inference in light of a new task . In the context of supervised learning , the meta-learner is given n datasets D1 , T1 , ... , Dn , Tn . Each dataset Di , Ti = ( XDi , yDi ) consists of Ti noisy function evaluations yi , t = fi ( xi , t ) + corresponding to a function fi : X 7→ Y and additive noise . In short , we write XDi = ( xi,1 , ... , xi , Ti ) > for the matrix of function inputs and yDi = ( yi,1 , ... , yi , Ti ) > for the vector of corresponding observations . The functions fi ∼ T are sampled from a task distribution T , which can be thought of as a stochastic process that governs a random function f : X 7→ Y . In standard Bayesian inference , we exogenously presume an – often carefully manually designed – prior distribution P ( h ) over learning hypotheses h : X 7→ Y , h ∈ H and combine it with empirical data to form a posterior Q ( h ) = P ( h|D ) . In contrast , in meta-learning we endogenously infer the prior P ( h ) in a data-driven manner , by using the provided meta-training data . While there exist different approaches , we focus on PAC-Bayesian meta-learning [ 22 , 23 , 12 ] due to its principled foundation in statistical learning theory . The PACOH framework by Rothfuss et al . [ 12 ] presumes a loss function l ( h , x , y ) and a parametric family { Pφ|φ ∈ Φ } of priors Pφ ( h ) with hyperparameter space Φ . In a probabilistic setting , one typically uses the negative log-likelihood as the loss function , i.e. , l ( h , x , y ) = − ln p ( y|h ( x ) ) . Given the meta-training data D1 , ... , Dn and a hyper-prior distribution P ( φ ) over Φ , PACOH aims to infer a hyper-posterior distribution Q ( φ ) over the parameters φ of the prior . Using PAC-Bayesian learning theory [ 24 ] , they derive a high-probability bound on the transfer error , i.e. , the generalization error for posterior inference on an unseen task f ∼ T with priors sampled from the hyper-posterior Q ( φ ) . This meta-level generalization bound also serves as an objective for the meta-learner , minimized w.r.t . Q : J ( Q ) = − 1 n n∑ i=1 1 Ti Eφ∼Q [ lnZβ ( Di , Ti , Pφ ) ] + ( 1 λ + 1 nT̃ ) KL [ Q||P ] + const . ( 1 ) Here , λ > 0 , T̃ = ( T−11 + ... + T −1 n ) −1 is the geometric mean of the dataset sizes , lnZ ( Di , Ti , P ) : = ∫ H P ( h ) exp ( − ∑Ti t=1 l ( h , xi , t , yi , t ) ) dh the generalized marginal loglikelihood and KL ( Q||P ) the Kullback–Leibler ( KL ) divergence between hyper-posterior and hyper-prior . To obtain asymptotically consistent bounds , λ is typically chosen as λ = √ n. Bayesian Optimization . Bayesian Optimization ( BO ) aims to find the global maximizer x∗ = arg maxx∈X f ( x ) of a function f : X → R over a bounded domain X . To obtain an estimate of x∗ , the BO algorithm iteratively chooses points x1 , ... , xT ∈ X at which to query f , and observes noisy feedback y1 , ... , yT ∈ R , e.g. , via yt = f ( xt ) + , ∼ N ( 0 , σ2 ) with σ2 ≥ [ 13 , 14 ] . BO methods form a Bayesian surrogate model of the function f based on previous observations Dt = { ( xt , yt ) } . Typically , a Gaussian Process ( GP ) GP ( m ( x ) , k ( x , x′ ) ) with mean m ( x ) and kernel function k ( x , x′ ) is employed to form a posterior belief p ( f ( x ) |Dt ) over function values [ 25 ] . In each iteration , the next query point xt+1 : = arg maxx∈X αt ( x ) is chosen as maximizer of an acquisition function αt ( x ) that is typically based on the p ( f ( x ) |Dt ) and trades-off exploration and exploitation [ 26–29 ] . 3 Related Work . Meta-Learning . Common approaches in meta-learning attempt to learn a shared embedding space [ 3 , 30–32 ] amortize inference by a meta-learned recurrent model [ 33–35 ] or learn the initialization of a NN so it can be quickly adapted to new tasks [ 6 , 7 , 36 ] . A growing body of work also uses probabilistic modeling to enable uncertainty quantification in meta-learning [ 9 , 11 , 37 ] . However , when only given a small number of meta-training tasks such approaches tend to overfit and fail to provide reliable uncertainty estimates . The problem of overfitting to the meta-training tasks has been brought to attention by [ 17 , 38 ] , followed by work that discusses potential solutions in the form of meta-regularization [ 8 , 12 , 18 ] . Our meta-learning framework builds on a sequence of work on PAC-Bayesian meta-learning [ 22 , 23 , 12 ] . However , instead of using a hyper-prior over the latent parameters of the learnable prior , we define the hyper-prior as stochastic process in the function space which gives us better control over the behavior of the predictive distributions of our meta-learned model in the absence of data . Learning hypotheses in the function space . Recent work bridges the gap between stochastic processes and complex parametric models by developing a variational inference in the function space [ 21 , 39 ] . Our framework in the function space heavily builds on these ideas ; in particular , the proposed approximations of the KL-divergence between stochastic processes [ 21 , 40 , 41 ] . Meta-Learning and Transfer for Bayesian Optimization . To improve the sample efficiency of BO based on previous experience , a basic approach is to warm-start BO by initially evaluating configurations that have performed well on similar tasks in the past [ 42–44 ] . Another common theme is to form a global GP based model across tasks [ 45–48 ] . However , such global GPs quickly become computationally infeasible as they scale cubically in the number of tasks times the number samples per task or require hand-designed task features . Conceptually , the most similar to our work is the approach of learning a neural-network based feature map that is shared across tasks and used for BO based on a Bayesian linear regression model in the shared feature space [ 49 , 50 ] . While these methods are highly scalable , they lack any form meta-level regularization and thus , in face of data-scarcity , are prone to overfitting and over-overconfident uncertainty estimates . In contrast , our method overcomes these issues thanks to its principled meta-regularization in the function space .

Summary: This paper discusses a meta-learning method with priors defined on function spaces. Building on PACOH, a meta-learning framework with PAC-Bayesian bounds, the paper proposes F-PACOH where the hyper-priors and priors defining meta-learning models in function spaces. In original PACOH, the hyper-priors are set to be parametric distributions (usually Gaussians), and the authors argue that such parametric priors cannot effectively capture the functional structure of the tasks that we ultimately want to enforce for meta-learners. On the other hand, functional priors directly regularize functions of interest to be close to prior knowledge, e.g., standard Gaussian processes, so result in more reliable uncertainty estimates.   The inference is done by optimizing the PAC-Bayesian bound, and the caveat here is the computation of KL-divergence between functional hyper-priors and hyper-posteriors. This can be approximated by the technique introduced in Sun et al., 2021, where a set of measurements points are sampled and the KL-divergence is approximated with the evaluations on those measurement points.  The paper mainly focuses on the application of the proposed method to Bayesian optimization. On several benchmarks, the proposed F-PACOH is demonstrated to quickly meta-learn and improves on similar BO tasks.