Meta-Learning Acquisition Functions for Transfer Learning in Bayesian Optimization

1 INTRODUCTION . Global optimization of black-box functions is highly relevant for a wide range of real-world tasks . Examples include the tuning of hyperparameters in machine learning , the identification of control parameters , or the optimization of system designs . Such applications oftentimes require the optimization of relatively low-dimensional ( . 10D ) functions where each function evaluation is expensive in either time or cost . Furthermore , there is typically no gradient information available . In this context of data-efficient global black-box optimization , Bayesian optimization ( BO ) has emerged as a powerful solution ( Močkus , 1975 ; Brochu et al. , 2010 ; Snoek et al. , 2012 ; Shahriari et al. , 2016 ) . BO ’ s data efficiency originates from a probabilistic surrogate model which is used to generalize over information from individual data points . This model is typically given by a Gaussian process ( GP ) , whose well-calibrated uncertainty prediction allows for an informed explorationexploitation trade-off during optimization . The exact manner of performing this trade-off , however , is left to be encoded in an acquisition function ( AF ) . There is a wide range of AFs available in the literature which are designed to yield universal optimization strategies and therefore come with minimal assumptions about the class of target objective functions . To achieve optimal data-efficiency on new instances of previously seen tasks , however , it is crucial to incorporate the information obtained from these tasks into the optimization . Therefore , transfer ∗Correspondence to : Michael.Volpp @ de.bosch.com learning is an important and active field of research . Indeed , in many practical applications , optimizations are repeated numerous times in similar settings , underlining the need for specialized optimizers . Examples include hyperparameter optimization which is repeatedly done for the same machine learning model on varying datasets or the optimization of control parameters for a given system with varying physical configurations . Following recent approaches ( Swersky et al. , 2013 ; Feurer et al. , 2018 ; Wistuba et al. , 2018 ) , we argue that it is beneficial to perform transfer learning for global black-box optimization in the framework of BO to retain the proven generalization capabilities of its underlying GP surrogate model . To not restrict the expressivity of this model , we propose to implicitly encode the task structure in a specialized AF , i.e. , in the optimization strategy . We realize this encoding via a novel method which meta-learns a neural AF , i.e. , a neural network representing the AF , on a set of source tasks . The meta-training is performed using reinforcement learning , making the proposed approach applicable to the standard BO setting , where we do not assume access to objective function gradients . Our contributions are ( 1 ) a novel transfer learning method allowing the incorporation of implicit structural knowledge about a class of objective functions into the framework of BO through learned neural AFs to increase data-efficiency on new task instances , ( 2 ) an automatic and practical metalearning procedure for training such neural AFs which is fully compatible with the black-box optimization setting , i.e , not requiring objective function gradients , and ( 3 ) the demonstration of the efficiency and practical applicability of our approach on a challenging simulation-to-real control task , on two hyperparameter optimization problems , as well as on a set of synthetic functions . 2 RELATED WORK . The general idea of improving the performance or convergence speed of a learning system on a given set of tasks through experience on similar tasks is known as learning to learn , meta-learning or transfer learning and has attracted a large amount of interest in the past while remaining an active field of research ( Schmidhuber , 1987 ; Hochreiter et al. , 2001 ; Thrun and Pratt , 1998 ; Lake et al. , 2016 ) . In the context of meta-learning optimization , a large body of literature revolves around learning local optimization strategies . One line of work focuses on learning improved optimizers for the training of neural networks , e.g. , by directly learning update rules ( Bengio et al. , 1991 ; Runarsson and Jonsson , 2000 ) or by learning controllers for selecting appropriate step sizes for gradient descent ( Daniel et al. , 2016 ) . Another direction of research considers the more general setting of replacing the gradient descent update step by neural networks which are trained using either reinforcement learning ( Li and Malik , 2016 ; 2017 ) or in a supervised fashion ( Andrychowicz et al. , 2016 ; Metz et al. , 2019 ) . Finn et al . ( 2017 ) , Nichol et al . ( 2018 ) , and Flennerhag et al . ( 2019 ) propose approaches for initializing machine learning models through meta-learning to be able to solve new learning tasks with few gradient steps . We are currently aware of only one work tackling the problem of meta-learning global black-box optimization ( Chen et al. , 2017 ) . In contrast to our proposed method , the authors assume access to gradient information and choose a supervised learning approach , representing the optimizer as a recurrent neural network operating on the raw input vectors . Based on statistics of the optimization history accumulated in its memory state , this network directly outputs the next query point . In contrast , we consider transfer learning applications where gradients are typically not available . A number of articles address the problem of increasing BO ’ s data-efficiency via transfer learning , i.e. , by incorporating information obtained from similar optimizations on source tasks into the current target task . A range of methods accumulate all available source and target data in a single GP and make the data comparable via a ranking algorithm ( Bardenet et al. , 2013 ) , standardization or multi-kernel GPs ( Yogatama and Mann , 2014 ) , multi-task GPs ( Swersky et al. , 2013 ) , the GP noise model ( Theckel Joy et al. , 2016 ) , or by regressing on prediction biases ( Shilton et al. , 2017 ) . These approaches naturally suffer from the cubic scaling behaviour of GPs , which can be tackled for instance by replacing the GP model , e.g. , with Bayesian neural networks with task-specific embedding vectors ( Springenberg et al. , 2016 ) or with adaptive Bayesian linear regression with basis functions shared across tasks via a neural network ( Perrone et al. , 2018 ) . Recently , Garnelo et al . ( 2018 ) proposed Neural Processes as another interesting alternative for GPs with improved scaling behavior . Other approaches retain the GP surrogate model and combine individual GPs for source and target tasks in an ensemble model with the weights adjusted according to the GP uncertainties ( Schilling et al. , 2016 ) , dataset similarities ( Wistuba et al. , 2016 ) , or estimates of the GP generalization performance on the target task ( Feurer et al. , 2018 ) . Similarly , Golovin et al . ( 2017 ) form a stack of GPs by iteratively regressing onto the residuals w.r.t . the most recent source task . In contrast to our proposed method , many of these approaches rely on hand-engineered dataset features to measure the relevance of source data for the target task . Such features have also been used to pick promising initial configurations for BO ( Feurer et al. , 2015a ; b ) . The method being closest in spirit and capability to our approach is proposed by Wistuba et al . ( 2018 ) . It is similar to the aforementioned ensemble techniques with the important difference that the source and target GPs are not combined via a surrogate model but via a new AF , the socalled transfer acquisition function ( TAF ) . This AF is defined to be a weighted superposition of the predicted improvements according to the source GPs and the expected improvement according to the target GP . Viewed in this context , our method also combines knowledge from source and target tasks in a new AF which we represent as a neural network . Our weighting of source and target data is implicitly determined in a meta-learning phase and is automatically regulated during the optimization on the target task to adapt online to the specific objective function at hand . Furthermore , our method does not store and evaluate many source GPs because the knowledge from the source datasets is encoded directly in the network weights of the learned AF . This allows our method to incorporate large amounts of source data while the applicability of TAF is restricted to a comparably small number of source tasks . 3 PRELIMINARIES . We are aiming to find a global optimum x∗ ∈ argmaxx∈D f ( x ) of some unknown objective function f : D → R on the domainD ⊂ RD . The only means of acquiring information about f is via ( possibly noisy ) evaluations at points in D. Therefore , at each optimization step t ∈ { 1 , 2 , . . . } , the optimizer has to decide for the iterate xt ∈ D solely based on the optimization historyHt ≡ { xi , yi } t−1i=1 with yi = f ( xi ) + . Here , ∼ N ( 0 , σ2n ) denotes independent and identically distributed Gaussian noise . In particular , the optimizer does not have access to gradients of f . To assess the performance of global optimization algorithms , it is natural to use the simple regret Rt ≡ f ( x∗ ) − f ( x+t ) where x+t is the input location corresponding to the best evaluation found by an algorithm up to and including step t. The proposed method relies on the framework of BO and is trained using reinforcement learning . Therefore , we now shortly introduce these frameworks . Bayesian Optimization In Bayesian optimization ( BO ) ( Shahriari et al. , 2016 ) , one specifies a prior belief about the objective function f and at each step t builds a probabilistic surrogate model conditioned on the current optimization history Ht . Typically , a Gaussian process ( GP ) ( Rasmussen and Williams , 2005 ) is employed as the surrogate model in which case the resulting posterior belief about f ( x ) follows a Gaussian distribution with mean µt ( x ) ≡ E { f ( x ) |Ht } and variance σ2t ( x ) ≡ V { f ( x ) |Ht } , for which closed-form expressions are available . To determine the next iterate xt based on the belief about f givenHt , a sampling strategy is defined in terms of an acquisition function ( AF ) αt ( · |Ht ) : D → R. The AF outputs a score value at each point in D such that the next iterate is defined to be given by xt ∈ argmaxx∈D αt ( x |Ht ) . The strength of the resulting optimizer is largely based upon carefully designing the AF to trade-off exploration of unknown versus exploitation of promising areas in D. There is a wide range of general-purpose AFs available in the literature . Popular choices are probability of improvement ( PI ) ( Kushner , 1964 ) , GP-upper confidence bound ( GP-UCB ) ( Srinivas et al. , 2010 ) , and expected improvement ( EI ) ( Močkus , 1975 ) . In our experiments , we will use EI as a not pre-informed baseline AF , so we state its definition here , EIt ( x ) ≡ Ef ( x ) { max [ f ( x ) − f ( x+t−1 ) , 0 ] ∣∣Ht } , ( 1 ) and note that it can be written in closed form if f ( x ) follows a Gaussian distribution . To perform transfer learning in the context of BO , Wistuba et al . ( 2018 ) introduced the transfer acqusition framework ( TAF ) which defines a new AF as a weighted superposition of EI on the target task and the predicted improvements on the source tasks , i.e. , TAFt ( x ) ≡ wM+1EI M+1 t ( x ) + ∑M j=1 wjI j t ( x ) ∑M+1 j=1 wj , ( 2 ) with the predicted improvement Ijt ( x ) ≡ max ( µj ( x ) − yj , maxt−1 , 0 ) . ( 3 ) TAF stores separate GP surrogate models for the source and target tasks , with j ∈ { 1 , . . . , M } indexing the source tasks and j = M + 1 indexing the target task . Therefore , EIM+1t denotes EI according to the target GP surrogate model and µj denotes the mean function of the j-th source GP model . yj , maxt denotes the maximum of the mean predictions of the j-th source GP model on the set of iterates { xi } ti=1 . The weights wj ∈ R are determined either based on the predicted variances of the source and target GP surrogate models ( TAF-ME ) or , alternatively , by a pairwise comparison of the predicted performance ranks of the iterates ( TAF-R ) . Reinforcement Learning Reinforcement learning ( RL ) allows an agent to learn goal-oriented behavior via trial-and-error interactions with its environment ( Sutton and Barto , 1998 ) . This interaction process is formalized as a Markov decision process : at step t the agent senses the environment ’ s state st ∈ S and uses a policy π : S → P ( A ) to determine the next action at ∈ A . Typically , the agent explores the environment by means of a probabilistic policy , i.e. , P ( A ) denotes the probability measures over A . The environment ’ s response to at is the next state st+1 , which is drawn from a probability distribution with density p ( st+1 | st , at ) . The agent ’ s goal is formulated in terms of a scalar reward rt = r ( st , at , st+1 ) , which the agent receives together with st+1 . The agent aims to maximize the expected cumulative discounted future reward η ( π ) when acting according to π and starting from some state s0 ∈ S , i.e. , η ( π ) ≡ Eπ [ ∑T t=1 γ t−1rt ∣∣∣ s0 ] . Here , T denotes the episode length and γ ∈ ( 0 , 1 ] is a discount factor .

Summary: This paper proposes a framework for meta learning neural acquisition functions for the Bayesian optimization of various underivable functions. The neural acquisition functions are learned using proximal policy optimization in an outer loop on different problems on the same domain, and the learned acquisition function can be deployed at test time in a practically vanilla Bayesian optimization procedure. The authors demonstrate the performance of the method through benchmarks on four problems.