Meta-Learning Acquisition Functions for Transfer Learning in Bayesian Optimization
Authors: Michael Volpp
Lukas P. Fröhlich
Kirsten Fischer
Andreas Doerr
Stefan Falkner
Frank Hutter
Christian Daniel
Venue:  (2020)
Keywords: Transfer Learning
Meta Learning
Bayesian Optimization
Reinforcement Learning

Abstract: Transferring knowledge across tasks to improve data-efficiency is one of the open key challenges in the field of global black-box optimization. Readily available algorithms are typically designed to be universal optimizers and, therefore, often suboptimal for specific tasks. We propose a novel transfer learning method to obtain customized optimizers within the well-established framework of Bayesian optimization, allowing our algorithm to utilize the proven generalization capabilities of Gaussian processes. Using reinforcement learning to meta-train an acquisition function (AF) on a set of related tasks, the proposed method learns to extract implicit structural information and to exploit it for improved data-efficiency. We present experiments on a simulation-to-real transfer task as well as on several synthetic functions and on two hyperparameter search problems. The results show that our algorithm (1) automatically identifies structural properties of objective functions from available source tasks or simulations, (2) performs favourably in settings with both scarse and abundant source data, and (3) falls back to the performance level of general AFs if no particular structure is present.

Published as a conference paper at ICLR 2020
META- ACQUISITION FUNCTIONS FOR TRANSFER LEARNING IN BAYESIAN OPTIMIZATION
Michael Volpp'-
Lukas P. Fröhlich 1.2
Kirsten Fischer
Andreas Doerr
Stefan Falkner_
Frank Hutter1.1
Christian Daniel"
Bosch Center for Artificial Intelligence, Renningen, Germany
2ETH Zürich, Zürich, Switzerland
3 Max Planck Institute for Intelligent Systems, Stuttgar//übbingen, Germany
University of Freiburg, Germany
ABSTRACT
Transferring knowledge across tasks to improve data-efficiency is one of the open key challenges in the field of global black-box optimization. Readily available algorithms are typically designed to be universal optimizers and, therefore, often suboptimal for specific tasks. We propose novel transfer learning method to obtain customized optimizers within the eelllestablished framework of Bayesian opti- mization, allowing algorithm to utilize the proven generalization capabilitiess of Gaussian processes. Using reinforcement learning to meta-train an acquisition func- tion (AF) structural information and to exploit it for improved data-efficiency. We present experiments a simulation-to-real transfer task well on several synthetic functions and on two hyperparameter search problems. The results show that our algorithm (I) automatically identifies structural properties of objective functions from available source tasks or simulations, (2) performs favourably in settings with both searse and abundant source data, and (3) falls back the performance level of general AFs if no particular structure is present.
of related tasks, the proposed method learns to extract implicit
INTRODUCTION
Global optimization of black-box functions is highly relevant for a wide range of real-world tasks. Examples include the tuning of hyperparameters in machine learning. the identification of control parameters, or the optimization of system designs. Such applications oftentimes require the optimiza- tion of relatively tow-dimensional (S 10D) functions where each function evaluation expensive in either time cost. Furthermore, there is typically no gradient information available.
In this context of data-efficient global black-box optimization, Bayesian optimization (BO) has emerged as powerful solution (Močkus, 1975) Brochu et al. 2010 Snoek et al. 2012 Shahriari jet al. 2016). BO's data efficiency originates from a probabilistic surrogate model which is used to generalize over information from individual data points. This model is typically given by Gaussian process (GP), whose well-calibrated uncertainty prediction allows for an informed exploration- exploitation trade-off during optimization. The exact manner of performing this trade-off, however, left to be encoded in an acquisition function (AF). There is a wide range of AFs available in the literature which designed to yield universal optimization strategies and therefore come with minimal assumptions about the class of target objective functions.
To achieve optimal data-efficiency new instances of previously seen tasks, however, it is crucial to incorporate the information obtained from these tasks into the optimization. Therefore, transfer
"oorrespondence Mi chael vo pp@de, bosch.com

---

Published as a conference paper at ICLR 2020
learning is an important and active field of research. Indeed, in many practical applications, op- timizations are repeated numerous times in similar settings, underlining the need for specialized optimizers. Examples include hyperparameter optimization which is repeatedly done for the same machine learning model on varying datasets or the optimization of control parameters for given system with varying physical configurations.
Following recent approaches (Swersky et al. 2013 Feurer et al. 2018 Wistuba et al. 2018), we argue that it is beneficial perform transfer learning for global black- optimization the framework of BO to retain the proven generalization capabilities of its underlying GP surrogate model. To not restrict the expressivity of this model, we propose to implicitly encode the task structure in a specialized AF, i.c., in the optimization strategy. We realize this encoding via a novel method which meta-learns neural AF, i.e., a neural network representing the AF, on set of source tasks. The meta-training is performed using reinforcement learning, making the proposed approach applicable to the standard BO setting, where we not assume access to objective function gradients.
Our contributions are (1) novel transfer learning method allowing the incorporation of implicit structural knowledge about a class of objective functions into the framework of BO through learned neural AFs to increase data-efficiency learning procedure for training such neural AFs which is fully compatible with the black- -bxx optimization setting, i.e, requiring objective function gradients, and (3) the demonstration of the efficiency and practical applicability of our approach on a challenging simulation -t-rral control task, on two hyperparameter optimization problems, as well as set of synthetic functions.
new task instances, (2) automatic and practical meta-
2 RELATED WORK
The general idea of improving the performance or convergence speed of a learning system on a given set of tasks through experience on similar tasks is known learning learn, meta-learning or transfer learning and has attracted a large amount of interest in the past while remaining an active
field of research (Schmidhuber 1987 Hochreiter al., 2001 Thrun and Pratt 1998 Lake al. 2016).
In the context of meta-learning optimization, large body of literature revolves around learning local optimization strategies. One line of work focuses on learning improved optimizers for the training of neural networks, c.g.. by directly learning update rules (Bengio al., [991] Runarsson and Jonsson 2000) or by learning controllers for selecting appropriate step sizes for gradient descent (Daniel et al. 2016). Another direction of research considers the more general setting replacing the gradient descent update step by neural networks which are trained using either reinforcement learning (Li and Malik 2016 2017) or in a supervised fashion (Andrychowicz et al. 2016 Metz et al. 2019). Finn ct al. (2017). Nichol et al. (2018), and Flennerhag et al. (2019) propose approaches for initializing machine learning models through meta-learning to be able to solve new learning tasks with few gradient steps.
We are currently aware of only one work tackling the problem of meta-learning global black-box optimization (Chen et al. 2017). In contrast our proposed method, the authors assume access to gradient information and choose supervised learning approach, representing the optimizer as a recurrent neural network operating on the raw input vectors. Based on statistics of the optimization history accumulated in its memory state, this network directly outputs the next query point. In contrast, we consider transfer learning applications where gradients are typically not available.
A number of articles address the problem of increasing BO's data-efficiency via transfer learning, i.e.. by incorporating information obtained from similar optimizations on source tasks into the current target task. A range of methods accumulate all available source and target data in a single GP and make the data comparable via a ranking algorithm (Bardenet et al., 2013), , standardization or multi-kernel GPs ( (Yogatama and Mann 2014). multi-task GPs (Swersky et al. 2013), the GP noise model (Theckel Joy et al.) 2016). or by regressing on prediction biases (Shilton et al. 2017). These approaches naturally suffer from the cubic scaling behaviour of GPs, which can be tackled for instance by replacing the GP model, e.g., with Bayesian neural networks with task-specific embedding vectors (Springenberg et al., 2016) with adaptive Bayesian linear regression with basis functions shared across tasks via a neural network (Perrone et al.) 2018). Recently, Garnelo et al. (2018) proposed Neural Processes as another interesting alternative for GPs with improved scaling behavior. Other
2

---

Published as a conference paper at ICLR 2020
approaches retain the GP surrogate model and combine individual GPs for source and target tasks in an ensemble model with the weights adjusted according to the GP uncertainties (Schilling et al. (0166). dataset similarities (Wistuba et al.; 2016), or estimates of the GP generalization performance on the target task (Feurer et al. 2018). Similarly, Golovin et al. (017) form a stack of GPs by iteratively regressing onto the residuals w.r.f. the most recent source task. In contrast our proposed method, many of these approaches rely on hand-engineered dataset features to measure the relevance of source data for the target task. Such features have also been used to pick promising initial configurations for BO (Feurer et al. 2015a]b)
The method being closest in spirit and capability our approach is proposed by Wistuba et al.) (2018). It is similar to the aforementioned ensemble techniques with the important difference that the source and target GPs are not combined via a surrogate model but via a new AF, the so- called transfer acquisition function (TAF). This AF is defined to be weighted superposition of the predicted improvements according to the source GPs and the expected improvement according to the target GP. Viewed in this context, our method also combines knowledge from source and target tasks in a new AF which we represent as a neural network. Our weighting of source and target data is implicitly determined in a meta-learning phase and is automatically regulated during the optimization on the target task to adapt online to the specific objective function at hand. Furthermore, our method does not store and evaluate many source GPs because the knowledge from the source datasets is encoded directly in the network weights of the learned AF. This allows our method to incorporate large amounts of source data while the applicability of TAF restricted to a comparably small number of source tasks.
PRELIMINARIES
We are aiming to find global optimum x² E arg ma f(x of some unknown objective function f: D + E on the domain D C DR The only means of acquiring information about f is via (possibly noisy) evaluations at points in D. Therefore, at each optimization step E (1.2....) }, the optimizer has to decide for the iterate o E D solely based on the optimization history He = {æ,, y | with y/: = -(x;) + e. Here, e ~ N (0, o denotes independent and identically distributed Gaussian noise. In particular, the optimizer does not have access to gradients of f. To assess the performance of global optimization algorithms, it is natural to use the simple regret R₂ = f(x') - f(=²') ) where x;"" the input location corresponding to the best evaluation found by an algorithm up to and including step l. The proposed method relies on the framework of BO and is trained using reinforcement learning. Therefore, we now shortly introduce these frameworks.
Bayesian Optimization In Bayesian optimization (BO) (Shahriari et al. 2016). one specifies prior belief about the objective function f and at each step t builds probabilistic surrogate model conditioned on the current optimization history H4. Typically, a Gaussian process (GP)
(Rasmussen and Williams| 2005) is employed the surrogate model in which case the resulting posterior belief about f(x) follows Gaussian distribution with mean s4 (x(xx) (x) = E {f (æ) H.) and variance of (x) V { (ax) H.). for which closed-form expressions are available. To determine the next iterate x, based on the belief about f given H4, a sampling strategy is defined in terms of
an acquisition function (AF) (- H2) : D + R. The AF outputs score value at each point in D such that the next iterate is defined to be given by at E arg max Ore( (# (H).. The strength of the resulting optimizer is largely based upon carefully designing the AF to trade-off exploration of unknown versus exploitation of promising areas in D.
There is wide range of general-purpose AFs available in the literature. Popular choices are probability of improvement (PI) (Kushner, 1964), GP-upper confidence bound (GP-UCB) (Srinivas) jt al,, 2010), and expected improvement (EI) (Močkus 1975). our experiments, we will use El as not pre-informed baseline AF, state its definition here.
(L) = Be) max [f(x) - [(tt)) H.).
and note that it can be written in closed form if fxx follows Gaussian distribution.
To perform transfer learning in the context of BO. Wistuba et al. (2018) introduced the transfer acqusition framework (TAF) which defines new AF weighted superposition of EI on the target
3

---

Published as a conference paper at ICLR 2020
task and the predicted improvements on the source tasks, i.e.,
TAF, WM++EE EI (a: -
w
(2)
M-
أ==1 W
with the predicted improvement
(x()) max (p'(z) vi v 0).
(3)
TAF stores separate GP surrogate models for the source and target tasks, with j (1,..., M) indexing the source tasks and j M + 1 indexing the target task. Therefore, E1, denotes El according to the target GP surrogate model and μr² denotes the mean function of the j-th source GP model. y denotes the maximum of the mean predictions of the j-th source GP model on the set of iterates ( {m: { The weights w/ E R are determined either based on the predicted variances of the source and target GP surrogate models (TAF-ME) or, alternatively, by pairwise comparison of the predicted performance ranks of the iterates (TAF-R).
Reinforcement Learning Reinforcement learning (RI.) allows an agent to learn goal-oriented be- havior via trial-and-error interactions with its environment (Sutton and Barto 1998). This interaction process is formalized as a Markov decision process: at step f the agent senses the environment's state 9; E S and uses a policy S + P(A) to determine the next action a E A. Typically, the agent explores the environment by means of a probabilistic policy, i.e., P(A) denotes the probability measures over A. The environment's response to a, is the next state St+1, which is drawn from probability distribution with density p( 1 8,, az). The agent's s goal is formulated in terms of a scalar reward r, r(tt, (t, 81+1). which the agent receives together with 81+1- The agent aims to maximize the expected cumulative discounted future reward f)()) when acting according to æ and
starting from some state sg E S, i.c. (TT) = E, ET= 1- re
su
Here, T denotes the episode
length and y (0, 1] is a discount factor.
METABO ALGORITHM
We devise a global black box optimization method that able to automatically identify and exploit structural properties of a given class of objective functions for improved data-efficiency. We stay within the framework of BO, enabling us to exploit the powerful generalization capabilities of GP surrogate model. The actual optimization strategy which is informed by this GP is classically encoded a hand-desipe aund-designed AF. Instead, we meta-train on a set of source tasks to replace this AF by neural network but retain all other elements of the proven BO-loop (middle panel of Fig. [ To distinguish the learned AF from a classical AF ag, we call such a network a neural acquisition function and denote it by 0r,0. indicating that it is parametrized by vector 6. We dub the resulting algorithm MetaBO.
Let F be the class of objective functions for which aim to learn neural acquisition function 04,6- For instance, F may be the set of objective functions resulting from different physical configurations of a laboratory experiment or from evaluating the loss function of machine learning model on different data sets. Often, such objective functions share structure which we aim to exploit for data-efficient optimization on further instances from the same function class. In many relevant cases, it is straightforward to obtain approximations to F, i.e., a set of functions FF' which capture relevant properties of but are much cheaper evaluate (e.g.. by using numerical simulations or results from previous hyperparameter optimization tasks (Wistuba et al.) (018)). During an offline meta-training phase, MetaBO makes use of such cheap approximations to identify the implicit structure of and to adapt 0 to obtain data-efficient optimization strategy customized to F.
Typically, the minimal set of inputs to AFs in BO is given by the pointwise GP posterior prediction (x) and (z). To perform transfer learning, the AF has to be able to identify relevant structure shared by the objective functions in F. In our setting, this is achieved via extending this basic set of inputs by additional features which enable the neural AF to evaluate sample locations. Therefore, in addition to the mean μ4 (z) and variance or (æ) at potential sample locations, the neural AF also receives the input location itself. Furthermore, we add to the set of input features the current.
4

---

Published as a conference paper at ICLR 2020
H4
[=(Sup] (G
p- (B..))
PPO
P
GP
BO loop
GP
i-th batch
6r.3
6/,2
otginnac Fsample
sample &,
MctaBO Training Loop
Neural AF in the BO loop
Policy architecture
Figure 1: Different levels of the MetaBO framework. Left panel: structure of the training loop for meta-learning neural AFs using RL (PPO). Middle panel: the classical BO loop with neural AF 01,0. At test time, there is difference classical BO, i.e., æ; is given by the arg max of the AF output. During training, the AF corresponds to the RI. policy evaluated on an adaptive set & C D. The outputs are interpreted logits of categorical distribution from which the actions 0r 2t, & are sampled. This sampling procedure is detailed in the right panel. We indicate by the dotted curve and tiny two-headed arrows that 0z a function defined on the whole domain which can be evaluated at arbitrary points Ei. to form the categorical distribution representing the policy No.
Table 1: The MetaBO setting in the RL framework.
RL
MetaBO
Policy my
Neural AF 04,0
Episode
Optimization on FF Optimization budget T
Episode length T
State
(Ee(E..)) (E...) Et.,, f, 7)
Action ae
Sampling point æ; E E
Reward Tz
Negative simple regret Rr
Transition pl 8141 8c, (t)
Noisy evaluation f, GP update
optimization step and the optimization budget T, as these features can be valuable for adjusting the exploration-expllotatioo trade-off (Srinivas al. 2010). Therefore, define
(, (x) (x,) (z) . 1.
This architecture allows learning a scalable neural AF, as we still base our architecture only on the pointwise GP posterior prediction. Furthermore, neural AFs of this form can be used plug-in feature in any state-of-the-ar BO framework. In particular, if differentiable activation functions chosen, neural AF constitutes differentiable mapping D IR and standard gradient-based optimization strategies can be used to find its maximum in the BO loop during evaluation. We further emphasize that after the training phase the resulting neural AF fully defined, i.c., there no need to calibrate any AF-related hyperparameters.
Training Procedure In the general BO setting, gradients of are assumed unavailable. This is oftentimes also true for the functions in F², for instance, when F' comprises numerical simulations or results from previous optimization runs. Therefore, we resort to RL. as the meta- algorithm, does not require gradients of the objective functions. Specifically, Policy Optimization (PPO) algorithm as proposed in Schulman et al. (2017). Tab. translates the MetaBO-setting into RL parlance.
the Proximal
We aim to shape the mapping (1,) during meta- veta-training in such a way that its maximum location corresponds to promising sampling location for optimization. The aego-tlgo PPO explores its state space using a parametrized stochastic policy ny from which the actions a, = D, are sampled depending the current state 8t, i.e., at no (- 8t). As the meta-algorithm requires access

---

Published as a conference paper at ICLR 2020
the global information contained in the GP posterior prediction, the state st at optimization step t formally corresponds to the functions pl and or (together with the aforementioned additional input features to the neural AF). To connect the neural AF 04 with the policy To and to arrive at practical implementation, we evaluate jrr and or on a discrete set of points & = { C D and feed these evaluations through the neural AF 04,00 ar.9 = aq,0 (fr.))).))) ,f( Et.,. T] for each point Et,u- These outputs are interpreted as the logits of categorical distribution, i.e., we arrive the policy architecture
at time, yielding one scalar output value
T (=) = Cat (f..), (E,,E,,
cf. Fig. right panel. Therefore, the proposed policy evaluates the same neural acquisition function or.o at arbitrarily many input locations Et., and preferably samples actions It E & with high (zz) This incentivizes the meta-algorithm to adjust 0 such that promising locations &t, attributed high values of (Et.nn)..)
Calculating sufficiently fine static set & of of evaluation points is challenging for higher dimensional settings. Instead, we build on the approach proposed by Snoek et al. (2012) and continuously adapt E to the current state of a1,8. At each step t, 04 is first evaluated on a static and relatively coarse Sobol grid (Sobol) 1967) Eglobal spanning the whole domain D. Subsequently, local maximizations of 04,0 are started from the k points corresponding to the best evaluations. We denote the resulting set of local maxima by Slocal,t- Finally, we define & = Élocal.e U Égloba]- The adaptive local part of this set enables the RL agent exploit what it has learned so far by picking points which look promising according to the current neural AF while the static global part maintains exploration. We refer the reader to App. B. for details.
The final characteristics of the neural AF are controlled through the choice of reward function. For the presented experiments we emphasized fast convergence to the optimum by using the negative simple regret as the reward signal, i.e., we set r; = R₂ | This choice does not penalize explorative evaluations which do not yield an immediate improvement and additionally serves as normalization of the functions f E F.. We emphasize that the knowledge of the true maximum is only required during training and that cases in which it is not known at training time do not limit the applicability of our method, as cheap approximation (e.g.. by evaluating the function on coarse grid) can also be utilized.
The left panel of Fig. depicts the resulting training loop graphically. The outer loop corresponds to the RL meta-training training iterations, each performing a policy update step no, + mo..1 To approximate the gradients of the PPO loss function, we record batch of episodes in the inner loop, i.e., set of (st, az, rt )tuuless by rolling out the current policy TØ, At the beginning of each episode, we draw some function f from the training set J and fix an optimization budget T. In each iteration of the inner loop we determine the adaptive set & and feed the state 9, through the policy which yields the action a = σ₁- We then evaluate o at 24 and use the result to compute the reward rt and to update the optimization history: He > Het1 = He U {Ii, }. Finally, the GP is conditioned on the updated optimization history Het1 obtain the next state $,+1-
5 EXPERIMENTS
We trained MetaBO on a wide range of function classes and compared the performance of the resulting neural AFs with the general-purpose AF expected improvement (EI as well as the transfer acquisition function framework (TAF) which proved to be the current state-of-the-ar solution for transfer learning in BO in extensive experimental study (Wistuba et al. 2018). We tested both the ranking-based version (TAF-R) and the mixture-of-expprt version (TAF-ME). We refer the reader to App. for a more detailed experimental investigation of MetaBO's performance.
If not stated differently, we report performance in terms of the median simple regret RS over 100 optimization runs on unseen test functions as function of the optimization step together with 30% 770% percentiles (shaded areas). We emphasize that all experiments use the same MetaBO
A Itermatively, a Dogaritmmicall-ttaafffommm version of this reward signal, r; = log R., can be used in situations where high-accuracy solutions shall he rewarded.
We also evaluated prohability of improvement (PI) as well as GP-upper confidence hound (GP-UCB) but do not present the results here to avoid clutter, as EI performed better in all our experiments.

---

Published as a conference paper at ICLR 2020
MetaBO (ours)
MetaBO-50 (ours)
Random
TAF-R-50
TAF-M--50
109
100
109
simple regret
simplic regest
10
simpli
10:2
6
12
18
24
30
24
30
6
12
18
24
30
(a) Branin (D
(b) Goldstein- Pricce (D
(c) Hartmann-3 (D
Figure 2: Performance on three global optimization benchmark functions with random translations
sampled uniformly from [-0.1, 0.1] and scalings from [0.9, 1.1]. To test TAF's performance, randomly picked M = 50 source tasks from this function class and evaluated both the sanking-based version (TAF-R-50) and the mixture-of-experts version (TAF-ME-50). We trained MetaBO on the same set of source tasks (MetaBO-50). In contrast to TAF, MetaBO can also be trained without man- ually restricting the set of available source tasks. The corresponding results are labelled "MetaBO". MetaBO outperformed El by clear margin, especially in early stages of the optimization. After few steps used identify the specific instance of the objective function, MetaBO also outperformed both flavors of TAF over wide ranges of the optimization budget. Results for TAF-20 can be found in Fig.
App.
hyperparameters, making method easily applicable in practice. Furthermore, MetaBO does not increase evaluation time considerably compared to standard AFs, cf. App. [A.2 Tab. 3 In addition, even the most expensive of our experiments (the iimulation-to-rell task, due to the simulation in the BO loop) required not more than 10h of training time on moderately complex architecture (10 CPU workers, 1 GPU), which is fully justified for our intended offline transfer learning use-case. To foster reproducibility, we provide a detailed exposition of the experimental settings in App. Band make the source code of MetaBO available online
Global Optimization Benchmark Functions We evaluated our method on set of synthetic function classes based on the standard global optimization benchmark functions Branin (D = 2).
Goldstein-Price (D 2). and Hartmann-3 (D =3) (Picheny et al. 2013). To construct the training set F', we applied translations in [-0.1, 0.1j. as well as scalings [0.9, 1.1].
As TAF stores and evaluates one source GP for each source task, its applicability is restricted to relatively small amount of source data. For the evaluations of TAF and MetaBO, we therefore picked random set of M 50 source tasks from the continuously parametrized family J' of available objective functions and spread these tasks uniformly over the whole range of translations and scalings (MetaBO-50, TAF-R-50, TAF-ME-50). We used NTAF 100 data points for each source GP of TAF. We also tested both flavors of TAF for M = 20 source tasks (with NTAF = 50) and observed that TAF's performance does not necessarily increase with more source data, rendering the choice of suitable source tasks cumbersome. Fig. shows the performance on unseen functions drawn randomly from F. To avoid clutter, we move the results for TAF-20 App. cf. Fig. 12 MetaBO-50 outperformed El by large margin, in particular at early stages of the optimization. by making use of the structural knowledge about F acquired during the meta-learning phase. Furthermore, MetaBO- 50 outperformed both flavors of TAF-50 over wide ranges of the optimization budget. This is due to its ability learn sampling strategies which go beyond a combination of a prior over D and a standard AF (as is the case for TAF). Indeed, note that MetaBO spends some initial non-greedy evaluations to identify specific properties of the target objective function, resulting in much more efficient optimization strategies. We investigate this behaviour further on simple toy experiments and using easily interpretable baseline AFs in App. A.I
We further emphasize that MetaBO does not require the user to manually pick suitable set of source tasks but that it can naturally learn from the whole set J of available source tasks by randomly picking new task from J at the beginning of each BO iteration and aggregating this information in the neural AF weights. We also trained this full version of MetaBO (labelled "MetaBO") on the
https: https////gghhth..c.om/sseeeeeeee /tte
7

---

Published as a conference paper at ICLR 2020
MetaBO (ours)
MetaBO-50 (ours)
TAF-R-100
IAF-ME-100
4,0
4.0
simple regres
2.0
0.0
0.0
10
20
5
20
25
(a) Evaluation in simulation.
(b) Evaluation hardware in (c).
(c) Exp. setup."
Figure 3: Performance on a simulation-to--eat task (cf. text). MetaBO and TAF used source data from a cheap numerical simulation. (a) Performance on an extended training set in simulation. (b) Transfer to the hardware depicted in (c), averaged over ten BO runs. MetaBO learned robust neural AFs with very strong optimization performance and online adaption to the target objectives, which reliably yielded stabilizing controllers after less than ten BO iterations while TAF-ME- 100, TAF-R-ION and El explore too heavily. Comparing the results for MetaBO and MetaBO-50 in simulation, we observe that MetaBO benefits from its ability to learn from the whole set of available source data, while TAF's applicability is restricted to a comparably small number of source tasks. We move the results for TAF-50 to App. A.4 Fig. 13
global optimization benchmark functions, obtaining performance comparable with MetaBO-50. We demonstrate below that for more complex experiments, such the simulation-to-reat task, MetaBO's ability to learn from the full set of available source tasks is crucial for efficient transfer learning. We also investigate the dependence of MetaBO's performance on the number of source tasks in more detail in App. A.2
As final test on synthetic functions, we evaluated the neural AFs objective functions outside of the training distribution. This can give interesting insights into the nature of the problems under consideration. We move the results of this experiment to App. A.3
Simulation-to-RRet Task Sample efficiency is of special interest for the optimization of real world systems. cases where an approximate model the system can be simulated, the proposed approach can be used to improve the data-efficiency on the real system. To demonstrate this, we evaluated MetaBO on a 4D dimulation-to-red experiment. The task was to stabilize a Furuta pendulum (Furuta ett al. [992] for 5 around the upper equilibrium position using a linear fatte-feedback controller. We applied BO to tune the four feedback gains of this controller (Fröhlich et al., 2020). To assess the performance of given controller, we employed logarithmic quadratic cost function (Bansal et al., 2017). If the controller was not able to stabilize the system or if the voltage applied to the motor exceeded some safety limit, we added penalty term proportional to the remaining time the pendulum would have had to be stabilized for successfully completing the task. We emphasize that the cost function is rather sensitive the control gains, resulting in a challenging black-box optimization problem.
To meta-learn the neural AF, we employed fast numerical simulation based on the nonlinear dynamics equations of the Furuta pendulum which only contained the most basic physical effects. In partieular, effects like friction and stiction were not modeled. The training distribution was generated by sampling the physical parameters of this simulation (two lengths, two masses), uniformly range of 75% 125% around the measured parameters of the hardware (Quanser QUBE - Servo Fig. 3(c).
We also used this simulation to generate M 100 source tasks for TAF (NTAF = 200).
Fig. ((a) shows the performance on objective functions from simulation. Again, MetaBO learned sophisticated sampling strategy which first identifies the target objective function and adapts its optimization strategy accordingly, resulting in very strong optimization performance. In contrast, TAF's superposition of a prior over D obtained from the source tasks with EI on the target task leads to excessive explorative behaviour. We move further experimental results for TAF-50 to App. A.4
Fig. 13
https:/ http://wwwwwwg//o wwwp:/paanser.co//poooeeooooooee servo-2
8

---

Published as a conference paper at ICLR 2020
MetaBO-35 (ours)
TAF-R-35
TAF-
SVM
Ada
SVM
AdaBoost
fraction unsleed
1.0
simpll rogret
10
10-7
0.8
10-1
10
12
15
12
15
6
15
3
12
15
(a) Simple regret (MetaBO's performance signal).
(b) Fraction of unsolved test tasks.
Figure 4: Performance on two 2D hyperparameter optimization tasks (SVM and AdaBoost). We trained MetaBO on precomputed data for 35 randomly chosen datasets and used the same datasets as source tasks for TAF. The remaining 15 datasets were used for this evaluation. MetaBO learned very data-efficient sampling strategies on both experiments, outperforming the benchmark methods by clear margin. Note that the optimization domain is discrete and therefore tasks can be solved exactly, corresponding to zero regret.
By comparing the performance of MetaBO and MetaBO-50 in simulation, we find that our archi- tecture's s ability to incorporate large amounts of source data is indeed beneficial on this complex optimization problem. The results in App. A.2 underline that this task indeed requires large amounts of source data to be solved efficiently. This is substantiated by the results on the hardware, on which we evaluated the full version of MetaBO and the baseline AFs obtained by training on data from simulation without any changes. Fig. 3(b) shows that MetaBO learned neural AF which generalizes well from the simulated objectives to the hardware task and was thereby able to rapidly adjust to its specific properties. This resulted in very data-efficient optimization on the target system, consistently yielding stabilizing controllers after less than ten BO iterations. In comparison, the benchmark AFs required many samples identify promising regions of the search space and therefore did not reliably find stabilizing controllers within the budget of 25 optimization steps.
As it provides interesting insights into the nature of the studied problem, investigate MetaBO's generalization performance to functions outside of the training distribution in App. A.3 We empha- size, however, that the intended use case of our method is on unseen functions drawn from the training distribution. Indeed, by measuring the physical parameters of the hardware system and adjusting the ranges from which the parameters are drawn to generate F according to the measurement uncertainty, the training distribution can be modelled in such a way that the true system parameters lie inside of it with high confidence.
Hyperparameter Optimization We tested MetaBO two 2D 2D- hyperparameter optimization (HPO) problems for RBF-based SVMs and AdaBoost. As proposed in Wistuba et al. (2018), we used precomputed results of training these models on 50 datasets " with 144 parameter configurations (RBF kernel parameter, penalty parameter C) for the SVMs and 108 configurations (number of product terms, number of iterations) for AdaBoost. We randomly split these datasets into 35 source datasets used for training MetaBO as well as for TAF and evaluated the resulting optimization strategies on the remaining 15 datasets. To determine when stop the meta-trainin training of MetaBO, we performed 7-fold cross validation on the training datasets. We emphasize that MetaBO did not use more source data than TAF in this experiment, underlining again its broad applicability in situations with both scarse and abundant source data. The results (Fig. 4 show that MetaBO learned very data-efficient neural AFs which surpassed El und TAF on both experiments.
General Function Classes Finally, we evaluated the performance of MetaBO function classes without any particular structure except a bounded correlation lengthscale. As there is only little structure present in this function class which could be exploited in the transfer learning setting, it is desirable to obtain neural AFs which fall back at least on the performance level of general-purpose AFs such EL. We performed two different experiments of this type. For the first experiment, we sampled the objective functions from a GP prior with qquared-exponentiie (RBF) kernel with lengthscales drawn uniformly from f e [0.05, 0.5] For the second experiment, we used a GP prior
A VVisualizations of the objective functions can be found on http: /www. hylap.org
We normalized the optimization domain to D =00, 1].
9

---

Published as a conference paper at ICLR 2020
MetaBO (ours)
Random
10
109
10P
simple regret
simple rogret
10-1
simpll
10"7
10"2
10-3
10:3
10
10
16
24
32
40
12
60
40
60
80
100
(a)
= 3. RBF kernel
(b) D 4,
RBF kernel
(c)
5, RBF kernel
10"
109
10a
simple regret
simple regret
simpll
10
10
10
10
10
10
16
84
112
(d) D> = 3, Matern-5/2 kernel
(c) 17 4, Matern-5/2 kernel
(f) = 5, Matern-5/2 kernel
Figure 5: Performance of MetaBO trained on D = 3-dimensional objective functions sampled from GP prior with RBF kernel (upper row) and Matern-5/2 kernel (lower row) with lengthscales drawn randomly from & [0.05, 0.5]. Panels (a, d) show the performance on these training distributions. As excluded the xe-fature from the neural AF inputs during training, the resulting AFs be applied to functions of different dimensionalities. We evaluated each AF on = and retraining MetaBO. We report simple regret w.r.t. the best observed function value, determined separately for each function in the test set.
without
with Matern-5/2 kernel with the same range of lengthscales. For the latter experiment also used the Matern-5/2 kernel (in contrast to the RBF kernel used all other experiments) as the kernel the GP surrogate model avoid model mismatch. For both types of function classes trained MetaBO version
on D = 3 dimensional tasks and excluded the æ-feature to study dimensionality-agnnne of MetaBO. Indeed, evaluated the resulting neural AFs without retraining for dimensionalities D {3, 5J. The results (Fig. 5) show that MetaBO is capable of learning neural AFs which perform better than least on on-par with El these general function classes.
6 CONCLUSION AND FUTURE WORK
We introduced MetaBO, a novel method for transfer learning in the framework of BO. Via a flexible meta-learning approach, inject prior knowledge directly into the optimization strategy of BO using neural AFs. The experiments show that our method consistently outperforms existing methods, for instance in simulation-to- -o-relll settings or on hyperparameter search tasks. Our approach is broadly applicable to a wide range of practical problems, covering both the cases of scarse and abundant source data. The resulting neural AFs can represent search strategies which go far beyond the abilities of current approaches which often rely weighted superpositions of priors over the optimization domain obtained from the source data with standard AFs. In future work, we aim to tackle the multi-task multi-fidelity setting (Valkov et al. 2018), where we expect MetaBO's sample efficiency to be of high impact.
ACKNOWLEDGEMENTS
We want to thank Julia Vinogradska, Edgar Klenske, Aaron Klein, Matthias Feurer, Gerhard Neumann, as well as the anonymous reviewers for valuable remarks and discussions which greatly helped to improve this paper.
10

---

Published as a conference paper at ICLR 2020
REFERENCES
Marcin Andrychowicz. Misha Denil, Sergio Gomez Colmenarejo, Matthew W. Hoffman, David Pfau, Tom Schaul, and Nando de Freitas. Learning Learn by Gradient Descent by Gradient Descent. In Advances in Neural Information Processing Systems, 2016.
Somil Bansal, Roberto Calandra, Ted Xiao, Sergey Levine, and Claire J. Tomlin. Goal-driven Dynamics Learning via Bayesian Optimization. In IEEE Annual Conference on Decision and Control, 2017.
Rémi Bardenet, Mátyás Brendel, Balázs Kégl, and Michèle Sebag. Collaborative Hyperparameter Tuning. In International Conference on Machine Learning, 2013.
Yoshua Bengio, Samy Bengio, and Jocelyn Cloutier. Learning a Synaptic Learning Rule. In International Joint Conference Neural Networks, 1991.
Eric Brochu, Vlad M. Cora, and Nando de Freitas. A Tutorial on Bayesian Optimization of Expensive Cost Functions, with Application to Active User Modeling and Hierarchical Reinforcement Learning. arXiv:1012.2593 2010.
Yutian Chen, Matthew W. Hoffman, Sergio Gómez Colmenarejo, Misha Denil, Timothy P. Lillicrap, Matt Botvinick, and Nando de Freitas. Learning to Learn without Gradient Descent by Gradient Descent. In International Conference on Machine Learning, 2017.
Christian Daniel, Jonathan Taylor, and Sebastian Nowozin. Learning Step Size Controllers for Robust Neural Network Training. In AAAI Conference on Artificial Intelligence, 2016.
Matthias Feurer, Aaron Klein, Katharina Eggensperger, Jost Tobias Springenberg, Manuel Blum, and Frank Hutter. Efficient and Robust Automated Machine Learning. In Advances in Neural Information Processing Systems, 2015a.
Matthias Feurer, Jost Tobias Springenberg, and Frank Hutter. Initializing Bayesian Hyperparameter Optimization via Meta-Learning. AAAI Conference on Artificial Intelligence, 2015b.
Matthias Feurer, Benjamin Letham, and Eytan Bakshy. Scalable Meta-Learning for Bayesian Optimization.
2018.
Chelsea Finn, Pieter Abbeel, and Sergey Levine. Model-Agnostte Meta-Learning for Fast Adaptation of Deep Networks. International Conference on Machine Learning, 2017.
Sebastian Flennerhag, Pablo Garcia Moreno, Neil Lawrence, and Andreas Damianou. Transferring Knowledge across Learning Processes. In International Conference Learning Representations, 2019.
Lukas P. Fröhlich, Edgar D. Klenske, Christian G. Daniel, and Melanie N. Zeilinger. Bayesian Optimization for Policy Search in fiihh-Dimensional Systems via Automatic Domain Selection. arXiv:2001.07394, 2020.
Katsuhisa Furuta, M. Yamakita, and S. Kobayashi. Swing-up Control of Inverted Pendulum Using Pseudo-State Feedback. Journal of Systems and Control Engineering, 206(4), 1992.
Marta Garnelo, Jonathan Schwarz, Dan Rosenbaum, Fabio Viola, Danilo J. Rezende, S. M. Ali Eslami, and Yee Whye Teh. Neural Processes. 1 2018.
Daniel Golovin, Benjamin Solnik, Subhodeep Moitra, Greg Kochanski, John Karro, and D. Sculley, Google Vizier: Service for Black-Box Optimization. International Conference on Knowledge Discovery and Data Mining, 2017.
GPy. GPy: A Gaussian Process Framework in Python, 2012.
Sepp Hochreiter, Steven A. Younger, and Peter R. Conwell. Learning to Learn Using Gradient Descent. In Artificial Newral Networks ICANN, 2001.
Harold J. Kushner. New Method of Locating the Maximum Point an Arbitrary Multipeak Curve in the Presence of Noise. Journal of Basic Engineering, 86(1), 1964.

---

Published as a conference paper at ICLR 2020
Brenden M. Lake, Tomer D. Ullman, Joshua B. Tenenbaum, and Samuel J. Gershman. Building Machines That Learn and Think Like People. arXiv:1604.00289, 2016.
Ke Li and Jitendra Malik. Learning to Optimize. arXiv:1606.01 2016.
Ke Li and Jitendra Malik. Learning to Optimize Neural Nets. #rXiv:1703.00441 2017.
Luke Metz, Niru Maheswaranathan, Brian Cheung, and Jascha Sohl-Dickstein. Learning Unsuper- vised Learning Rules. In International Conference on Learning Representations, 2019.
Jonas Močkus. On Bayesian Methods for Seeking the Extremum. In Optimization Techniques IFIP Technical Conference, 1975.
Alex Nichol, Joshua Achiam, and John Schulman. On First-Order Meta-Learning Algorithms. 1rXiv:1803.02999, 2018.
Valerio Perrone, Rodolphe Jenatton, Matthias Seeger, and Cedric Archambeau. Scalable Hyperpa- rameter Transfer Learning. In Advances in Neural Information Processing Systems, 2018.
Victor Picheny, Tobias Wagner, and David Ginsbourger. A Benchmark of Kriging- Infill Criteria for Noisy Optimization. Structural and Multidisciplinary Optimization, 48(3), 2013.
Carl Edward Rasmussen and Christopher K. I. Williams. Gaussian Processes for Machine Learning. The MIT Press, 2005.
Thomas Philip Runarsson and Magnus Thor Jonsson. In IEEE Symposium on Combinations of Evolutionary Computation and Neural Networks, 2000.
Nicolas Schilling, Martin Wistuba, and Lars Schmid--Thieme. Scalable Hyperparameter Optimization with Products of Gaussian Process Experts. European Conference on Machine Learning and Knowledge Discovery in Databases, 2016.
Jürgen Schmidhuber. Evolutionary Principles in Self-Referential Learning. On Learning how to Learn: The Mettt-MMtt--Mt...---e Diploma Thesis, Technische Universitat München, Germany, 1987.
John Schulman, Philipp Moritz, Sergey Levine, Michael I. Jordan, and Pieter Abbeel. High- Dimensional Continuous Control Using Generalized Advantage Estimation. 186.43850..02438. 2015.
John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal Policy Optimization Algorithms. arXiv:1707.06344 20337,2017 2017.
Bobak Shahriari, Kevin Swersky, Ziyu Wang, Ryan P. Adams, and Nando de Freitas. Taking the Human Out of the Loop: A Review of Bayesian Optimization. Proceedings of the IEEE, 104(1), 2016.
Alistair Shilton, Sunil Gupta, Santu Rana, and Svetha Venkatesh. Regret Bounds for Transfer Learning in Bayesian Optimisation. In International Conference on Artificial Intelligence and Statistics, volume 54, 2017.
Jasper Snoek, Hugo Larochelle, and Ryan P. Adams. Practical Bayesian Optimization of Machine Learning Algorithms. In Advances in Neural Information Processing Systems, 2012.
Ilya Mecrovich Sobol. On the Distribution of Points in Cube and the Approximate Evaluation of Integrals. Zhurnal Vychislitelnoi Matematiki Matematicheskoi Fiziki, 7(4), 1967.
Jost Tobias Springenberg, Aaron Klein, Stefan Falkner, and Frank Hutter. Bayesian Optimization with Robust Bayesian Neural Networks. In Advances in Neural Information Processing Systems, 2016.
Niranjan Srinivas, Andreas Krause, Sham Kakade, and Matthias W. Seeger. Gaussian Process Opti- mization in the Bandit Setting: No Regret and Experimental Design. In International Conference Machine Learning, 2010.
12

---

Published as a conference paper at ICLR 2020
Richard S. Sutton and Andrew G. Barto. Reinforcement Learning Introduction. Computation and Machine Learning. MIT Press, 1998.
Adaptive
Kevin Swersky, Jasper Snoek, and Ryan P. Adams. Multi-Task Bayesian Optimization. In Advances in Neural Information Processing Systems, 2013.
Tinu Theckel Joy, Santu Rana, Sunil Gupta, and Svetha Venkatesh. Flexible Transfer Learning Framework for Bayesian Optimisation. In Advances in Knowledge Discovery and Data Mining. 2016.
Sebastian Thrun and Lorien Pratt. Learning to Learn. Kluwer Academic Publishers, Norwell, MA, USA, 1998.
Lazar Valkov, Rodolphe Jenatton, Fela Winkelmolen, and Cédric Archambeau. A Simple Transfer- learning Extension of Hyperband. NeurIPS Workshop on Meta-Learning, 2018.
Martin Wistuba, Nicolas Schilling, and Lars for Automatic Hyperparameter Optimization. In European Conference on Machine Learning and Knowledge Discovery in Databases, 2016.
Two-Stage Transfer Surrogate Model
Martin Wistuba, Nicolas Schilling, and Lars Schmidt-Thiemee Scalable Gaussian Process-basee Transfer Surrogates for Hyperparameter Optimization. Journal of Machine Learning Research, 107(1), 2018.
Dani Yogatama and Gideon Mann. Efficient Transfer Learning Method for Automatic Hyperparameter Tuning, In International Conference on Artificial Intelligence and Statistics, 2014.
13

---

Published as a conference paper at ICLR 2020
A ADDITIONAL EXPERIMENTAL RESULTS
A.1 INTERPRETATION OF NEURAL AF SEARCH STRATEGIES
We provide additional experimental results to demonstrate that MetaBO's neural AFs learn represen- tations that go beyond some kind of standard AF combined with a prior over D.
Emergence of Non-Greedy Search Strategies To obtain intuition about the kind of search strate- gies MetaBO is able to learn, we devised two classes of one-dimensional toy objective functions.
The first class of objective functions (Rhino-1, cf. Fig. 6 is generated by applying random translations sampled uniformly from t E [-0.2, 0.2) to a function which is given by the superposition of two Gaussian bumps with different heights and widths and fixed distance,
(n, 1) = 0.5 WV (zx = 0.3 - 1, a 0.1) + 3.0 (N P 0.7 - 4,0 = 0.01), (6)
where we define N [x| μt a) exp(-1/2 (xr n)2 /oa). The second class of objective functions
(Rhino-2, cf. Fig. 7 is given by uniformly sampling the parameter h E [0.6, 0.9] of the function
faz(2, h) = h (V(( μ 0.2, = 0.1) 2.0 -V μ = h, a 0.01) 1.0.
For both of these function classes it is intuitively clear that the optimal search strategy involves a first non-greedy evaluation to identify the specific instance of the target function. Indeed, for all instances of these function classes, the smaller and wider bumps overlap and encode information about the position of the sharp global optimum. Therefore, an optimal strategy spends the first evaluation at a fixed position zo where all smaller and wider bumps have non- non-negligible heights you Then, for both function classes, the global optimum z can be determined exactly from y/o (if we assume noiseless evaluations), such that x" can be found in the second step. Figs. 6 7 show that MetaBO indeed learns such non-greedy optimization strategies, which go far beyond a simple combination of a prior over 'D with some kind of standard AF. As mentioned in the main part of this paper, we suppose that MetaBO employs similar strategies on more complex function classes. For instance, we observe in the experiments on the global optimization benchmark functions (Fig. 2 that MetaBO consistently starts with higher regret than the pre- rre-infrmedd TAF which suggests that it learned to spend a few non-greedy evaluations at the beginning of an optimization run to identify the specific instance of the target function.
Additional Baseline Methods To provide further evidence that MetaBO's neural AFs learn repre- sentations that go beyond simple prior over DD combined with some kind of standard AF, we show results for two additional baseline AFs which rely on such a naive combination.
We define the AF GMM- UCB as the following convex combination of a Gaussian Mixture Model (GMM) and the standard AF UCB:
GMM - UCB (z)
GMM (a+) + (1 w) UCB (æ))
(8)
The GMM is defined to have ceomp components and fitted to the best designs from each of the M
source tasks. Further, UCB is defined as
UCB (x) = /xx) + (a(x),
and we choose 3- 2 as is common in BO.
Furthermore, we define EPS-GREEDY as the AF which in each optimization samples without replacement from the set of best designs of each of the source tasks step with probability c and uses standard EI with probability e.
Note that these baseline methods are similar in spirit to the TAF-approach evaluated in the main part of this paper. Indeed, TAF, GMM-UCB, and EPS-GREEDY all rely on some kind of prior over D determined using the source data which is combined through weighted superposition with some standard AF. However, TAF uses more principled methods (TAF-ME, TAF-R) to adaptively determine the weights of this superposition.
To obtain optimal performance of GMM-UCL and ERE-GY,EDY, we chose the parameters for these methods by grid search on the test se d.....t. the median simple regret summed from to
"This yields upper hound on the possible performance of GMM-UCB and EPS-GREEDY, as in practice one would have estimate the parameters using separate validation set.
14

---

Published as a conference paper at ICLR 2020
Figure 6: Visualization of three BO episodes with neural AFs on the 1D Rhino-1 task. Each column of this figure correspond to one episode with three optimization steps. The uppermost row corresponds to the prior state before the objective function was queried. The fourth row depicts the state after three evaluations. Each subfigure shows the GP mean (dashed blue line), GP standard deviation (blue shaded area), and the ground truth function (black) in the upper panel as well as the neural AF the lower panel. Dashed red lines indicate the maxima of the ground truth function and of the neural AF, Red and green crosses indicate the recorded data (the red cross corresponds to the most recent data point). Each instance of this task is generated by randomly translating an objective function with two peaks of different heights and widths. The distance between the local and global optimum is the same for each instance. MetaBO learns a sophisticated sampling strategy, spending a non-greedy evaluation at the beginning of each episode at a position where the smaller but wider peaks overlap for every instance the function class to gain information about the location of the global optimum. Using this strategy, MetaBO is able to find the global optimum very efficiently.
15

---

Published as a conference paper at ICLR 2020
0.00
0.25
0.75
1.00
0.00
0.25
0.75
1.00
0.00
0.25
1.00
0.00
0.25
0.75
1,00
0.00
0.25
0.50
0.75
1.00
0.00
0.25
0.50
0.75
1.00
0.00
0.25
0.75
1.00
0.00
0.25
0.50
0.75
1.00
0.00
0.25
0.50
0.75
1.00
Figure Visualization of three episodes from the 1D Rhino-2 task. Each column of this figure correspond to one episode with two optimization steps. The uppermost row corresponds to the prior state before the objective function was queried. The third row depicts the state after two evaluations. Each subfigure shows the GP mean (dashed blue line), GP standard deviation (blue shaded area), and the ground truth function (black) in the upper panel as well as neural AF in the lower panel. Dashed red lines indicate the maxima of the ground truth function and of the neural AF. Red and green crosses indicate the recorded data (the red cross corresponds to the most recent data point). Each instance of this task is generated by sampling the height h of wide bump at fixed location
0.2 and placing sharp peak at x = h. MetaBO learns sophisticated sampling strategy, spending a non-greedy evaluation at æ 0.2 at the beggining each episode to gain information about the location of the global optimum. Using this strategy, MetaBO is able to find the global optimum very efficiently.
16

---

Published as a conference paper at ICLR 2020
MetaBO (ours)
MetaBO-50 (ours)
GMM-UCB
EPS-GREEDD
10"
10
simple simpleregnt regret
simple regret
10
simple
10"
10-1
12
18
24
30
24
30
12
18
24
30
(a) Branin (D= 2)
(b) Goldstein-Price (D) = 2)
(c) Hartmann-3
(D
3)
Figure 8: Performance on three global optimization benchmark functions with random translations
sampled uniformly from [-0.1, 0.11 and scalings from [0.9, We present results for two additional baseline methods (GMM-UCB, EPS-GREEDY) which rely weighted superposition of a prior over 'D obtained from M = 50 source tasks and a standard AF and can thus be easily interpreted. As MetaBO produces more sophisticated search strategies, these approaches are not able to surpass MetaBO's performance.
Table 2: Optimal parameters of GMM-UCB and EPS-GREEDY (determined on the test set).
w
ncoope
e
Branin
0.22
3
linear schedule
Goldstein-Price
0.22
1
0.55
Hartmann-3 0.11
2
linear schedule
t T= 30. To tune w for GMM-UCB we tested 10 linearly spaced points in [0.0, 1.0] as well as a schedule which reduces from 1.0 to 0.0 over the course of one episode. Furthermore, we tested numbers of MMMM-components ncomp E {1, 2, 3, 4, 5.5.. Similarly, for EES-GREEDY we tested e on 10 linearly spaced points in [0.0, 1.0) and also evaluated a schedule which reduces from 1.0 to 0.0 over an episode.
In Fig. we display the performance of GMM-UCB and EPS-GREEDY on the global optimization benchmark functions Branin, GolddteinnPPrict and Hartmann-3 with the optimal parameter config- urations (ef. Tab. 2 and with M/ 50 source tasks. MetaBO outperforms both GMM-UCB and EPS-GREEDY which provides additional evidence that neural AFs learn representations which go beyond simple combination of standard AFs with a prior over D.
A.2 DEPENDENCE ON THE NUMBER OF SOURCE TASKS
We argued in the main part of this paper that one main advantage of MetaBO over existing transfer learning methods for BO is its ability to process a very large amount of source data because it does not store all available data in GP models (in contrast to TAF) but rather accumulates the data in the neural AF weights. For tasks where source data is abundant (c.g., when it comes from simulations, cf. Fig. 3) this frees the user from having to select a small subset of representative source tasks by hand, which can be intricate or even impossible for complex tasks. addition, we showed in our experiments that MetaBO's 's applicability is not restricted to such cases, but that it also performs favourably with the same amount of source data as presented to the baseline methods on tasks which do not require a very large amount of source data to be solved efficiently (cf. Figs. 2
In Fig. 9 we provide further evidence for this aspect by plotting the performance of MetaBO for different numbers M of source tasks on the Branin function and on functions from the simulation of the Furuta pendulum stabilization task. The results indicate that on the Branin function a small number of source tasks is already sufficient obtain strong optimization performance. In contrast, the more complex stabilization task requires a much larger amount of source data to be solved reliably.
We emphasize that MetaBO's evaluation runtime does not depend the number M of source tasks because a neural AF evaluation only requires one forward pass through a neural AF of fixed size. Therefore, it scales well to the regime of abundant source data. In contrast, TAF-ME's runtime scales
17

---

Published as a conference paper at ICLR 2020
Number of steps to crgret ihreshold
Number of steps to regret threshold
10
10a
10a
10
10*
Number of source
Number source tisks
(a) Branin function
(b) Furuta pendulum in simulation
Figure 9: Dependence of MetaBO's performance on the number of source tasks provided during training on the Branin function (cf. Fig. 2(a) and on the stabilization task for the Furuta pendulum in simulation (cf. Fig.3((a) {. We show the number of steps MetaBO requires reach given performance in terms of median regret over 100 test functions in dependence of the number M of source tasks. As in the main part of this paper, we chose a constant budget T = 30 on the Branin function and of T = 50 on the stabilization task. The dashed red line indicates the number of source tasks seen by the full version of MetaBO (a new function is sampled from the training distribution at the beginning of each optimization episode) at the point of convergence of meta- veetatraiinigg For the Branin function we chose the regret threshold R = 10 -3 which corresponds to the median final performance of TAF after / 30 steps as presented in the main part of this paper (Fig. 2(a). For the Furuta stabilization task, we chose the regret threshold R 1.0, which corresponds approximately to the regret that has be reached in simulation allow stabilization on the real system. The results show that on the Branin function already small number of source tasks is enough to obtain a powerful optimization strategy. In contrast, neural AFs trained on the more complex simulation-to-- task benefit from MetaBO's ss ability to process a very large amount of source tasks.
Table 3: Comparison of evaluation runtimes per BO episode with budget = 30 in s for various AFs, averaged over 10 BO runs. We show MetaBO's runtime for M 50 source tasks well for the full version (where a new function sampled from the training distribution in each BO run). For TAF, we indicate M and the number N of data points per source task by TAF-ME-M-N and TAF-R-M-N. Note that the absolute figures of the reported runtimes obviously depend on the hardware architecture used for the evaluation.
Branin
Goldstein-Price
Hartmann-3
EI
0.13
0.13
0.16
MetaBO-50
0.60
0.55
0.82
MetaBO-full
0.62
0.59
0.81
TAF-ME-50-100
13
14
24
TAF-ME-50-200
29
35
TAF-ME-100-100
17
19
30
TAF-ME-1100-20
47
49
65
TAF-R-50-
R-55--0000
50
50
56
TAF-R-50-200
61
60
69
100
100
110
PAF-R-100-22
120
120
140
linearly in the number M of source tasks and quadratically in the number N of data points per source task, while TAF-R shows an even stronger dependence on M due to the computation of the pairwise ranks. We underline this scaling behavior by presenting measured evaluation runtimes in Tab.
18

---

Published as a conference paper at ICLR 2020
0.5
20
aucep to regrer thresteld
#tsepp regret threshrold
0.5
25
shoope to regret thresbolo
0.6
0.6
0.7
0.7
0.7
20
0.8
18
walling
08
0.8
15
1.0
1.0
1.1
10
1.1
10
1.2
1.2
1.3
1.3
1.4
14
1.4
0.0
0,5
0.35
0.45
0.05
044
0:45
0,0
0.0
03
Irunslation
Iranclation
Iranslation
(a) Branin (D
(b) Golistein-Price (n
Hartmann-3
Figure 10: Generalization of neural AFs functions outside of the training distribution (translations E [-1.1, scalings s E [0.9, 1.1], red square) on Branin, Goldstein-Pries and Hartmann-3. We evaluated the neural AFs on 100 test distributions with disjoint ranges of transla- tions and scalings, each corresponding to one tile of the heatmap. The r- and y-labels of each tile denote the lower bounds of the translations f and scalings of the respective test distribution from which the parameters were sampled uniformly (for each dimension we sampled the translation and its sign independently). The color encodes the number of optimization steps required reach given regret threshold. White tiles indicate that this threshold could not be reached withtin T = 30 optimization steps. The regret threshold was fixed for each function separately: we set it to the 1%-percentile of the set of regrets corresponding to function evaluations on Sobol grid of one million points the domain of the original objective functions.
A.3 GENERALIZATION BEHAVIOR
As described in the main part of this paper, MetaBO's ' primary use case is transfer learning, i.e., to speed up optimization on target functions similar to the source objective functions. Put differently, we are mainly interested in MetaBO's performance on unseen functions drawn from the training distribution. Nevertheless, studying MetaBO's generalization performance to functions outside of the training distribution can give interesting insights into the nature the tasks we considered in the main part. Therefore, present study of MetaBO's generalization performance on the global optimization benchmark functions (Fig. 10 as well as on the simulation-to-Pr experiment (Fig. [T]
The results the simulation-to-ral task show that the neural AF generalizes better to heavy and long than to lightweight and short pendula. We suppose that this result is related to the fact that lightweight and short pendula show much faster dynamics due to their small moments of inertia than heavier and longer ones and are thus much harder to stabilize. Put more precisely, the change of the optimization landscape is much than in the other direction. Similar conclusions can be drawn for the translated and scaled global optimization benchmark functions.
pronounced when moving to lighter and smaller pendula
A.4 FULL SET OF RESULTS FROM MAIN PART
Global Optimization Benchmark Functions We provide the full set of results for the experiment on the global optimization benchmark functions. Fig. 12 we also include results for TAF with M = 20, showing that TAF's performance does not necessarily increase with more source data.
Simulation-to-Rell Experiment We provide the full of results for the experiment on the global optimization benchmark functions, including the results for TAF-50, ef. Fig. [3
B EXPERIMENTAL DETAILS
To foster reproducibility, provide detailed explanation of the settings used in our experiments and make source code available online
19

---

Published as a conference paper at ICLR 2020
Number of steps to regret threshold
Intended use case
25
20
10
0.5
0.7
0.9
1.1
1.3
1.5
1.7
1.9
Factor f on measured physical parameters
Figure 11: Generalization of neural AFs to functions outside of the training distribution (75% to 125% of measured physical parameters, red square) on the simulation-to-real task. We eval- uated neural AFs test distributions with disjoint ranges of physical parameters (masses and lengths of the pendulum and arm). We sampled each physical parameter p; uniformly on 0.9 corresponds the interval contain-
[J me, (J + 0.2) Pi, measured) Therefore, ing the measured parameters. We plot f on the r-axis and the number of steps required to reach a regret threshold of R 1.0 on the y-axis. Following our experience, this corresponds approximately to the regret that has to be reached in simulation to allow stabilization on the real system. We emphasize that the intended use case of MetaBO is on systems inside of the training distribution marked in red, as this distribution is chosen such that the true parameters are located inside of it with high confidence when taking into account the measurement uncertainty. Note that for small the system becomes very hard to stabilize (lightweight and short pendula) such that the optimization landscape differs significantly from the training distribution, which why the regret threshold cannot be reached within 30 steps for f < 0.5.
MesalO
TAR-R-00
TAR-M-50
10P
10
10"
simple regret
simplic sogret
10-7
10-2
10
12
18
24
12
(a) Branin (D 2)
(b) Goldstein-Price (D = 2)
(c) Hartmann-3 (D 3)
Figure 12: Performance on three global optimization benchmark functions with random translations
sampled uniformly from [-0.1, 0.1] and scalings from [0.9, 1.1]. To test TAF's performance, we randomly picked M source tasks from this function class and evaluated both the ranking-based version (TAF-R-M) and the mixture-of--expers version (TAF-ME-M) We show results for M e {20, 50}. Note that TAF's performance does not necessarily increase with more source data. We trained MetaBO on the same set of source tasks TAF-50 (MetaBO-50)). In contrast to TAF, MetaBO can also be trained without manually restricting the set of available source tasks. The corresponding results are labelled "MetaBO". MetaBO outperformed El by clear margin, especially in early stages of the optimization. After few steps used to identify the specific instance the objective function, MetaBO also outperforms both flavors of TAF over wide ranges of the optimization budget.
20

---

Published as a conference paper at ICLR 2020
MesBO (ours)
McuBO-30 (oun)
H1
TAF-8-30
TAF-R-00
TAF-ME-SO
TAF-ME-IIO
4.0
simple regrett
2,0
simple
2.0
0.0
0.0
10
20
30
40
50
5
10
15
20
25
(a) Evaluation in simulation.
(b) Evaluation on hardware in (c).
(c) Exp. setup."
Performance on Aimulation-to-real task (ef. text). MetaBO and TAF used source data from cheap numerical simulation. (a) Performance on an extended training set in simulation. (b) Transfer to the hardware depicted in (c), averaged over ten BO runs. MetaBO learned robust neural AFs with very strong early-time performance and online adaption to the target objectives, which reliably yielded stabilizing controllers after less than ten BO iterations while TAF-ME-50. TAF-ME-100, TAF-R-50, TAF-R-100, and El explore too heavily. Comparing the results for MetaBO and MetaBO-50 in simulation, we observe that MetaBO benefits from its ability to learn from the whole set of available source data, while TAF's applicability is restricted to comparably small number of source tasks.
Figure 13:
B.1 GENERAL IMPLEMENTATION DETAILS
In what follows, we explain all hyperparameters used in our experiments and summarize them in Tab. We emphasize that we used the same MetaBO hyperparameters for all our experiments, making our method easily applicable in practice.
We used the implementation GPy (GPy, 2012) with squared- exponential kernels (Matern-5/2 kernels for the corresponding experiments on general function classes) with automatic relevance determination and a Gaussian noise model and tuned the corre- sponding hyperparameters (noise variance, kernel lengthscales, kernel signal variance) offline by fitting a GP to the objective functions in the training and test sets using type-2 maximum likelihood. We also used the resulting hyperparameters for the source GPs of TAF. We emphasize that our method is fully compatible with other (online) hyperparameter optimization techniques, which we did not use in our experiments to arrive a consistent and fair comparison with as few confounding factors as possible.
Gaussian Process Surrogate Models
As standard, we used the parameter-fret version of EI. For TAF, we follow Wistubal et al. (208)) and evaluate both the ranking- maasid (TAF-R) as well as the product-of-ex ME) versions. We detail the specific choices for the number of source tasks M and the number of datapoints NTAF contained in each source GP in the main part of this paper.
Baseline AFs
(TAF-
For EI we used the midpoint of the optimization domain D as initial design. For TAF we did not use an initial design it utilizes the information contained in the source tasks to warmstart BO. Note that MetaBO also works without any initial design.
Maximization of the AFs Our method is fully compatible with any state-of-the-are method for maximizing AFs. In particular our neural AFs can be optimized using cradient-based techniques. We chose switch off any confounding factors related AF maximization and used hierarchical gridding approach for all evaluations as well as during training of MetaBO. For the experiments with continuous domains D. ie. all experiments except the HPO task, we first put multistart Sobol grid with NMS points over the whole optimization domain and evaluated the AF on this grid. Afterwards, we implemented local searches from the k maximal evaluations via centering l local Sohol grids with NLs points, each spanning approximately one "unit cell" of the multistart grid, around the k maximal evaluations. The AF maximum is taken to be the maximal evaluation of the AF on these & Sobol grids. For the HPO task, the AF maximum can be determined exactly because the domain is discrete.
Reinforcement Learning Method Optimization (PPO) (Schulman et al. 2017) the algorithm to train the neural AF
We use the trust trust-region policy gradient method Proximal Policy
21

---

Published as a conference paper at ICLR 2020
Table 4: Parameters of the MetaBO framework used in experiments.
Description
Value in experiments
BO/AF parameters
Cardinality NMS of multistart grid
Branin, Goldstein
1000
Hartmann-3
2000
Simulation-to-rcal
10000
GPs (D 1. 2.
500, 1000, 2000, 3000, 4000
Cardinality NLs of local search grid
NMs
Number of multistarts
MetaBO parameters
Cardinality of Egiohal
NMS
Cardinality of Elocal,t
k
Neural AF architecture
200 - 200 200 - 2000, relu activations
PPO parameters (Schulman et al.
2017)
Batch size
1200
Number of epochs
Number of minibatches
20
Adam learning rate
1-10 10
CPI-loss clipping parameter
0.15
Value network architecture
200 200 - 200 200, relu activations
Value coefficient in loss function
1.0
Entropy coefficient in loss function
0.01
Discount factor y
0.98
GAE-λ (Schulman al. 2015
0.98
Reward Function If the true maximum of the objective functions is not known at training time, we compute R2 with respect an approximate maximum and define the reward to be given by r; = - This is the case for the experiment on general function classes (GP samples) where we used grid search approximate the maxinum as well as for the simulation-to- -o-rell task on the Furuta pendulum where we used the performance of linear quadratic regulator (LQR) controller as an approximate maximum. For the experiments on the global optimization benchmark functions as well as on the HPO tasks, we do know the exact value of the global optimum. In these cases, we use a logarithmic transformation of the simple regret, i.e., rt = log R2 as the reward signal. Note that we also consistently plot the logarithmic simple regret in our evaluations for these cases.
Neural AF Architecture We used multi-layer perceptrons with delt-activvtiin functions and four hidden layers with 200 units each represent the neural AFs.
Value Function Network To reduce the variance of the gradient estimates for PPO, value function V, (s:), i.e., an estimator for the expected cumulative reward from state st, can be employed (Schulman et al. 2015). In this context, the optimization step and the budget T are particularly informative features, as for given sampling strategy on a given function class they allow quite reliable predictions of future regrets. Thus, we propose to use separate neural network to learn a value function of the form V, (s,) = V (t, T). We used an MLP with relu-activations and four hidden layers with 200 units each for the value network.
Computation Time For training MetaBO, we employed ten parallel CPU-workers to record the data batches and one GPU to perform the policy updates. Depending on the complexity of the objective function evaluations, training neural AF for a given function class took between approximately 30 min and 10 on this moderately complex architecture.
22


Reviews:
[{"review_id": "SkgpLghVYr", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The authors present MetaBO, which uses reinforcement learning to meta-learn the acquisition function (AF) for Bayesian Optimization (BO) instead of using a standard constant AF. The authors shows that MetaBO enables transferring knowledge between tasks and increasing sample efficiency on new tasks. The paper is mostly clearly written and I am not aware of existing work on meta-learning the AF for BO. However, the approach is related to Chen et al, which is cited in the text but not used as a baseline. It is also not shown clearly enough how the performance of MetaBO depends on the number of training tasks and distance between training and test tasks. I therefore consider the paper as borderline.\n\nMajor comments\n=============\n1. The presented approach is very similar to Chen et al, which is discussed in the related work section but not used as a baseline. Although Chen et al assumed that f(x) is differentiable, their approach can be easily generalized to non-differentiable functions by using RL as Chen et al discussed in the last paragraph of section 2.1. Chen et al does not depend on a GP and is therefore more scalable. The source code is publicly available (https://github.com/deepmind/learning-to-learn) and you can also adapt your implementation by removing the GP part.\n\n2. Global Optimization Benchmark Functions: How does the performance of MetaBO depend on the number of training samples (number of training tasks times the budget T)?\n\n3. Figure 3: How does MetaBO generalizes to functions that are translated and scaled at the same time? This can be visualized as a heatmap with the scaling and translation on the x and y axis, and using the color to show the number of steps to reach a certain reward. How does the generalization performance depend on the noise level, where the noise can be sampled from standard normal distribution? Why does EI perform better if the function is translated more?\n\n4. Simulation-to-Real task: How does the generalization performance of MetaBO depend on the distance between training and source tasks (x-axis: distance; y-axis: steps to reach a certain reward)? You sampled test tasks 10%-200% around the true parameters. Test tasks can therefore have identical or similar parameters than training tasks.\n\n5. Simulation-to-Real task: How does the performance depend on the number of training tasks (x-axis: # training tasks; y-axis: steps to reach a certain performance)?\n\nMinor comments\n=============\n6. Section 1, 2nd paragraph: The performance of BO also depends on the GP kernel and kernel hyper-parameters, not only the AF. Please mention this. Similarly, ‘no need to calibrate any hyperparameter’ in section 4 ignores GP hyper-parameters. Please clarify.\n\n7. Section 2, 4th paragraph: A Neural Process (https://arxiv.org/abs/1807.01622) is another scalable alternative to a GP. Please cite.\n\n8. Section 3, 2nd paragraph: Please cite standard AFs such as EI, PI, UCP. \n\n9. Section 4, last paragraph before ‘Training procedure’. The state s_t is undefined at this point. This section misses a clear description of the state, reward, and transition function of the MDB. Does the state s_t take previous function evaluations into account (e.g. via a RNN state), or only \\mu and \\sigma at the current step t? Does the state include the time step as described in the text and in the section about the value network in the appendix but not in table 1.\n\n10. Section 4, ‘the state corresponds to the entire functions’.  It only depends on the first two moments (and the time step t?).\n\n11. Section 4: replace ‘not to be available’ by ‘unavailable’.\n\n12. Section 4: reference or describe ‘Sobol grid’.\n\n13. Section 4: The approach to maximize the AF on grid points does not scale to high-dimensional search spaces. Please also clarify how global and local grid points were chosen. In particular, ‘local maximization’ is unclear. Also, ‘cheap approximation’ of the global maximum of f(x) is infeasible if the search space is high-dimensional. \n\n14. Please move figure 3 above figure 4.", "extra_scores": {"experience_assessment": "I have published one or two papers in this area.", "review_assessment:_checking_correctness_of_experiments": "I carefully checked the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper thoroughly.", "review_assessment:_checking_correctness_of_derivations_and_theory": "N/A"}, "extra_text": {}}, {"review_id": "S1xux0YRKr", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "\nThis paper proposes a framework for meta learning neural acquisition functions for the Bayesian optimization of various underivable functions. The neural acquisition functions are learned using proximal policy optimization in an outer loop on different problems on the same domain, and the learned acquisition function can be deployed at test time in a practically vanilla Bayesian optimization procedure. The authors demonstrate the performance of the method through benchmarks on four problems.\n\nI recommend that this paper be accepted for publication. The paper is well written and it proposes a novel direction for research. However, I think that the authors should look further inside their newly designed acquisition functions, not merely treat them as black boxes. Find below some questions and comments.\n\n\nDue to the inclusion of the sample position x in the state tuple, I am curious as to what the authors think is the difference between their method and a learning-to-learn type of approach. Is the acquisition function learning to favor specific zones in the search space based on previous experiments? Some more experiments or insights on this would be useful to better understand what makes this method succesful.\n\nWhy was a categorical distribution used for the policy? These samples are located in D, aren't you getting rid of information by assuming they are completely independent? Aren't you also biasing the distribution by adding the local maxima to the set of ξ (Xi)?\n\nAlso, the right-most block in Figure 1 shows a continuous probability distribution, which is incorrect. If the distribution is indeed categorical, there is no continuity between points.\n\n\nMinor mistakes:\n\n- page 5, paragraph 3: \"This choice does not penalize explorative evaluations which do not yield and immediate improvement\" should read \"an immediate improvement\"\n- Figure 4b, MetaBO-50 is missing\n\n***********\nPost rebuttal:\n************\n\nI have read the other reviews and the various replies by the authors. I'd say you did a good job in answering most questions and added a lot of valuable information in the appendices. I maintain my score.", "extra_scores": {"experience_assessment": "I have published one or two papers in this area.", "review_assessment:_checking_correctness_of_experiments": "I carefully checked the experiments.", "review_assessment:_thoroughness_in_paper_reading": "N/A", "review_assessment:_checking_correctness_of_derivations_and_theory": "N/A"}, "extra_text": {}}, {"review_id": "BJl7k5fPtH", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Summary: The authors propose a meta-learning based alternative to standard acquisition functions (AFs), whereby a pretrained neural network outputs acquisition values as a function of hand-chosen features. These neural acquisition functions (NAFs) are trained on sets of related tasks using standard RL methods and, subsequently, employed as drop-in replacements for vanilla AFs at test-time. \n\n\nFeedback:\nOverall, the proposed method makes sense and would benefit from further experimental ablation. Using RL to automatically derive (N)AFs is a nice change of pace from the hand-crafted heuristics that dominate BO. I like the ideas at play here and hope that you will convince me to amend my score.\n\nResults on synthetic functions presented in the body of the paper demonstrate that NAF outperforms, e.g., EI when transferring between homogenous tasks. In contrast, results when transferring between relatively heterogenous functions (Fig. 9) indicate that the aforementioned performance gain reflect NAFs ability to specialize. Two things remain unclear however: \n    a. What types of regularity are NAFs able to exploit?\n    b. How quickly do NAFs benefits fall off as tasks become increasingly heterogenous?\n\n\nRegarding (a), I am not yet convinced that NAFs learn representations that go beyond standard AFs combined with a prior over $x$. To help test this hypothesis, here is a sketch of a simple baseline algorithm:\n  1. Fit, e.g., a Gaussian Mixture Model to the top $k=1$ designs $x^{*}_{i}$ on observed tasks $i \\in [1, N]$, \n  2. Given a new task $f_{j}$, let log-likelihood $GMM(x)$ act as a 'prior' of sorts on $x$ \n  3. Use cross-validation to tune the scalar parameter $w$ of a new AF defined as the convex combination:\n\n        GMM-UCB(x_{k}) = w * GMM(x_{k}) + (1 - w) * UCB(x_{k})\n                                       = w * GMM(x_{k}) + (1 - w) * [\\mu_{k} + \\sqrt{\\beta} * \\sigma_{k}].\n\nI suggest using UCB both because NAF could easily learn it from its inputs and because EI values often decay dramatically over the course of BO (I usually set UCB's confidence parameter to a fixed value $\\beta = 2$). \n\nFurther simplifying this idea, you could instead use an $\\epsilon$-greedy style heuristic that, with probability $\\epsilon$, samples without replacement from the set of historical minimizers and otherwise uses a standard AF. These baselines are comparatively straightforward and easily interpreted, so I hope that you will consider adding something along these lines.\n\n\nAdditionally, here are some questions/suggests to help probe (a-b):\n  1. Another baseline: EI with multi-task GP? The cubic scaling should be fine for, e.g., 'xxx-20' multi-task variants.\n  2. Extend experiments on functions drawn from GP priors (Fig 9):\n      i. How does homogeneity (as enforced via the GP hyperprior) impact performance when transferring knowledge?\n      ii. Rate of convergence suggests sampled tasks may be too easy; consider using Matern-5/2 and smaller lengthscales [*].\n  3. What happens if you expand the task augmentation process to further include, e.g., flips and rotations?\n  4. How do 'dimension-agnostic' versions of NAF (where $x$ is excluded from its input) perform on other synthetic tasks?\n  5. Visualizing NAF (or the search strategies it produces) would be useful for building intuition.\n  6. How were NAF input features chosen? Were alternatives such as also passing the 'best seen' value, considered?\n  7. How easy to use are NAFs in comparison to alternative AFs (both in terms of training and test-time maximization)?\n  8. Please report regret in log-scale (in appendix); currently, it is hard to tell what is going on in some places. Similarly, the tracked regret level in Figures 3 & 7 changes between tasks without explanation?\n\n\nIn summary, I genuinely want NAF to succeed but am not yet convinced of its performance. If you can provide empirical results to help extinguish my doubts, I will gladly change my assessment.\n\n\nNitpicks, Spelling, & Grammar: \n  - Some minor spelling and/or grammatical error, but the paper reads fairly well.\n  - On [Chen et al., 2017]:  To the best of my knowledge, these RNN-based methods only require the gradient of the loss function. For example, using GP-based EI as the training signal only requires differentiating through EI + GP rather than through the target function $f$. Similarly, in cases where gradients are not available, the authors elude to use of RL algorithms such as REINFORCE.\n\n[*] For Matern-5/2, just change the prior on your basis functions' weight parameters (https://github.com/metabo-iclr2020/MetaBO/blob/master/metabo/environment/objectives.py#L295) from standard normal to multivariate-t with 5 degrees of freedom.", "extra_scores": {"experience_assessment": "I have published one or two papers in this area.", "review_assessment:_checking_correctness_of_experiments": "I carefully checked the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper thoroughly.", "review_assessment:_checking_correctness_of_derivations_and_theory": "N/A"}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper explores the idea of using meta-learning for acquisition functions. It is an interesting and novel research direction with promising results. 

The paper could be strengthened by adding more insights about the new acquisition function and performing more comparisons e.g. to Chen et al. 2017. But in any case, the current form of the paper should already be of high interest to the community

Decision: Accept (Spotlight)