Reinforced Few-Shot Acquisition Function Learning for Bayesian Optimization
Authors: Bing-Jing Hsieh
Ping-Chun Hsieh
Xi Liu
Venue: NeurIPS 2021 Poster (2021)
Keywords: Bayesian optimization
acquisition function
few-shot learning
reinforcement learning

Abstract: Bayesian optimization (BO) conventionally relies on handcrafted acquisition functions (AFs) to sequentially determine the sample points. However, it has been widely observed in practice that the best-performing AF in terms of regret can vary significantly under different types of black-box functions. It has remained a challenge to design one AF that can attain the best performance over a wide variety of black-box functions. This paper aims to attack this challenge through the perspective of reinforced few-shot AF learning (FSAF). Specifically, we first connect the notion of AFs with Q-functions and view a deep Q-network (DQN) as a surrogate differentiable AF. While it serves as a natural idea to combine DQN and an existing few-shot learning method, we identify that such a direct combination does not perform well due to severe overfitting, which is particularly critical in BO due to the need of a versatile sampling policy. To address this, we present a Bayesian variant of DQN with the following three features: (i) It learns a distribution of Q-networks as AFs based on the Kullback-Leibler regularization framework. This inherently provides the uncertainty required in sampling for BO and mitigates overfitting. (ii) For the prior of the Bayesian DQN, we propose to use a demo policy induced by an off-the-shelf AF for better training stability. (iii) On the meta-level, we leverage the meta-loss of Bayesian model-agnostic meta-learning, which serves as a natural companion to the proposed FSAF. Moreover, with the proper design of the Q-networks, FSAF is general-purpose in that it is agnostic to the dimension and the cardinality of the input domain. Through extensive experiments, we demonstrate that the FSAF achieves comparable or better regrets than the state-of-the-art benchmarks on a wide variety of synthetic and real-world test functions.

Reinforced Few-Shot Acquisition Function Learning for Bayesian Optimization
Bing-Jing Hsieh', Ping-Chun Hsieh', Xi Liu²
DDepartment of Computer Science, National Yang Ming Chiao Tung University, Hsinchu, Taiwan AApplied Machine Learning, Facebook AI, Menlo Park, CA, USA (bingjing2000. cs08g, pinghsieh) edu. tw, xlw.llflli@. Fliu1efb.ccm com
Abstract
Bayesian optimization (BO) conventionally relies on handcrafted acquisition func- tions (AFs) sequentially determine the sample points. However, it has been widely observed in practice that the best- best-performing AF in terms of regret can vary significantly under different types of black-box functions. It has remained a challenge to design one AF that can attain the best performance over wide variety of black-box functions. This paper aims to attack this challenge through the perspective of reinforced few-shot AF learning (FSAF). Specifically, we first connect the notion of AFs with Q-functions and view deep Q-network (DQN) as surrogate differentiable AF. While it serves a natural idea to combine DQN and an existing few-shot learning method, we identify that such a direct combination does not perform well due to severe overfitting, which is particularly critical in BO due to the need of a versatile sampling policy. To address this, we present a Bayesian variant of DQN with the following three features: (i) It learns a distribu- tion of Q-networks as AFs based on the Bellback-Leibler regularization framework. This inherently provides the uncertainty required in sampling for BO and mitigates overfitting. (ii) For the prior of the Bayesian DQN, we propose to use a demo policy induced by off-the-sheff AF for better training stability. (iii) On the meta-level, we leverage the meta-loss of Bayesian aggnotic meta-learning, which serves natural companion the proposed FSAF, Moreover, with the proper design of the Q-networks, FSAF is general-purpose in that it agnostic to the dimension and demon-
the cardinality of the input domain. Through extensive experiments, strate that the FSAF achieves comparable or better regrets than the state-of- state-of-the-atr benchmarks on a wide variety of synthetic and real-world test functions.
Introduction
Bayesian optimization (BO) has served as a powerful and popular framework for global optimization in many real-world tasks, such as hyperparameter tuning [14,, robot control (5), automatic material design (6-8), etc. To search for global optima under small sampling budget and potentially noisy observations, BO imposes Gaussian process (GP) prior on the unknown black-box function and continually updates the posterior as more samples are collected. BO relies acquisition functions (AFs) to determine the sample location, i.e., those with larger AF values are prioritized than those with smaller ones. AFs are often designed to capture the trade-off between exploration and exploitation of the global optima. The design of AFs has been extensively studied from various perspectives, such optimism in the face of uncertainty (e.g., GP-UCB 9 )) optimizing informatiion-hhereet metrics (e.g., entropy search methods (10412) ) and maximizing one-step improvement (e.g., expected improvement El [ 14). ). As result, AFs are often handerafted according different perspectives of the trade-off, and the best- pest-performing AFs can vary significantly under diffeeent types of black-box
35th Conference on Neural Information Processing Systems (NcurIPS 2021).

---

This phenomenon is repeatedly observed in our experiments in Section 4 Therefore, one critical issue in BO to design an AF that can adapt variety of black-box functions.
functions [TI.
To achieve better adaptability to new tasks in BO, recent works propose to leverage mela-data, the data previously collected from similar tasks [15 For example, in the context of hyperparameter optimization under a specific dataset, the meta-data could come from the evaluation of previous hyperparameter configurations for the same learning model over any other related dataset. In (T5), meta-data is used to fine-tune the initialization of the GP parameters and thereby achieves better GP model selection for each specific task. However, the potential benefit of using meta-data for more efficient exploration via AFs is not explored. On the other hand, in [7]. meta-data is split into multiple subsets and then used construct a transferable acquisition function based on some off-the-shelf AF (e.g. EI) and an ensemble of GP models. Each of the GP models is learned over a separate subset of the meta-data. However, to achieve effective knowledge transfer, this approach would require sufficiently large amount of meta-data, which significantly limits its practical use. As a result, there remains a critical unexplored challenge in BO: how to design an AF that can effectively adapt to wide variety of back-box functions given only small amount of meta-data?
Training Curve
Ackley
Dixon-Price 7D
Powell 10D
1.-
1.4
DON MAML
DON MAML
Random
Simple Regret
1.00
Simple Begret.
Simple Regret
0.75
2.2
0.00
0.0
103 Iteration
202
30c
20
do
40
60
80
2b
65
Number of Samples
Number Samples
Number of Samples
(a)
(b)
(c)
(d)
Figure 1: An illustration of the overfitting issue of DQN+MAML trained with GP functions: (a) Training curves of DQN+MAML (As EI and Random do not require any training, their lines are flat); (b)-(d) Average simple regrets of DQN+MAML and EI in testing under benchmark functions.
To tackle this challenge, we propose to rethink the use of meta-data in BO through the lens of few-shot acquisition function (FSAF) learning. Specifically, our goal is to learn an initial AF model that allows few-shot fast adaptation to each specific task during evaluation. Inspired by the similarity between AFs and functions, we use deep Q-network (DQN) Q-network would output an indicator for each candidate sample point given its posterior mean and variance well other related information. Given the parametric nature and the differentiability of a Q-network, it is natural to leverage optimization-basee few-shot learning approaches, such as model-agnostic meta-learning (MAML) [18. for the training of few-shot AFs. Despite this natural connection, we find that direct combination of standard DQN and MAML (DQN+MAML) is prone to overfitting, as illustrated by the comparison of training and testing results. Note that in Figure (with detailed configuration in Appendix ( DQN+MAML achieves better regret than EI on the training set but suffers from much higher regret during testing. We hypothesize this is because both DQN and MAML are prone to overfitting [19 This issue could be particularly critical in BO due to the uncertainty required in adaptive sampling. Based on our findings and inspired by (23), we propose a Bayesian variant of DQN with the following three salient features: (i) The Bayesian DQN learns a distribution of parameters of DQN based on the Kullback- (KI.) regularization framework. Through this, the problem of minimizing the Q-learning in DQN is converted into a Bayesian inference problem. (ii) For the prior of the Bayesian DQN, we propose to use demonstration policy induced by an off-the-shelf AF to further stabilize the training process; (iii) We then use the chaser meta-loss in (20), which serves as a natural companion to the proposed Bayesian DQN. As shown by the experimental results in Section 4 the proposed design effectively mitigates overfitting and achieves good generalization under various black-box functions. Moreover, FSAF is trained solely synthetic GP functions (with details in Appendix (B) and is able to adapt to a broad variety of functions based only on a small amount of metadata (which could be costly to generate). With the proper design of the Q-networks, FSAF is general-purpose in the sense that it is agnostic to both the dimension and the cardinality of the input domain.
surrogate differentiable AF, i.e., the
(TD) error
The main contributions of this paper can be summarized as follows:
2

---

We consider novel setting of fhw-shot acquisition function learning for BO and present the first few-shot acquisition function that can use small amount of meta-data achieve better task-specific exploration and thereby effectively adapt to wide variety of black-box functions. Inspired by the similarity between AFs and Q-functions, we view DQN as a parametric and differentiable AF and use as the base of our FSAF We identify the important overfitting issue in the direct combination of DQN and MAML and thereafter present a Bayesian variant of DQN that mitigates overfitting and enjoys stable training through a demo-based prior.
We extensively evaluate the proposed FSAF in a variety of tasks, including optimization benchmark functions, real-world datasets, and synthetic GP functions. We show that the proposed FSAF can indeed effectively adapt variety of tasks and outperform both the conventional benchmark AFs as well as the recent state-of-the-ar meta-learning BO methods.
2 Preliminaries
Our goal is to design a sampling policy to optimize a black-box function f X + R, where X C R denotes the compact domain of f. f is black-box in the sense that there are special structural properties (e.g., concavity or linearity) or derivatives (e.g., gradient or Hessian) about f available to the sampling policy. In each step i, the policy selects V4 E X and obtains noisy observation yt = f(x)) + Et, where e are i.i.d. zero-mean Gaussian noises. To evaluate a sampling policy, we define the simple regret as Rtgret(t) == maxzex (x=) gax I (I:.), which quantifies the best sample up to t. For convenience, we let J, == { y:) J denote the observations up to step
Bayesian optimization. To optimize f in a sample- of objective functions a GP prior, which is fully characterized by a mean function and a covariance function, and then determines the next sample based on the resulting posterior [4 25 In each step t, given F,, the posterior predictive distribution of cach E X is (x), of (z)), where (z) == [[f(=)) Fi] and (2) = (V[f(z) FiI)+ & can be derived in closed form. In this way, BO can be viewed as sequential decision making problem. However, it is typically difficult to obtain the exact optimal policy due to the curse of dimensionality (24. To obtain tractable policies, BO algorithms construct AFs V (x;; Ft). which resort to maximizing one-step look-ahead objectives based on F, and the posterior [24]. For example, El chooses the sample location based on the improvement made by the immediate next sample in expectation, i.e., (=; F.) = B(f(x) maxigige- [(x;) Ft), which enjoys a closed form in 14e (z), of (x)),, and maxi (x.))
manner, BO first imposes on the space
Meta-learning with few-shot fast adaptation. Meta-lcarning is a generic paradigm for generalizing the knowledge acquired during training to solving unseen tasks in the testing phase. In the few-shot setting, meta- meta-learning is typically achieved via bi-level framework: (i) On the upper level, the training algorithm is meant to determine a proper initial model with an aim to facilitating subsequent
task- susk-specific adaptation; (ii) On the lower level, given the initial model and the task of interest, a fast adaptation subroutine configured to fine-tune the initial model based on small amount of task-specific data. Specifically, during training, the learner finds model parameterized by 0 based on a collection of tasks T, where each task 7 is associated with a training set D.' and a validation set Drat. For any initial model parameters 0 and training set D of a task T, let M0, (O, be an algorithm that outputs the adapted model parameters by applying few-shot fast adaptation to 0 based on D.. The performance of the adapted model is evaluated on D by meta-loss function [(M D),), D.""). Accordingly, the overall training can be viewed as solving the following optimization problem:
0- := argmin Σ L(M(0, D"), ).")
TET
By properly configuring the loss function, the formulation in I is readily applicable to various learning problems, including supervised learning and RL. Note that the adaptation subroutine is typically chosen as taking one few gradient steps with respect to L(---) or some relevant loss function. For example, under the celebrated MAML [18), the adaptation subroutine is M(6 D")) = 0 V L(8, D"), where 77 denotes the learning rate.
Reinforcement learning and Q-function. Following the conventions of RL, we use 8t: drs and r, to denote the state, action, and reward obtained at each step 1. Let R and 7y be the reward function and the discount factor. The goal to find stationary randomized policy that maximizes the total expected discounted reward E| 2 y're]. To achieve this, given policy n, helper
oc:
3

---

function termed Q-function is defied as (=(s.a) == E| 7 R(s,,,,)sss, 5.00 = a: m|. The optimal Q-function can then be defined as (s, a) == max, Q" a), for each state-action pair. One fundamental property of the optimal Q- Q-function is the Bellman optimality equation, i.e., (*( = EE [r + - max,' (* a') 8t a, = a]. Then, the Bellman optimality operator can be defined by [B (s, a) == R(s, a) + y E. | Q(s', d')j. It is known that Q* is the unique fixed point of B. We leverage this fact to describe the proposed FSAF in Section 3.2
3 Few-Shot Acquisition Function
3.1 Deep Q-Network as Differentiable Parametric Acquisition Function
Based on the conceptual similarity between acquisition functions and Q-functions, in this section we present how to cast deep Q-network as a parametric and differentiable instance of acquisition function. To begin with, we consider the Q-network architecture with state and action representations as the input, as typically adopted by Q-learning for large action spaces (26).
State-action representation. In BO, an action corresponds to choosing one location to sample from the input domain X. and the state at each step can be fully captured by the collection of sampled points {(x,, )/)) However, this raw state representation appears problematic as its dimension depends on the number of observed sample points. Inspired by the acquisition functions, we leverage the posterior mean and variance the joint state-action representation for each candidate sample location. In addition, we include the best observation so far (defined as yr := argmax si 1/i) and the ratio between the current timestamp and total sampling budget T,, which reflects the sampling progress in BO. In summary, at each step f. the state-action representation of each x E X is designed to be 4-tuple (z), of (z), Vf f), which is agnostic to the dimension and cardinality of X.
Reward signal. To reflect the sampling progress, we define the reward ra as function of the simple regret, i.e., r+ (Regrett (t). where g R+ - IR is strictly decreasing function. Practical examples include g(2) - and g(=) = log z.
Remark One popular approach to construct a representation of fixed dimension is through embed- ding. From this viewpoint, the posterior mean and variance can be viewed natural embedding generated by GP inference in the context of BO. It is an interesting direction to extend the proposed design by constructing more general state and action representations via embedding techniques.
Remark 2 The main contribution of FSAF is to present the first few-shot learning based acquisition function for BO, and it is worth noting that FSAF is not the first approach that addresses BO through the lens of RI. as policy-based RI. has been adopted for solving BO in MetaBO (27)- As both FSAF and MetaBO address BO through the lens of RI, they share some common choices for the representation (c.g., (x) and (z)). Despite the similarity in the representation and the fact that using a good state-action representation is important in RL, we found that a good representation itself does not guarantee good regret performance. As shown by the results in Figure D DQN+MAML uses the same state-action representation as FSAF, but it still suffers from severe overfitting and poor regret. Accordingly, in this paper, we show that the key to achieving low regret under various black-box functions is the appropriate use of small amount of metadata through the design of few-shot adaptation in FSAF.
3.2 A Bayesian Perspective of Deep Q-Learning for Bayesian Optimization
One major challenge in designing an acquisition function for BO is to address the wide variability of black-box functions and accordingly achieve a favorable explore-cxploi trade-off for each task. To address this by deep Q-learning described in Section [.I1 instead of learning single Q-network as in the standard DQN (28), we propose to learn a distribution of Q-network parameters from a Bayesian inference perspective to achieve more robust exploration and more stable training. Inspired by (23, we adapt the regularized minimization problem to Q-learning in order to connect Q-learning and Bayesian inference as follows. Let C(#) be the cost function that depends the model parameter 0. Instead of finding a single model, the Bayesian approach finds distribution 4()) over 0 that minimizes the cost function augmented with a KL-regularization penalty, i.e.,
min (())
[C(0)] + aDxL (g || (0)
(2)
where go denotes prior distribution over and a is a weight factor of the penalty and ( ||) denotes the Kullback-Leible (KL) divergence between two distributions. Note that q()) essentially

---

induces distribution over the sampling policies тe, and accordingly qu can be interpreted as constructing a prior over the policies. By setting the derivative of the objective in 2 with respect to the measure induced by to be zero, one can verify that the optimal solution to is
1 Z exp
C(0) a
q (0())
(o()),
(3)
where Z is the normalizing factor. One immediate interpretation of (3) is that (0) can be viewed as the posterior distribution under the prior distribution (o()) and the likelihood exp(-C(0)/a), To adapt the KL-regularized minimization framework to value-based RL for BO, the proposed FSAF algorithm is built on the following design principles for C(0) and (0):
Use mean-squared TD error as the cost function: Recall from Section 2 that the optimal Q- function is the fixed point of the Bellman optimality backup operation. Hence, we have BQ = Q if and only if Q is the optimal Q-function. Based on this observation, one principled choice of C()) is the squared TD error under the operator B3 i.e., | - Q11-. Moreover, in practice, DQN typically incorporates a replay buffer Ro (termed the Q-replay buffer) as well as target Q-network to achieve better training stability (28). Therefore, we choose the cost function as
[ ((++>ymg max 0''','';88 - a() d'EA
((0) = [ss,,,,
(4)
where p denotes the underlying sample distribution of the replay buffer and 0- is the parame- ter of the target Q-network] In practice, the cost (()) is estimated by the empirical average over mini-batch D of samples (ss, a, r,, 8') drawn from the replay buffer, i.e., C(O) ~ Ć(0)= [D. E(sanns)) (rr + ma Q(5', al; ())) (s, a; 0))².
Construct an informative prior with the help of the existing acquisition functions: In 2 the KL-penalty with respect to a prior distribution is meant to encode prior domain knowledge as well as provide regularization that prevents the learned parameter from collapsing into a point estimate. One commonly-used choice is uniform prior (i.e., g(0) for some positive constant c), under which the KL-penalty reduces to the negative entropy of q. Given that BO is designed to optimize -xpensive-t--evaluaae functions, it is therefore preferred to use more informative prior for better sample efficiency. Based on the above, we propose to construct a prior with the help of a demo policy TD induced by existing popular AFs (e.g.. EI or PI), which inherently capture critical information structure of the GP posterior. Define a similarity indicator 8(TT, TD) of Ty and TD as
8(TO; #) == (es-p, [log(Fo(s,a)]]
where we slightly abuse the notation and let p denote the state distribution induced by the replay buffers. Since the term log(za(s, a)) in (5) is the log-likelihood of that the action of we matches that of TD at state 8, 8(TT, TD) reflects how similar the two policies are on average (with respect propose to design the prior (0) to be
to the state visitation distribution of me). Accordingly,
(o) x exp (ō(=0; m
(6)
As it is typically difficult directly evaluate 8(To; TD) in practice we construct another replay buffer RD (termed the demo replay buffer), which stores the state-action pairs produced under the state distribution o and the demo policy TD. and estimate d( AD) by the empirical average over mini-batch D' of state-action pairs drawn from the demo replay buffer, i.c., 8(TT, TD) RJ (TT,, TD)
p E(.,))E log(Te(s, a))).
Update the Q-networks by Stein variational gradient descent: The solution in (3) is typically intractable to evaluate and hence approximation is required. We leverage the Stein variational gradient descent (SVGD), which is general-purpose approach for Bayesian inference. Specifically, we build up N instances of Q-networks (also called particles in the context of the variational methods) and update the parameters via Bayesian inference. Let (f())) denote the parameters of the
'I 4 the cost function depends implicitly on the target network parameterized by 0 Despite this, as the target network is updated periodically from 2(-;;)), f), for notational convenience we do not make explicit the dependence of the cost function 0- in the notation C(O).
5

---

n-th Q-network and use ④ as shorthand of {0(1) Under SVGD [29] and the prior described in 6 the Stein variational gradient of each particle can be derived as
g (3)
g63 c( + (TTcc, ( k(B(0) g(m)) + Vac k(Ol), ((m)).
where -(..) is a kernel function. As mentioned above, practice C(0) and 0(TT, mp) are estimated by the corresponding empirical ((0) and o(TT, TD). respectively. Let g(m) (0) denote the estimated Stein variational gradient based on Č(Ø) and ö(TT, TD). Accordingly, the parameters of each Q-network are updated iteratively by SVGD as
g(x) + g(n) + n g(()
(8)
where 17 is the learning rate. For ease of notation, we use MSvGD D) denote the subroutine that applies one SVGD update of (8) to all Q-networks parameterized by θ based on some dataset
D. Note that the update scheme (8) serves as a natural candidate for the few-shot adaptation subroutine of the meta-learning framework described in Section 2 In Section 3.3
we will put
everything together and describe the full meta-learning algorithm of FSAF.
The presented Bayesian DQN bears some high-level resemblance to the prior works 33-- 33]. which are inspired by the classic principle of Thompson sampling for exploration. In [3 32) the Q-function is assumed to be linearly parameterized with a Gaussian prior on the parameters such that the posterior can be computed in closed form. On the other hand, without imposing the linearity assumption, 32] approximates the intractable posterior by maintaining an ensemble of Q-networks as practical heuristic of Thompson sampling to DQN. Different from 30 32), approach Bayesian DQN through the principled framework of KI. regularization for parametric Bayesian inference and solve it via SVGD, without any linearity assumption. [3 starts from an cntropy-regularizee formulation for Q-learning and assumes that the target Q-value is drawn from a Gaussian model to obtain a tractable posterior from the perspective of variational inference. By contrast, we do not rely on the Gaussian assumption and directly find the posterior by SVGD, and for more efficient training we consider prior induced by an acquisition function. More importantly, the presented Bayesian DQN naturally helps substantiate the few-shot learning framework described in Section for BO.
Remark 3
Remark 4 The regularized formulation in (2) has been extensively applied in the class of policy- based methods in RL. For example, the entropy-rogularized policy optimization (34) has been applied to enable "soft version" of policy iteration, which gives rise the popular soft Q-learning (35] and soft actor-critic algorithms [36]. Another example is the Stein variational policy gradient method 233, which connects the policy gradient methods with Bayesian inference. Different from the prior works, we take a different path to connect the value- based RL approach with Bayesian inference.
Remark 5 The similarity indicator defined in (5) has similar form as the loss term in some of the classic imitation learning algorithms. For example, given an expert policy Te, DAgger [37] is designed to find a policy x' that minimizes a surrogate loss E, (((ssTe)),, where ((.-) some loss function (c.g., 0-1 loss or hinge loss) that reflects the dissimilarity between T' and πT. Despite this high-level resemblance, one fundamental difference between (5) and imitation learning is that the demo policy TD is not a true expert in the sense that mimicking the behavior of TD is not the ultimate goal of the FSAF learning process. Instead, the goal of FSAF is to learn a policy that can better adapt to new tasks and thereby outperform the existing AFs in various domains. The penalty defined in (6) only meant to provide some prior domain knowledge to achieve more sample-efficient training. We further validate this design by providing training curves in Section 4
3.3 Meta-Learning via Bayesian MAMI
Based on the Bayesian DQN design in Section 3.2 the natural way to substantiate the meta-learning framework in (1) is to leverage the Bayesian variant of MAML (20 In the context of BO, each task typically corresponds to optimizing some type black-box functions (e.g., GP functions from an RBF kernel with some lengthscale). FSAF implements the bi-level framework of (I) as follows: (i) On the lower level, for each task т. FSAF enforces few-shot fast adaptation by taking of SVGD as described in (8) and thereafter obtains the fast-adapted parameters denoted by Or,k; (ii) On the upper level, for each task т, FSAF computes a meta-loss that reflects the dissimilarity between the approximated posterior induced by O, K and the true posterior distribution in 6 As the
steps

---

true posterior is not available, one practical solution is to approximate the true posterior by taking S additional SVGD gradient steps based on rr,K and obtaining a surrogate denoted by G 20. This design can be justified by the fact that G becomes better approximation for the true posterior as 5 increases due to the nature of SVGD. For any two collections of partieles O' = {0)()) } and
@" = {0"(")), j. define D(O', (") ;= Σ x== ||0r(()) - 0"(0)) 11 (called chaser loss in 20 Then, for any task T, the meta-loss of FSAF is computed as
Li (m r) = D stopgraddop
As will be shown by the experiments in Section 4 performance. The pseudo code of the training procedure of FSAF is provided in Appendix A Remark 6 Based on the convention of few-shot learning for RI,, one shot in few-shot adaptation refers to trajectory task, a shot corresponds to complete trajectory of states and actions collected under a policy. In the context of BO, one-shot adaptation means that each Q-network performs sequential sampling based on its Q-value for generating one trajectory and then uses this trajectory for the gradient updates for fast adaptation. The above notion is consistent with that in Bayesian MAML 20 Appendix C.2].
using small K and S smpirically leads to favorable
rollout generated under the target task. For example, in MuJoCo control
Remark 7
This paper focuses value-based methods for training a few-shot acquisition function. Based on the proposed training framework, it is also possible to extend the idea to design an actor- critic counterpart of our FSAF We believe this is an interesting direction for future work.
4 Experimental Results
We demonstrate the effectiveness of FSAF on wide variety of classes of black-box functions and diseuss how FSAF addresses the critical challenges described in Section T Unless stated otherwise, we report the mean simple regrets over 100 evaluation trials.
Popular benchmark methods. We evaluate FSAF against various popular benchmark methods,
including EI [13,, PI (38) GP-UCB (9) MES 12), TAF [16). Spearmint [.] HEBO [39]. and MetaBO 27 The configuration and hyperparameters of the above methods are as follows. GP-UCB,
EI, and Pl are classic peeneral-uupoos AFs that have been shown to achieve good regret performance for black-box functions drawn from GP. For GP-UCB, we tune its exploration parameter by a grid search between 10-1 and 10-6 Among the family of entropy search methods, MES is strong and computationally efficient benchmark method that achieves superior performance for several global optimization benchmark functions and GP functions 02). For MES, we use Gumbel sampling and take one sample for y,, as suggested by the original paper [2]. TAF uses metadata to construct a transfer acquisition function by forming mixture of experts based
ensemble of GP models. For TAF, we consider a mixture of five experts and use the same meta-data as FSAF. Speartmint is popular and highly optimized BO framework proposed by [.] For Spearmint, we use the source poo
code and the default setting provided by (1. HEBO is one recent general-purpose BO approach that addresses the heteroseedasticity and otn-stationarity of black-box functions, and
the source code") Hand and the default configuration suggested by [39] MetaBO is a neural AF trained via policy- based RL and recently achieves superior performance in BO. For fair and reproducible evaluations, we use the pre-trained model of MetaBO provided by (27.. As the original MetaBO does not address the use of meta-data, for more comprehensive comparison, we further consider a few-shot variant of MetaBO (termed MetaBO-T), which is obtained by performing 100 more training iterations on the pre-trained MetaBO model using the meta-data for few-shot fine-tuning.
Configuration of FSAF, For training, we construct a collection of training tasks, each of which is class of GP functions with either an RBF, Matemn3/2, or spectral mixture kernel with different parameters (e.g., lengthscale and periods). For the reward design of FSAF, we use g(z) = log to encourage high-accuracy solutions. We choose N = 5, K 1, and S = 1 given the limitation of GPU memory. For testing, we use the model with the best average total return during training as our initial model, which is later fine-tuned via few-shot fast adaptation for each task. For a fair comparison, we ensure that FSAF and MetaBO-T use the same amount of meta-data in each experiment. The source code for our experiments has been made publicly available
"The source code is obtained from https: //ttts://githbb com/ SttpperrSnoek/sperrraaae
3 HHEBO the winner of NeurlPS 2020 Black- Box Optimisation Challenge. The source code obtained
from htttps:///gtthiiie cow www.///wwwiionnnnoommcc www.//hhaaiiinna//aE0000
http:://ggttub..cooo... http:///gghhhbbcccc////ggassssss
7

---

Egghelder
Ackley
Styblinkk-TTagg 20
0040
Dicon-Price 7D
Powel
924
MES
TAF-ME
Single Ragget
Simple Regrest
Spearmin HEBO
Regret
Singlo Regret
Simgle Regret
6.2
12
6023
0020
0.2
0033
20
65
30
226
33
60
320.46
5.00
22
63
520 Le
66 Number Sampls
20
25
326.40
0.05
33
60 100360
Number Samples
Number Samples
Number Samples
Number
Samples
(a) Figure Mean simple regrets FSAF and the benchmark methods under optimization test functions.
(h)
(d)
(c)
Does FSAF adapt effectively to a wide variety of black-box functions? To answer this, we first evaluate FSAF and the benchmark methods on several types of standard optimization benchmark functions, including: (i) Ackley and Eggholder, which are functions with large number of local optima but with different symmetry structures; (ii) Dixon-Price a valley-shaped function; (iii) Styblinski- Tang. a smooth function with a couple of local optima; (iv) Powell, a 10-dimensional function (the highest input dimension among the five functions). As the y values of these functions can be one or more orders of magnitude different from each other, for ease of comparison, we scale all the values of the functions to [-2, 2). Such scaling still preserves the salient structure and variations of each test function. To construct the training and testing sets, we apply random translations and re-scalings of up to +/- 10% to z and y values, respectively. In this case, we consider 5-shot adaptation for FSAF and use the same amount of meta-data for MetaBO-T. From Figure 2 we observe that FSAF is constantly the best or among the best of all the methods under all the test functions. We observe that MetaBO performs poorly under functions with many local optima but performs better under smooth functions (e.g., Styblinski-Tang and Powell). This might be due to the fact that MetaBO was trained with smooth GP functions and lacks the ability to adapt to functions with more local variations. We also find that MetaBO-T benefits from meta-data and improves upon MetaBO in some functions. While this manifests the potential benefits of using meta-data for MetaBO, this also suggests that frute-force fine-tuning is not effective and a more careful design like FSAF is needed. Spearmint performs well under the dives-dime functions (e.g.. Eggholder and Ackley) but has high regrets under the higher-dimensional Powell function. This appears consistent with the results in [4)] HEBO is strong under Powell and Ackley but suffers under Eggholder and Styblinski-Tane Tang. Moreover, among the benchmark methods, the best-performing AF indeed varies under different types of functions. This corroborates the commonly-seen phenomenon and our motivation.
Grid
HPOBench
Stability
PM2.5 14D
4D
Asteroid 120
TSAF
GPAUCS
0.00
MesuBC-- Mocubic
Simple Regget
0L153
0L175
TAR ve
Speammen
Simple Regret
2.4
Simple Regret
Simple Regret
0:05
F1
II:80
0.3
0.09
0.00
0075
0053
0.025
35
1203c0
1201.0
600
25
Mumber of Samples
Number of Samples
Number of Samples
Namber of Samples
Number of Samples
(a)
(b)
(c)
(d)
(c)
Figure 3: Mean simple regrets of FSAF and other benchmark methods under real-world test functions.
We proceed to evaluate FSAF on test functions obtained from five open-source real-world tasks in different application domains. Based on the smoothness characteristicç5] the datasets can be categorized as: (i) Smooth in all dimensions: Asteroid size prediction (d = 12); (ii) Smooth in all but one dimension: hyperparameter optimization for XGBoost (d 6) and air quality prediction in PM 2.5 (d- 14); (iii) Smooth in about half of the dimensions: maximization of electric grid stability
'To better understand the characteristics of each real-world dataset, we extract the smoothness information through marginal likelihood maximization surrogate GP model with an RBF kernel.
8

---

(d = 12); (iv) Non-smooth in all dimensions: location selection for oil wells (d = 4). The detailed description of the datasets is in Appendix [ In this setting, we consider 1-shot adaptation for FSAF, rather sample-efficient scenario of few-shot learning. From Figure we observe that FSAF remains the best or among the best for all the five real-world test functions, despite the salient structural differences of the datasets. Based on the above discussion, we confirm that FSAF indeed achieves favorable adaptability. More experimental details are provided in the supplementary material.
Training Curve
Ackley 2D
Dixon-Price 7D
Powell 10D
12
Random
1.2
1 particle
Average of Negatuv LinSSimple Regres
web
Begret
5 particles
Simple Rogret
0.25
0.5
0.5
0.20
2.4
0.5
Simple
0.35
0.4
0.30
0.2
0.05
0.0
100
200
300
400
500
60
60
20
20
40
60 80
iteration
Number Samples
Number of Samples
Number of Samples
(a)
(b)
(c)
(d)
Figure 4: Ablation study for FSAR: (a) Training curves of FSAF and its ablations (EI and Random as baselines); (b)-(d) Testing performance of FSAF with 1 and 5 particles of Q-networks.
Does FSAF mitigate the overfitting issue? To answer this, show the training curves of FSAF and its ablations, including: (i) FSAF without using the demo replay buffer (termed "w/o demo"";; ;; (ii) FSAF with the chaser meta-loss replaced by TD loss (termed "w/o chaser"); (iii) FSAF with only 1 particle (termed "1 particle"), which is equivalent to MMN+ with an additional demo replay buffer. The training metric plotted the negative logarithm of simple regret episodes in each iteration), which is consistent with the rewards of FSAF and can better demonstrate the differences in training. The results of El and random sampling are given as references.
30 (averaged over
The Bayesian variant of DQN does mitigate overfitting. This is confirmed by the fact that the
training curves of FSAF with and 5 particles are quite close, while in Figures 4(b) 4(d) the testing
performance of FSAF with particles appears much better than that of particle.
The demo-based prior helps improve the training stability. This is verified by that without the demo replay buffer, the training progress becomes apparently slower initially and is subject to more variations throughout the training.
The chaser meta-loss appears effective in FSAF. Interestingly, we find that chaser meta-loss results in stable and effective training, while the TD meta-loss can barely make any progress.
Does FSAF benefit from the few-shot gradient updates? To better understand the effect of few- shot gradient updates, Figure 5 shows the simple regrets of FSAF under different number of gradient updates (i.e., K defined in Section (.3) for the optimization benchmark functions. We find that the adaptation effect does increase with K for small K's for most of the cases. This appears consistent with the general observations of MAML-like algorithms (8].
Eggholder 2D
0.49
Addey 2D
0.0305
5.0305
STYBLINSKI TANG 2D
0,0200
Dixon-Price 7D
0:30
POWELL 10D
FReE win upeate
Simple Regret
0.5
all updutc
Simple Regret
0.30
0.55
0.053
0.0275
6.0100
Simple Regret
6.00
025
16022
0.015
0.06
0.
023
0.0000
0.00
0.30
0.0155
0.0050
0.3
30 60
120160
2.05
30
60
30
120100
5.0100
30
1201
0.0015
30
60
42
0.00
30
60
120140
Number of Samples
Number of Samples
Number of Samples
Number of Samples
Number of Samples
(b)
(d)
Figure 5: Simple regrets of FSAF number of gradient steps for optimization benchmark functions.
Remark (Application scopes of the benchmark methods) Recall that FSAF is positioned as a meta-learning BO algorithm that can leverage small amount of metadata for few-shot fast adaptation. In contrast, MetaBO is not designed for few-shot fast adaptation but for transfer learning in BO. Despite the difference, we choose MetaBO and its fine-tuned version MetaBO-T as important

---

benchmark methods since there are not many meta-learning BO algorithms and MetaBO appears to be a strong benchmark method in the class of meta-learning for BO 277. On the other hand, while the two important benchmark methods Spearmint and HEBO do not address fast adaptation with metadata, they share the application scope of optimizing general black-box functions with FSAF.
[1] proposes to leverage meta-data to fine-tune the initialization of the GP kernel parameters for the off-the-shelf AFs (called FSBO in [3]). By contrast, FSAF uses meta-data for fast adaptation of an AF, which is a direction orthogonal to FSBO. Moreover, it is possible for our FSAF and FSBO to complement each other. We provide experimental results to verify this argument in Appendix D
Remark 9 (FSAF and [5]) As mentioned in Section [
Related Work
Few-shot learning. Few-shot learning has recently attracted much attention since the data scarcity has become a bottleneck to many real-world machine learning tasks [41]. Using prior knowledge, few- shot learning can rapidly generalize a pre-trained model to new tasks given only few examples. One of the most representative frameworks for few-shot learning is MAML [8]. where an initialization of parameters is pre- pre-trained to be close to the tasks drawn from the task distribution, and thus few gradient steps are sufficient to adapt it to a specific task. There are several follow-up studies for MAML. For instance, in (42, vanilla MAML is shown to be sensitive to the network hyperparameters. often leading to training instability. To overcome that, learning the majority of hyperparameters end to end is proposed. In 20]_ a Bayesian version of MAML is proposed to learn complex uncertainty structure beyond a simple Gaussian approximation. In 43 MAML is revisited under multimodel task distributions. To better adapt to different modes, a modulation network is proposed identify the mode of the task distribution and then customize the meta-learned prior for the identified mode.
Bayesian optimization via non-myopic methods. To go beyond myopic AFs, recently there has been some interest in designing non-myopic strategies for BO from the perspective of dynamic characterized the optimal non-myopic strategies for BO. To
programming (DP). For example, [44 45
47 proposed to approximately solve the DP problem for BO by different simulation techniques. 48 focused on the two-step lookahead AFs for BO and proposed an efficient Monte-Cario method to find such AFs. Instead of fixing the horizon in advance, [49] proposed principled way to select the rolling horizon for approximate dynamic programming in BO. 50 proposed to achieve non-myopia by maximizing a lower bound on the multi-step expected utility, [5 proposed tree-based AF to enable approximate multi-step lookahead in a one-shot manner by leveraging the reparameterization trick. The proposed FSAF is also non non-myopic by nature as it implicitly takes multi-step effect into account by using Q-networks as AFs. Different from the above multi-step solutions, FSAF is meant to serve as an AF that can adapt to a wide variety of black-box functions given only a small amount of meta-data.
tackle the intractable DP problem, 46
Meta Bayesian optimization. Meta-learning for BO has been discussed in many prior studies. Some of them (32)53) use a single GP model as the prior of all the tasks. However, real-world applications, different tasks may have different task-specific features that fail to be captured by a single model. To mitigate the issue, several studies propose to learn an ensemble of GPs as the prior. For instance, Wistuba et al. [66] proposed to use a transferable AF that uses several GPs to evaluate the target dataset then weights expected improvement by the result of the evaluation for fitting in the current task. Feurer et al. [17 proposed to use an ensemble of GP models obtained from each subset of meta-data and use all of them for inference in the new task. But these works still need a sufficiently large amount of data for knowledge transfer to new tasks. [40] proposed to train a general-purposs black-box function optimizer by using recurrent neural networks. MetaBO (2 used policy- RL to train a neural AF that learns structural properties of a set of source tasks to enable knowledge transfer to related new tasks. FSBO [15] provides a sew-shot deep kernel network for a GP surrogate that can quickly adapt its kernel parameters to unseen tasks. However, the benefits of using meta-data for more efficient exploration via few-shot fast adaptation of AFs were not explored by 15 27
Concluding Remarks
This paper tackles the critical challenge of how to effectively adapt an AF for BO to a wide variety of black-box functions through the lens of few-shot acquisition function (FSAF) learning. One potential limitation of FSAF is the need of a small amount of meta-data. Without any few-shot adaptation, the performance may not always be satisfactory, as shown in Figure Despite this, through extensive experiments, we show that FSAF is indeed a promising reneral-purposss approach for BO.
10

---

Acknowledgment
The work Bing-Jing Hsich and Ping-Chun Hsich is supported in part by the Ministry of Science and Technology (MOST) of Taiwan under Contract Numbers MOST (09-2636-BB009-011 and MOST 110-2628-E-449-01 E-A49-014.
Funding Transparency Statement
Funding in direct support of this work: MOST grants under Contract Numbers MOST 109-2636- E-009-012 and MOST 110-2628-E-A49-044, awarded to Ping-Chun Hsieh in National Yang Ming Chiao Tung University. There is no additional revenue related to this work.
References
[1] Jasper Snoek, Hugo Larochelle, and Ryan P Adams. Practical Bayesian Optimization of Machine Learning Algorithms. In Advances in Neural Information Processing Systems, 2012.
[2] Jia Wu, Xiu- Yun Chen, Hao Zhang. Li-Dong Xiong. Hang Lei, and Si-Hao Deng. Hyperparameter optimization for machine learning models based on Bayesian optimization. Journal of Electronic Science and Technology, 17(1):26-40, 2019.
Chris Thornton, Frank Hutter, Holger Hoos, and Kevin Lcyton-Browe Auto-weka: Combined selection and hyperparameter optimization classification algorithms. In Proceedings of the 19th ACM SIGKDD international conference on Knowledge discovery and data mining, pages 847-855, 2013.
[4] Yuting Zhang, Kihyuk Sohn, Ruben Villegas, Gang Pan, and Honglak Lee. Improving object detection with deep convolutional networks via bayesian optimization and structured prediction. In Proceedings of the IEEE Conference Computer Vision and Puttern Recognition, pages 249-258, 2015.
[5] Roberto Calandra, André Seyfarth, Jan Peters, and Marc Peter Deisenroth. Bayesian optimization for learning gaits under uncertainty. Annals of Mathematics and Artificial Intelligence, 76(1):5 23, 2016.
[6] Peter I. Frazier and Jialei Wang. Bayesian optimization for materials design. Springer Series in Materials Science, page 15,75, Dee 2015. ISSN 2812-
[7] Ryan-Rhys Griffiths and José Miguel Hernández Hernández-Lobato. Lobato. Constrained Bayesian optimization for automatic chemical design using variational autoencoders. Chemical science, 11(2):777555 2020.
[8] Bobak Shahriari, Kevin Swersky, Ziyu Wang. Ryan P Adams, and Nando De Freitas. Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE, 104(1):148 -17-1155-,,,,,- 2015.
19] Niranjan Srinivas, Andreas Krause, Sham Kakade, and Matthias Seeger. Gaussian process optimization in the bandit setting: no regret and experimental design. In Proceedings of International Conference on Machine Learning, pages 1015-1022, 2010.
[10] Philipp Hennig and Christian Schuler. Entropy search for information-effciiet global optimization. Journal of Machine Learning Research, 13(6), 2012.
[11] José Miguel Hernánlez-Lobatoo Matthew W Hoffman, and Zoubin Ghahramani. Predictive entropy search for efficient global optimization of black-box functions. Advances in Neural Information Processing Systems, 27:918-922 2014.
[12] Zi Wang and Stefanie Jegelka. Max-value entropy search for efficient bayesian optimization. In Interna- tional Conference Machine Learning, pages 3627-3635, 2017.
[13] Jonas Močkus. On Bayesian methods for seeking the extremum. In Optimization Techniques IFIP Technical Conference, pages 400-404. Springer. 1975.
[14] Donald R Jones, Matthias Schonlau, and William J Welch, Efficient global optimization of expensive black-box functions. Journal of Global optimization, 13(4):455 492, 1998.
[15] Martin Wistuba and Josif Grabocka. Few-shot bayesian optimization with deep kernel surrogates. In International Conference on Learning Representation, 2021.
[16] Martin Wistuba, Nicolas Schilling, and Lars Schmidt- Thieme. Scalable gaussian process-based transfer surrogates for hyperparameter optimization. Machine Learning, 107(1):43-77) 2018.

---

[17] Matthias Feurer. Benjamin Letham, and Eytan Bakshy. Scalable meta-learning for Bayesian optimization. 2018.
[18] Chelsea Finn, Pieter Abbeel, and Sergey Levine. Model-agnostic meta-learning for fast adaptation of deep networks. In International Conference on Machine Learning, pages 1 1126-1135, 2017.
[19] Nikhil Mishra, Mostafa Rohaninejad, Xi Chen, and Pieter Abbecl. A simple neural attentive meta-learner. In International Conference on Learning Representations, 2018.
[20] Jaesik Yoon, Taesup Kim, Ousmane Dia, Sungwoong Kim, Yoshua Bengio, and Sungjin Ahn. Bayesian -adel-agnostic meta-lcaming. In Advances in Neural Information Processing Systems, pages 7332-7342, 2018.
[21] Justin Fu, Aviral Kumar, Matthew Soh, and Sergey Levine. Diagnosing bottlenecks deep Q-learning algorithms. In International Conference on Machine Learning, pages 20300, 2019.
[22] Luisa Zintgraf, Kyriacos Shiarli, Vitaly Kurin, Kalja Hofmann, and Shimon Whiteson. Fast context adaptation via meta-leariing, In International Conference on Machine Learning, pages 7693-7702, 2019.
[23] Yang Liu, Prajit Ramachandran, Qiang Liu, and Jian Peng. Stein variational policy gradient. Conference on Uncertainty in Artificial Intelligence, 2017.
[24] Peter I Frazier. A tutorial on Bayesian optimization. arXiv: 2018.
[25] Eric Brochu, Vlad M Cora. and Nando De Freitas. tutorial on Bayesian optimization expen- sive cost functions, with application to active user modeling and hierarchical reinforcement learning. 2010.
=a1Xiv:1012.25900
[26] Tom Van de Wiele, David Warde-Farley, Andriy Mnih, and Volodymyr Mnih. Q-learning in enormous action spaces via amortized approximate maximization. arXiv:2001. 08,16, 2020.
[27] Michael Volpp. Lukas P. Fröhlich, Kirsten Fischer, Andreas Doerr, Stefan Falkner, Frank Hutter, and Christian Daniel. Meta-learning acquisition functions for transfer learning in bayesian optimization. In International Conference on Learning Representation, 2020.
[28] Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Andrei A Rusu, Joel Veness, Mare G Bellemare, Alex Graves, Martin Riedmiller, Andreas K Fidjeland, Georg Ostrovski, et al. Human-level control through deep reinforcement learning. Nature, 518(7540):529-33 -33,,99-332333 2015.
[29] Qiang Liu and Dilin Wang. Stein variational gradient descent: general purpose Bayesian inference algorithm. In Advances in Neural Information Processing Systems, pages 2378-2386, 2016.
[30] Ian Osband, Benjamin Van Roy, and Zheng Wen. Generalization and exploration via randomized value functions. In International Conference on Muchine Learning, pages 2377-2386, 2016.
[31] Kamyar Azizzadenesheli, Emma Brunskill, and Animashree Anandkumar. Efficient exploration through Bayesian deep Q-networks. In Information Theory and Applications Workshop (ITA), pages 1-9, 2018.
[32] Ian Osband, Charles Blundell, Alexander Pritzel, and Benjamin Van Roy. Deep exploration via bootstrapped DQN. In Advances in Neural Information Processing Systems, pages 4026 4034, 2016.
[33] Yunhao Tang and Alp Kucukelbir. Variational deep Q network. arXiv:1711.11225 2017.
[34] Brian D Ziebart. Modeling purposeful adaptive behavior with the principle of maximum causal entropy. 2010.
[35] Tuomas Haarnoja, Haoran Tang, Pieter Abheel, and Sergey Levine. Reinforcement learning with deep energy-based policies. In International Conference on Machine Learning, pages 1352-1361, 2017.
[36] Tuomas Haarnoja, Aurick Zhou, Pieter Abbeel, and Sergey Levine. Soft actor-critic: Off policy maximum entropy deep reinforcement learning with a stochastic actor. In International Conference on Machine Learning, pages 1861-1870, 2018.
[37] Stéphane Ross, Geoffrey Gordon, and Drew Bagnell. A reduction imitation learning and structured prediction to no-regret online learning. In Proceedings of International Conference on Artificial Intelligence and Statistics (AISTATS), pages 627-635, 2011.
[38] H. J. Kushner. A new method of locating the maximum point of an arbitrary multipeak curve in the presence of noise. Journal of Basic Engineering, -6:97-106, 1964.
12

---

[39] Alexander I Cowen-Rivers, Wenlong Lyu. Rasul Tutunov, Zhi Wang, Antoine Grosnit, Ryan Rhys Griffiths. Alexandre Max Maraval, Hao Jianye, Jun Wang, Jan Peters, et al. An empirical study of assumptions in Bayesian optimisation. arXiv preprint arXiv:2012.0333 2020.
[40] Yutian Chen, Matthew W Hoffman, Sergio Gómez Colmenarejo, Misha Denil, Timothy P Lillicrap. Matt Botvinick, and Nando Freitas. Learning learn without gradient descent by gradient descent. In International Conference on Machine Learning, pages 748-756, 2017.
[41] S. Thrun and L. Pratt. Learning Learn. Springer US, 2012. ISBN 9781 171155555. URL https. //books. . google. . com. wtt.coom...///eeoo twwbbtkwwb//ttoooo A_trpidddx______...
[42] Antreas Antoniou, Harrison Edwards, and Amos Storkey, How train your maml. In International Conference on Learning Representations, 2018.
[43] Risto Vuorio, Shao-Hua Sun, Hexiang Hu, and Joseph J. Lim. Multimodal model-agnostic meta-learning via task-aware modulation, 2019.
[44] Michael A Osborne, Roman Garnett, and Stephen J Roberts. Gaussian processes for global optimization.
In International Conference Learning and Intelligent Optimization (LION3), pages 1-15, 2009.
[45] David Ginsbourger and Rodolphe Le Riche. Towards Gaussian process- process-based optimization with finite time horizon. In Advances in Model- Oriented Design and Analysis, pages 89 96. 2010.
[46] Javier González, Michael Osbome, and Neil I lawrence. GLASSES: Relieving the myopia of Bayesian optimisation. In Proceedings of International Conference on Artificial Intelligence and Statistics (AISTATS). pages 790-799, 2016.
[47] Remi R Lam, Karen E Willcox, and David H Wolpert. Bayesian optimization with finite budget: An approximate dynamic programming approach. In Advances in Neural Information Processing Systems, pages 883-891, 2016.
[48] Jian Wu and Peter Frazier. Practical two-step lookahead Bayesian optimization. Advances in Neural Information Processing Systems, 32:9813-9822 2019.
[49] Xubo Yue and Raed AL Kontar. Why non-myopic Bayesian optimization is promising and how far should look-ahead? A study via rollout. In International Conference on Artificial Intelligence and Statistics (AISTATS), pages 2808-2818, 2020.
[50] Shali Jiang, Henry Chai, Javier Gonzalez, and Roman Garnett. BINOCULARS for efficient, nonmyopic sequential experimental design. In International Conference on Machine Learning, pages 4794-4803, 2020.
[51] Shali Jiang, Daniel Jiang. Maximilian Balandat, Brian Karrer, Jacob Gardner, and Roman Garnett. Effi- cient nonmyopic Bayesian optimization via one-shot multi-step trees. Advances Neural Information Processing Systems, 33, 2020.
[52] Kevin Swersky, Jasper Snock, and Ryan Adams. Multi-task Bayesian optimization. Advances in Neural Information Processing Systems, 26:2004-2012, 2013.
[53] Dani Yogatama and Gideon Mann. Efficient transfer learning method for automatic hyperparameter tuning. In Proceedings of International Conference on Artificial Intelligence and Statistics (AISTATS), pages 1077-1085, 2014.
[54] Ziyu Wang, Tom Schaul, Matteo Hessel, Hado Hasselt, Mare Lanctot, and Nando Freitas. Dueling network architectures for deep reinforcement. learning. International Conference on Machine Learning, pages 1995-2003, 2016.
[55] Vadim Arzamasov, Klemens Böhm, and Patrick Jochem. Towards concise models grid stability. IEEE International Conference on Communications, Control, and Computing Technologies for Smart Grids (SmariGridComm) pages 1 6, 2018
[56] Victor Basu. Prediction of asteroid diameter with the help of multi-layer perceptron regressor. 2019.
[57] Alex Nichol, Joshua Achiam, and John Schulman. On first-order meta-learning algorithms. arXiv: 10Xiv:880300299) 2018
[58] Massimiliano Patacchiola, Jack Turner, Elliot J Crowley, Michael O' Boyle, and Amos J Storkey. Bayesian meta-learning for the sew-shot setting via deep kernels. Advances in Neural Information Processing Systems, 33, 2020.
13

---

Checklist
1. For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope? [Yes] The abstract states that the main contribution is the design of
few-shot acquisition function. Please see Section 3 for the design and Section 4 for the
experimental results.
(b) Did you describe the limitations of your work? [Yes] Please see Section 6
(c) Did you discuss any potential negative societal impacts of your work? [N/A]
(d) Have you read the ethics review guidelines and ensured that your paper conforms them?
[Yes]
you are including theoretical results...
Did you state the full set of assumptions of all theoretical results? [N/A] Did you include complete proofs of all theoretical results? [N/A]
If you ran experiments...
(a) Did you include the code, data, and instructions needed to reproduce the main experimental results (either in the supplemental material or as a URL)? [Yes] Please see the supplementary material.
(b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)? [Yes] Please see Section 4 and Appendices B C for details.
(c) Did you report error bars (e.g., with respect to the random seed after running experiments multiple times)? [Yes]
(d) Did you include the total amount of compute and the type of resources used (e.g.. type of GPUs, internal cluster, cloud provider)? Please Appendix B for the description of computing resources. [Yes]
you are using existing assets (e.g., code, data, models) or curating/releasing new assets...
If your work uses existing assets, did you cite the creators? [Yes]
(b) Did you mention the license of the assets? [Yes] Please see the code the supplementary material.
(c) Did you include any assets either in the supplemental material URL? [Yes] Please see the supplementary material.
(d) Did you discuss whether and how consent was obtained from people whose data you'r 're using/curating? [Yes] Please see the supplementary material.
(c) Did you discuss whether the data you are using/curating contains personally identifiable information offensive content? [N/A]
5. If you used crowdsourcing or conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, if applica- ble? [N/A]
(b) Did you describe any potential participant risks, with links Institutional Review Board
(IRB) approvals, if applicable? [N/A]
(c) Did you include the estimated hourly wage paid participants and the total amount spent participant compensation? [N/A]
14


Reviews:
[{"review_id": "qJCvXUiU23P", "reviewer": "Reviewer_dJ7J", "summary": "This paper proposed an algorithm for few-shot Bayesian optimization. It conducted experiments to show that their algorithm could adapt to a variety of black-box function and prevent overfitting.", "questions": "", "limitations": "", "rating": 7, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Originality. Although the algorithm in this paper might be similar with some former work, it applied some technique to improved both its validity and novelty. It also cited related work clearly.\n\nQuality. This is a complete work. Their experiment result showed that the algorithm they design is valid. However, it used median of regret instead of mean, which might require further explanation.\n\nClarity. This paper is well-written. However, some details might need further explanation. For example, in line 136, it said \"Based on the conceptual similarity between acquisition functions and Q-functions,.....\". However, the similarity is not that clear, since it seems that there are not any concepts in Bayesian optimization that are similar with transition in RL. Maybe it is more similar with the reward in bandits. Besides, 'Section 3.2' in line 159 might be 'Section 3.1'.\n\nSignificance. The algorithm in this paper can be applied to general black-box function, which might be useful in application and allow people to build better algorithm in Bayesian optimization based on their algorithm.", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "This author adequately addressed the limitations in this paper. It is up front about the similarity between their algorithm and former work in remark and  related work. Their work does not have potential negative societal impact. It also addressed its need for meta-data in conclusion.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "8"}, "extra_text": {}}, {"review_id": "TeOAnhAJpvE", "reviewer": "Reviewer_wSpJ", "summary": "The authors propose to combine meta-learning, reinforcement learning and few-shot learning to learn an acquisition function for Bayesian optimization which is outperforming the hand-crafted alternatives. The proposed method is evaluated on a variety of problems against the most common acquisition functions.", "questions": "", "limitations": "", "rating": 7, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The authors nicely motivate and explain their method. The proposed method is in most places well-explained. There are few cases where the additionally information is hidden in the appendix. The components of the method and the decisions made by the authors are well explained and supported by ablation studies whenever needed. The experimental section provides strong evidence that the proposed method is an improvement over the state-of-the-art.\n\nI think the authors could highlight the fact that FSAF is trained on synthetic data (which we get for free) and requires only metadata for very few tasks (which might be costly to generate). I point that out because typically meta-learning methods use many tasks and infer from them to a new task so the reader might just expect the very same setup. I believe this is a strong point of the method and I only realized this after actively looking for which metadata is used in the appendix.\n\nThe authors claim that the proposed method requires only few tasks and therefore requires less metadata than related approaches such as [16]. One question unanswered is how many observations per task are required. In practice, it is probably more common to have metadata for many tasks but for each only few observations. I hope the authors can be a bit more specific under which circumstances their method works and under which it doesn’t.\n\nI am curious how this method compares to [16] and the acquisition function used by the winners of the NeurIPS BBO challenge: https://arxiv.org/abs/2012.03826\n\nWhy was the median instead of the mean reported? This is rather uncommon and might indicate that the method has some problems learning good solutions every time.\n\nLine 289: N, K and S are not used in the main paper. A short remark pointing to the appendix would be helpful here.", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "-", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "6"}, "extra_text": {}}, {"review_id": "NHN1599KmoE", "reviewer": "Reviewer_doq2", "summary": "The paper proposes \"Reinforced Few-Shot Acquisition Function Learning\" (FSAF), a meta-learning approach for designing few-shot acquisition functions (AFs) for Bayesian Optimization (BO). \n\nThe authors map BO onto an reinforcement learning (RL) problem, i.e., identify the AF with an RL policy which proposes evaluation points, given the current state of the optimization in form of the Gaussian Process (GP) posterior (and further features). The authors develop a Bayesian version of Deep Q-learning (DQN), leveraging Stein Variational Gradient Descent (SVGD), and combine it with ideas from Bayesian MAML (BMAML) to meta-train an initial AF on function samples from GP priors. This initial AF can be adapted to a family of target optimization problems using a few (~1-5) context optimization problems from this family for fast (few-shot) adaptation. \n\nThe method is evaluated on various sets of optimization problems, showing favorable performance compared to existing hand-designed AFs (EI, PI, GP-UCB, MES) as well as the transfer-learning approach MetaBO [1].", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Originality: FSAF is a novel meta-learning approach for AF-design. It stands out from existing work in that it focuses on few-shot adaptation, i.e., the architecture is trained to use a small context set of unseen optimization problems to quickly adapt to a given class of optimization problems, which is a novel approach to meta-learning for BO.\n\nWhile the authors adequately describe that their proposed algorithm is a novel combination of existing work (DQN, SVGD, BMAML), I have several concerns about the presentation of the general setup and the application scopes of the compared methods:\n\n- The authors do not state clearly that the general idea for mapping BO onto an RL problem (Sec. 3.1) is not new: [1] propose to use exactly the same general RL setting with the same state-action representation and reward function (indeed, the frameworks are similar enough that the authors can just extend the software framework from [1] by their novel training algorithm). This is not adequately cited.\n- Similarly, I feel that the presented experimental comparison to [1] can be misleading. [1] is a its core transfer-learning algorithm, i.e., belongs to the class of algorithms discussed in ll. 39-51 (where [1] is not discussed). Indeed, [1] devise an algorithm which is supposed to be trained on optimization problems which come from the same distribution as the target problems and do not consider a fast adaptation step. In contrast, the authors explicitly train their AF for fast adaptation. I appreciate that the authors consider the dimensionality-agnostic MetaBO [1] version trained on GP samples and also test a naively fine-tuned version (MetaBO-T) and state that [1] does not consider a fast adaptation step, but still I feel that the comparison to [1] is not adequate, as [1] was conceived with a different application scope in mind (transfer learning vs. few-shot adaptation). If the incentive of the authors is to compare to transfer-learning methods, they should also include comparisons to, e.g., ensemble methods for transfer learning as discussed in ll. 39-51 (e.g., the TAF framework proposed in [2]) and use the context set of $K$ optimization problems as the source tasks.\n- The authors do not cite [3], which propose learning global optimizers in a very similar setting as FSAF. Indeed, in contrast to the aforementioned class of transfer-learning methods [1,2], [3] train a general-purpose optimizer (much like FSAF) on GP samples and evaluate the resulting optimizer on a range of different target distributions. While [3] also do not consider a fast adaptation step, this work fits the setting of FSAF better than [1,2]. It is known that the results of [3] are hard to reproduce, so I do not expect the authors to provide experimental comparisons, but I feel that the experimental evaluation lacks a clear discussion of the application scopes of the compared methods.\n- As noted in the previous paragraph, I feel that natural competitors for FSAF are general-purpose global optimizers (instead of approaches like [1,2] which are designed specifically for the transfer-learning setup). I appreciate the comparisons to general-purpose AFs (EI, PI, MES) but I would also like to see comparisons as presented in [3], e.g., to a highly optimized BO framework like Spearmint [4].\n\nQuality: The author's approach to design a few-shot AF for BO is interesting and the proposed combination of DQN, SVGD, and BMAML is clever and appropriate.\n\nClarity: The overall writing and organization of the paper is good. Please refer to my notes under \"Originality\" regarding my concerns about the clarity of the experimental evaluation.\n\nSignificance: The proposed method is an interesting and important contribution, as — to the best of my knowledge — it is the first approach for few-shot AF learning in the context of BO. Nevertheless, I feel the paper is not ready for publication yet due to my concerns regarding the experimental evaluation and the lack of discussion regarding the application scopes of the compared methods. I therefore lean to rejection but am I willing to increase my score if the authors address my concerns.\n\nReferences:\n\n[1] Volpp et al., \"Meta-Learning Acquisition Functions for Transfer Learning in Bayesian Optimization\", ICLR 2020\n\n[2] Wistuba et al., \"Scalable Gaussian-process Transfer Surrogates for Hyperparameter Optimization\", JMLR 2018\n\n[3] Chen et al., \"Learning to Learn without Gradient Descent by Gradient Descent\", ICML 2017\n\n[4] Snoek et al., \"Practical Bayesian Optimization of Machine Learning Algorithms\", NeurIPS 2012\n\n\n-------------------------------------------------------------------------\nUpdate after reading the author's response:\n\nI thank the authors for their response. I raise my score because the authors\n\n- clarify the application scope of FSAF and the baseline methods (\"general-purpose optimizer\" vs. \"transfer-learned optimizer\" vs. \"meta-learned/fast adaptation optimizer\")\n- include a further transfer-learning approach (TAF) in their comparison\n- include general-purpose optimizers (Spearmint, HEBO) in their comparison\n- improve the citation of related work (Chen et al., Volpp et al.)", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "No potential negative societal impact.", "review_text": "", "extra_scores": {"ethical_concerns": "No ethical concerns.", "needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "6"}, "extra_text": {}}, {"review_id": "4S-zDJm-ag0", "reviewer": "Reviewer_9aK8", "summary": "This paper proposes to learn a general quickly-adaptable acquisition function (AF) using reinforcement learning. Particularly, it first connects AFs in BO with Q functions in RL, and then learns an adaptable AF with DQN + MAML. In order to mitigate the overfitting problem, it considers a Bayesian approach and applies Bayesian MAML. The empirical performances on a variety of black-box functions show very promising performance compared to many baseline BO algorithms.\n\nExploring the connection between AFs in BO and Q functions in RL is very interesting and novel AFAIK. Presenting BO as a RL problem allows one to learn a non-myopic AF which is also a good property compared to standard myopic AFs. Besides, training a fast adaptable AF at the presence of meta-data (tuning data of similar functions) is a nice contribution although there has been some prior work on learning an adaptable GP model.\n\nThe extensive experiments and ablation are useful and convincing.\n\nMy main concern of the paper that prevents me from giving a higher rating is on the use and the required amount of meta-data. Please see my question in below.", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "This paper makes a nice observation that AFs can be viewed as the Q function in RL. Then learning a non-myopic AF can be done with DQN. It would be nice to train a general AF that compares comparably or even better than standard BO methods. Unfortunately, this paper shows it is not the case at the end of experiments. (It's actually not clear to me how FSAF w/o update compares to other baselines, and I think this will be very useful to know) Therefore, the authors consider a few-shot setting, where the learned AF can be quickly adapted to the function of interest if given some meta data. They propose to meta-learn the base AF with MAML, but it overfits to the training data. So Bayesian MAML is further employed to mitigate the overfitting and stabilising the training.\n\nThe proposed model and the use of training algorithm method are technically\tsound.\n\nThe entire model and training algorithm is complicated with multiple components, including GP, prior policy, Bayesian MAML, meta data, two replay buffers, etc. This paper makes a fairly good effort to explain the key components with the limited space. Nonetheless, it is not very straightforward to understand the entire algorithm. I would suggest moving Figure 6 to the main text for better clarity. A few key information is still missing from the main text, listed as follows:\n- How do you define a stochastic policy given the Q function, and does this method work in a continuous state space?\n- How do you generate the transitions in the reply buffer during meta-training?\n- Where does the meta data come from and how do you construct the replay buffer of the meta data during both meta-training and evaluation?\n\nIt would be nice explain them in the revision.\n\nMy main concerns/questions about the paper are as follows,\n\n- What is the meta-data used in adaptation. What does 1 shot mean? During meta-training, for every pair of support function(s) and target function, are they the same function? E.g., in the Electrical Grid Stability Dataset, the 60000 parameter configurations are divided into 40 sets from which 1 validation set is used for adaptation. Does it mean all the configurations are the input points of a shared black-box function and we have to evaluate all the configurations in the validation set to provide the meta data (totally 1500 configurations)? If so, it becomes meaningless to look at the optimisation performance with additional dozens of samples.\n\n- In evaluation, is meta-data used by other BO methods e.g. EI, PI, GP-UCB? I guess it's straight ward to fit the GP hyper-parameters with the meta data, and if the meta data comes from the same function to optimise, we can even get the posterior to start sampling.\n\n- Related to the question about the definition of policy defined from Q. If \\pi_\\theta is defined as softmax over the Q output logits, there is a normalisation term implicitly in Eq. 5. That term will become a problem in evaluating the prior in Eq 6.\n\n- The prior distribution in Eq 6 could be un-normalizable if \\delta doesn't decay fast enough as \\theta moves to infinity.\n\n- How does FSAF perform in the region with more samples than 80? A typical BO algorithm usually run from 10s to 100s. Does the discounting factor impact how long FSAF can generalise for? Can you please have some comments on the choice of reward function g(.)?\n\n- How does FSAF scale to higher dimensional input space?\n\n\nSome minor comments:\n\n- Related work on meta BO: Chen et al, (2017) applies meta-learning to train a neural black-box optimization algorithm directly, instead of learning the AF, using GP samples as training functions.\n\nLine 322 in p8: Figure 4 --> Figure 3. Figure 4 should be referred to from the paragraph of line 328.\n\n\nRefs:\n\nChen, Y., Hoffman, M.W., Colmenarejo, S.G., Denil, M., Lillicrap, T.P., Botvinick, M. and Freitas, N., 2017, July. Learning to learn without gradient descent by gradient descent. In International Conference on Machine Learning (pp. 748-756). PMLR.\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "The dependence on the amount of meta-data could be future discussed and ablated.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "3"}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper proposed a novel approach that combines meta learning and reinforcement learning for designing few-shot acquisition functions for Bayesian Optimization. All reviewers find the problem setup interesting and appreciate the novelty and applicability of the proposed algorithm. After a few rounds of interaction during the discussion phase, the reviewers are convinced about the empirical significance of the proposed work.

When preparing a revision, the authors are strongly encouraged to take into account the reviews and accommodate the changes reflected in the author discussions---in particular, to further strengthen the empirical analysis, incorporate new references, clarification of the technical challenge, as well as elaborate on details of the FSAF algorithm and its application scope.

Decision: Accept (Poster)