End-to-End Meta-Bayesian Optimisation with Transformer Neural Processes
Authors: Alexandre Max Maraval
Matthieu Zimmer
Antoine Grosnit
Haitham Bou Ammar
Venue: NeurIPS 2023 poster (2023)
Keywords: meta-learning
bayesian optimisation
neural process
transformer
end-to-end
reinforcement learning

Abstract: Meta-Bayesian optimisation (meta-BO) aims to improve the sample efficiency of Bayesian optimisation by leveraging data from related tasks. While previous methods successfully meta-learn either a surrogate model or an acquisition function independently, joint training of both components remains an open challenge. This paper proposes the first end-to-end differentiable meta-BO framework that generalises neural processes to learn acquisition functions via transformer architectures. We enable this end-to-end framework with reinforcement learning (RL) to tackle the lack of labelled acquisition data. Early on, we notice that training transformer-based neural processes from scratch with RL is challenging due to insufficient supervision, especially when rewards are sparse. We formalise this claim with a combinatorial analysis showing that the widely used notion of regret as a reward signal exhibits a logarithmic sparsity pattern in trajectory lengths. To tackle this problem, we augment the RL objective with an auxiliary task that guides part of the architecture to learn a valid probabilistic model as an inductive bias. We demonstrate that our method achieves state-of-the-art regret results against various baselines in experiments on standard hyperparameter optimisation tasks and also outperforms others in the real-world problems of mixed-integer programming tuning, antibody design, and logic synthesis for electronic design automation.

End-to-End Meta-Bayesian Optimisation with Transformer Neural Processes
Alexandre Maraval" Huawei Noah's ' Ark Lab al alexandre mwravall waw.lll@hawwe..commmmm com
Matthieu Zimmer* Huawei Noah's 's Ark Lab matthieu. zimmer@huawei......m.... com
Antoine Grosnit Huawei Noah's Ark Lab Technische Universität Darmstadt
Haitham Bou Ammar
Huawei Noah's ' Ark Lab, University College London haitham.c aamar@hhuwwaa com
antoine.grosni gww.sitttnuaawaae com
antoine.
Abstract
Meta-Bayesian optimisation (meta-BO) aims to improve the sample efficiency Bayesian optimisation by leveraging data from related tasks. While previous methods successfully meta-learn either surrogate model tion independently, joint training of both components remains an open challenge. This paper proposes the first end-to-end differentiable meta-BO framework that generalises neural processes to learn acquisition functions via transformer archi- tectures. We enable this end-to-end framework with reinforcement learning (RL. to tackle the lack of labelled acquisition data. Early on, we notice that training transformer-basee neural processes from scratch with RL is challenging due insufficient supervision, especially when rewards are sparse. We formalise this claim with combinatorial analysis showing that the widely used notion of regret as reward signal exhibits logarithmic sparsity pattern in trajectory lengths. To tackle this problem, we augment the RL objective with an auxiliary task that guides part of the architecture to learn a valid probabilistic model as an inductive bias. We demonstrate that our method achieves state-of of off-the-rt art regret results against various baselines in experiments on standard hyperparameter optimisation tasks and also outperforms others in the real-world problems of mixed-integer programming tuning, antibody design, and logic synthesis for electronic design automation.
acquisition fune-
Introduction
Bayesian optimisation (BO) techniques are sample-efficient sequential model-based solvers opti- mising expensive-oo-evalutte black-box objectives. Traditionally, BO methods operate in isolation focusing on one task at a time, setting that led to numerous successful applications, including but not limited to hyperparameter tuning [I], drug and chip design [2). and roboties [3] Although successful, focusing on only one objective isolation may increase sample complexities on new tasks since standard BO algorithms work tabula-rasa they start optimising from scratch, ignoring previously observed black-box functions.
To improve sample efficiency on new target tasks, meta-BO makes use of data collected on related source tasks [4] and attempts to transfer knowledge in between. Those methods are primarily composed of two parts: 1) (meta) surrogate model predicting the function to optimise and 2) (meta) acquisition function (AF) estimating how good queried location is. Regarding surrogate
"These authors contributed equally to this work
37th Conference on Neural Information Processing Systems (NcurIPS 2023).

---

modelling, prior work relies on Gaussian processes (GPs) to perform transfer across tasks due to their data efficiency [. or focuses on deep neural networks [2 gaining from their representation flexibility. As for acquisition functions, previous works assumed a fixed GP and trained neural 14
network to perform transfer
Although successful, those methods rely on the assumption that the two steps described above can be handled independently, potentially missing benefits from considering both steps together. Therefore, this paper advocates for an end-to-end training protocol for meta-BO where we train model predicting AF values directly from observed data, without relying on GP surrogate.
As shown in TO 12 [5] [16], Neural processes (NP) are a good candidate for meta-learning due to their structural properties, thus propose to use new transformer-basee NP to model the acquisition function directly [17.
Nevertheless, the lack of labelled AF data prohibits supervised learning protocol. Here, we rely on reward functions to assess the goodness of AFs and formulate new reinforcement learning (RL. problem. Our RL formulation attempts to learn an optimal policy that selects new evaluation probes by minimising per-task cumulative regrets.
Early on, we faced very unstable training with minimal learning due to the combination of RL with transformers [18]. Furthermore, we notice that this problem amplifies when the reward function is sparse, which is the case in our setting, as we formally show in Section 3.2 To mitigate this difficulty, we introduce an inductive bias in our transformer architecture via an auxiliary loss [9] that guides a part of the network to learn valid probabilistic model, effectively specialising as a neural process with gradient updates. Having developed our framework, we demonstrate state-of-the-arr regret performance in hyperparameter optimisation of real-world problems and combinatorial sequence optimisation for antibody and chip design (see Section 4 Those solid empirical results show the value of end-to-end training in meta-BO to further improve sample complexity.
Contributions We summarise our contributions as follows: developing the first end-to-end transformer-bee based architecture for meta-BO predicting acquisition function values, ii) identify- ing logarithmic reward sparsity patterns hindering end-to-end learning with RL, iii) introducing NP-based inductive biases for successful model updates, and iv) demonstrating new statc-of-the-- regret results on a wide range of traditional (hyperparameter tuning) and non-traditionll (chip and antibody design) real-world benchmarks.
2 Background
2.1 Bayesian Optimisation
In BO, we sequentially maximise an expensive-to-evaluate black-box function f(x) for variables x E X. BO techniques operate in two steps. First, based on previously collected evaluation data, we fit probabilistic surrogate model to emulate fæx allowing us make probabilistic predictions of the function's 's behaviour on unobserved input points. Given the first step's probabilistic predictions, the second step in BO optimises an AF that trades off exploration and exploitation find new input query. Tnew E X, for evaluation. Both the model and acquisition choices play a critical role in the success of BO. Many works adopt GPs as surrogate models due to their sample efficiency and practical uncertainty estimation (20] Moreover, on the acquisition side, the common practice is to optimise one from set of widely adopted AFs such as Expected Improvement (EI) 21
2.2 Transfer in Bayesian Optimisation
Transfer techniques in BO 1 aim to improve the optimisation of newly observed largel black- box function by reusing knowledge from past experiences gathered in related source domains, i.e. solve maxzex /Tanger (x+) by leveraging information from K source black-boxes Sí (z),..., fx (xr). We assume this information is available to our agent via Di,..., DK such that each dataset Dj =
(ax (k) (8) } consists of nz (noisy) evaluations of fx( for all & E [1 K]. Following the meta-neg literature [4] 5 1 we impose no additional assumptions on the process that collects Di,... Dx. allowing for a diverse set of source optimisation algorithms, including but not limited to
3/
BO, genetic and evolutionary algorithms
[22], and sampling-han -aasd strategies (3] Many algorithms
that use source data to improve performance on target domains exist. We categorise those based
2

---

on the part they customise within the BO pipeline, i.e., by registering if they affect initial points 24 [5], search spaces 26]. surrogate models [5] or AFs [4]. Since our work devises end-to-end
meta-BO pipelines, we now elaborate on critical surrogate models needed for the rest of the paper and leave a detailed presentation of related work to Section 5
Although GPs [20] are a prominent tool for single-task BO, high computational complexity [27). among other drawbacks (e.g. smoothness assumptions or limited scalability in terms of dimensions), can limit their application to transfer scenarios. Of course, one can generalise GPs to support transfer using multi-task kernels [2 or GP ensembles [ for example. However, recent trends demonstrated superior performance to such extensions when using deep networks (e.g., neural processes) to meta-learn probabilistic surrogates [3]
Neural Processes A popular method for effective meta-learning consists of directly inputring context points (i.e., available data adjust target task) for model to adapt to unseen tasks [15]. We can accomplish such a goal by relying on neural processes (NPs), a class of models that combine the flexibility of deep neural networks with the properties of stochastic processes [10], Given an observation set of input-output pairs Dobs and set of Rerred locations x: (pred) at which we desire to make predictions, NP parameterised by 0 outputs distribution pe [ Dots) that approximates the true posterior over the labels y (fpee)) Among the different parameterisations of (.).). e.g., in 31) transformer architectures [17] recently emerged as compelling choices of NPs 12). In this work, we also adopt transformer architecture inspired by this architecture' broad software support and state-of-the-al supervised learning results as reported in
2.3 Reinforcement Learning
In Section 3 we attempt to learn an end-to-end model that predicts acquisition values. Of course, we cannot fit the parameters of such model with supervised learning due to the lack of labelled acquisition data. RL [2) is a viable alternative for learning from delayed and non-differentiable reward signals in those cases. In RL, we formalise problems as Markov decision processes (MDP):
M = A, P. R. 7). where S and AA denote the state and action spaces, P S x A x S + [0, 1] the state transition model, R the reward function dictating the optimisation goal, and y E [0, 1) discount factor specifying the degree to which rewards are discounted over time. A policy π : S x A - [o, is an action-selection rule that is defined as a probability distribution over state-action pairs, where π (a; (s:) represents the probability of selecting action a; in state 8,. An RL agent aims to find an optimal policy x² that maximises (discounted) expected returns. For determining '², we the state-of-the-art proximal policy optimisation (PPO) algorithm (33
3 End-to-End Meta-Bayesian Optimisation
While transfer techniques in BO saw varying degrees of success in many applications, current approaches lack end-to-end differentiabilityy where model learning and acquisition discovery arise as two separate steps. Specifically, the surrogate model gradients hardly affect the acquisition network, and the acquisition's 's network gradients fail to back- proppppaaaatee to the surrogate's 's updates.
Enabling end-to-end transfer frameworks which we learn the surrogate and acquisition jointly hold the promise for more scalable and easier-to-deploy algorithms that are more robust to input data or task changes. Following such a framework, we also expect more accurate predictions that can lead to better regret results (see Section 4) since we optimise the entire transfer pipeline. including the intermediate probabilistic model and AF representations. Additionally, end-to-end training techniques allow us to mitigate the need for domain-speciff expertise and permit stable implementations that benefit fully from GPU and computing hardware.
The most straightforward way to enable end-to-end training in Bayesian optimisation is to introduce a deep network that acquires search variables and historical evaluations of black-box functions as inputs and outputs acquisition values after a set of nonlinear transformations. Of course, it is challenging to fit the weights of such a network due, in part, to lack of labelled acquisition data where search variables and history of evaluations are inputs and acquisition values are labels.
3

---

3.1 Reinforcement Learning for End-to-End Training
Our approach utilises RL fit the network's parameters 0 from minimal supervision, circumventing the need for labelled acquisition data. To formalise the RL problem, we introduce an MDP where:
State: 8; = (, 7] (history, BO time-step & budget) Action: a, = 2t (choice of new probe),
with Hi { ,1),,.... ,..., 1. denoting the history of black-box evaluations up-to the current time-step t. Adding the current BO step t and the maximum budget in our state variable 8, helps balance exploration versus exploitation trade-offs as noted in [4]. Our MDP's transition function is straightforward. updating Hy by appending newly evaluated points, ie.. M4+1 = Hy U {(x,, 3r)) and incrementing the time variable t. Regarding rewards, we follow the well-established literature 13, (44 and define rt maxi<ege y/ to correspond to simple regret. Given such an MDP, agent attempts to find parameterised policy mo which, when conditioned on 8t, proposes new probe DG that minimises cumulative regret (i.e., the sum of total discounted simple regrets).
Extensions to Multi-Task Reinforcement Learning The above MDP describes RL configuration that learns 0 in a single BO task. We now extend this formulation to multi-task scenarios allowing for meta-learning setup. To do so, introduce set of MDPs Mi..... MM with K being the total number of available tasks. This paper considers assume that all MDPs share the same state and action spaces and leave cross-domain extensions as an interesting avenue for future work. We define each MDP, Mg. as previously introduced such that:
multi-task learning scenarios. As such,
States: s (k) m7(,0) ((()) T(()
Actions:
(k) a zt (k) Rewards: r((h)
max (K) Igegk()
Vk.
Moreover, for each task k the transition model updates task-specifie histories with H +++1 = H1, LI
{ {(=; (k)) "tt { and increments {(k), Contrary to the single task setup, we now seek policy To which
performs well on average across K tasks:
T Σ
arg max (TTO arg max ER-
Bygees ((A)
r
(1)
(1)
Prr
where Prakks denotes the task distribution. Furthermore, the per-task history distribution Pro (71 jointly parameterised by and defined as:
(
(k) 11ro01 (T
(k) -1)
(ze ( -1 121100001 ... (k) (z( nn (z0(6))
Pro
me
(2)
with nn (x( denoting an initial (action) distribution from which we sample the first decision
(k) While it appears that off off-the-shelf PPO can tackle the problem in Equation | Section 3.2 details difficulties associated with such an RL formulation, noting that in BO situations, the sparsity of reward signals can impede the learning of the network's weights 0. Before presenting those arguments, we now differentiate from the closest prior art, clarifying critical MDP differences.
Connection & Differences to 13 Although we are the first to propose end-to and-to-end multi-task MDPs for meta-B BO, others have also used RL to discover AFs, for example. Our parameterisation archi- tecture and MDP definition significantly differ from the prior art, particularly from [3), the closest method to our work. Our approach uses a transformer-based deep network model to parameterise the whole pipeline (see Section 3.4. In contrast, the work in [3 assumes pre-trained fixed Gaussian process surrogate and uses mulli--ayer perceptrons only to discover acquisitions. Our state variable requires historical information from which we jointly learn probabilistic model and an acquisition end-to-end instead of requiring posterior means and variances of a Gaussian process model. Hence, our framework is more flexible, allowing to model non-smooth black-box functions while overcoming some drawbacks of GP surrogates, like training and inference times.
3.2 Limitations of Regret Rewards in End-to-End Training
To define Equation I we followed the well-established literature of meta-BO 4 13 and utilised simple regret reward functions. Although this choice is reasonable, we face challenges in applying such rewards in end-to-end training. Apart from difficulties associated with end-to-end training

---

of deep architectures [8]. our RL algorithm is subject to additional complexities when estimating gradients from Equation I due to the sparsity of the reward function. To better understand this problem, we start by noticing that for a reward component (((a) to contribute to the cumulative summation Ee It-, we need to observe a function value V/r have seen so far, i.e., V/ the average number of such informative events (when y/
(R) that outperforms all values we Ti (k) maxigeet Ve (8) During the early training stages of RI, we can quantify
(k) > maxist<t Ve) by a combinatorial caleulation of the number of cycles in permutation of T(*) elements, leading us to the following lemma.
argument that frames this calculation
Lemma 3.1. Consider a task with a horizon length (budget) T, and define Tt maxıgts: ye the simple regret as introduced in Equation 7 For history Hr. let mH denote the total number of informative rewards, i.e. the number of steps at which y/s > maxişeet yr. Under a random policy me. the number of informative events is logarithmic in T such that: Exopee [mx] = O (log T),,
where Png is induced by no as in Equation 2
We defer the proof of Lemma 3.1 Appendix A due space constraints. Here, we note that this result implies that the information contained in one sampled trajectory is sparse at the beginning of RL training when the policy acts randomly. Of course, this increases the difficulty of estimating informative gradients of Equation | when updating 0. One can argue that the sparsity described in Lemma [.]]only holds under random policies during the early stages of RL training and that sparsity patterns decrease as policies improve. Interestingly, simple regret rewards do not necessarily confirm this intuition. To realise this, consider the other end of the RI. training spectrum in which policies have improved to near optimality such that To - To. Because To has been trained to maximise regret, it will seek to suggest the optimal point of the current task, as early as possible in the BO trajectory. Consequently, the policy is encouraged to produce trajectories with even sparser rewards during later training stages, further complicating the problem of informative gradient estimates of Equation]
3.3 Inductive Biases and Auxiliary Tasks
Learning from sparse reward signals is well-known difficulty in the reinforcement learning literature
[32].
Many solutions, from imitation learning, (35] to exploration bonuses [36], improve reward
signals to reduce agent-environment interactions and enhance gradient updates. Others (37 attempt to
define more informative rewards from prior knowledge or via human interactions [88 Unfortunately.
both of those approaches are hard to use in BO. Indeed, manually engineering
ope-ifeccfic
rewards is notoriously difficult and requires domain expertise and extensive knowledge of the source and target black-box functions we wish to optimise. Furthermore, learning from human feedback is data-intensive, conflicting with the goal of sample-efficient optimisation.
Another prominent direction demonstrating significant gains is the introduction of auxiliary tasks (losses) within RL that allows agents to discover relevant inductive biases leading to impressive empirical successes [19 39 40). Inspired by those results, we propose introducing an inductive bias in our method via an auxiliary supervised loss. Since we have at our disposal the collected source tasks datasets on which we are training our architecture D(1) ..... D(K), we augment our objective such that our RL agent maximises not only rewards but also the likelihood of making correct predictions on these labelled datasets.
Supervised Auxiliary Loss Consider source task is and consider that we split its corresponding dataset into an observed set and a prediction set D(t) (R)
obs Li (()) We define the auxiliary perd loss to be exactly the likeliooddoood of functions values ( at predicted locations x (pred) given observations Dohs:
(pred) (( = E =E,- k- (ik) ((k) log (1 (v |
(prod) (D(k)) ].
Interestingly, this part of our model specialises as a neural process (Section 2.2 with ((k) being the history H4(4) and (pred) being the input points at which we wish to make predictions. To represent po(-), we use a head of our transformer to predict multi-modal Riemannian (or bar plot probability density function) posteriors as 11 41). We compute Equation 3 on random iid splits of D(k) and not directly trajectories generated by the policy. This is because the policy improves, the trajectories
5

---

it generates are composed of less diverse (non-iid.) points. Indeed as training progresses, To becomes better and therefore finds the optimal point in D(K) more rapidly. To fully take advantage of the labelled data at our disposal, we evaluate this auxiliary loss on iid. data, ie. splits of D(A) into obs and ((Ae sampled uniformly at random. It is sensible from a BO standpoint as well since we want the introduced inductive bias to encourage our network to maximise the likelihood not only in a neighbourhood of the optimiser but also in the rest of the dataset.
3.4 Neural Acquisition Processes
We now introduce in more detail our transformer architecture and note some of its important properties. We call this architecture the neural acquisition process (NAP) because it is a new type of NPs jointly predicting AF values and distribution over actions. Similarly other NP [ and queried locations as input. We parameterise it by 0 and denote the acquisition prediction by og (pre)) H. t, T).
takes context
Since ag outputs real numbers, we still need to define how we can obtain a valid probability dis- tribution over the action space to form a policy
Algorithm 1 Neural Acquisition Process training.
To (-1,)) Defining such a distribution over the
Require: Source tasks training data {D(K) ]=
1==1
whole action space is hard if A is continuous. Hence, similarly to Volpp et al. [13.. we evaluate the policy тh only on the fimite set of locations (pree)) for given task] Therefore, we have:
initial parameters 9, budgets T(k) = T, dis- count factor 7, learning rate !]
for each epoch do
select task & and dataset D(k), set Ho = {0}
for 1 1..... do
( (7n8 H.,T.T
(pred) no [27
81)
x Σ, Fppoal coola) (H,T,T)
I2 ~ To (-((
b predict action
y/ = fx) (24))
> execute action
T. Vze
collect reward update hist.
We now have all the necessary components to define the full objective combining Equation [ 3 [()) (((m)) + (O), where À is a hy- perparameter balance between the objec- tives. We summarise the full training algorithm in Alg. detailing each of the components.
He+1 + He U (x),
end for
R 2t-1 7tr
a cumul. reward b split source data
D + Doba U D
C. = po ( (pred) jz (prod) Date)
D aux. loss update
x- n( VoR+ VoL)
Finally, we study some desirable properties of NAPs, explain how we can achieve them, and why they are important in the context of BO.
end for
Property 3.2 (History-order invariance). An NP is history-order invariant for any choice of permutation function that changes the order of the points in history, g(æ, vHH)) = g(æ, H).
Unlike vanilla transformers, we do not use a positional encoding. It allows NAP to treat the history H as a set instead of an ordered sequence 42 and to be invariant to the order of the history. To form an (x, y) pair in this set, we sum the embedding of and y [II], whereas for queried locations only use the embedding of x for a token (see Fig. [ left). is important for BO since the order in which we collect the points is not relevant for assessing how promising is a new queried location. Previous meta-RI. algorithms [43 also shown the importance of relying on order-invariance.
'ith continuous action space, could optimise x (ped) to maximise the NAP outputs before forming the distribution over the discrete actions.
Table 1: We compare the properties of different transformer architectures. is the number of tokens needed encode the meta-data, and D denotes the dimensionality of X. History-order inv. Query ind. AF values
Tokens
NAP (ours) TNPs [2
t Mpred
x
x
t Mpred
OptFormer
x
x
L. + (D + 2)(t + (pree)
PFN 11
X
+ Upred

---

Property 3.3 (Query independence). A NP is query independent if for any choice of queried locations (plred) ( aefrrd)), we have g ((pllrd)) H.) (g( (ord) ....... g (o H)).
In NAP, every token in the history access each other through the self-attention mask, whereas the elements of a(pred) can only attend to themselves and to tokens in the history H (see Fig. | right).
Because the tokens inside a!(eed) cannot access each other and we do not use positional encoding. NAP is query independent, which is important to make consistent predictions of AF values for BO they should not depend the other queried locations. Additionally, NAP is fully differentiable enabling end -o-ed training but also optimisation of the queried locations via gradient ascent for continuous action spaces. In Table we highlight the differences with other state-of ttethhrtt models regarding those properties.
arl (
agl (2
201; VI
222 M2
25, Us
2r 51
Self-attention layer (x n)
222: 1/2
333:
(peed)
Embedding
31
22
1/2
22
(mnad)
ang
(mall)
T
(reed)
Figure 1: Our proposed NAP architecture (left) and example of the masks applied during inference (right). We apply independent embedding on ai yi. I and T. The colored squares mean that the tokens on the left can attend the tokens on the top in the self-attention layer.
4 Experiments
We conduct experiments on hyperparameter optimisation (HPO) and sequence optimisation tasks to assess NAP's efficiency. Regarding HPO, we run our algorithm on datasets from the HPO-B benchmark [44] and in real-world settings of tuning hyperparameters for Mixed-Integer Programming (MIP) solvers. For sequence optimisation, we test NAP on combinatorial black-box problems from antibody design and synthesis flow optimisation for electronic design and automation (EDA).
Baselines We compare our method against popular meta-learning baselines, including few-shot Bayesian optimisation (FSBO) [5], MetaBO [3 as well as OptFormers [I] when applicable. Moreover, we show how classical GP-based BO, equipped with an adequate kernel and an El acquisition function, which we title GP-EI, performs across domains. We first GP on the meta- training datasets to enable fair comparison. We initialise the GP model with those learnt kernel parameters test time. This way, the GP baseline can benefit from the information provided in the source tasks. We also explore training neural process directly on source task datasets and then using a fixed acquisition. For that, we introduce NP-EI that combines the same base architecture from with EI. Additionally, we contrast NAP against random search (RS).
Following the standard practice in meta-BO, we report our results in terms of normalised regrets for casier comparison across all tasks. We attempt to re-implement all baselines across all empirical domains, extending previous implementations as needed, e.g., developing MetaBO [3] and FSBO
5 versions for combinatorial and mixed spaces to enable a fair comparison in MIP solver tuning, antibody and EDA design tasks.
Remark on OptFormer We re-run OptFormer with an El AF on hyperparameter tuning tasks by extending the implementation in [1 to the discrete search space version of HPO-B. However, the lack of a complete open-source implementation and interpretable meta-data in the other benchmarks (c.g.. in antibody design and EDA) prohibited successful execution.
We adapt the GP kernel to each task depending on the nature of the optimisation variable, e.g., Matérn in continuous domains, categorical for combinatorial search space, and mixed when having both.
7

---

4.1 Hyperparameter Optimisation Results
Hyperparameter Optimisation Benchmarks We experiment on the HPO-B benchmark 44 which contains datasets of (classification) model hyperparameters and the models' corresponding accuracies across multiple types and search spaces. Due to resource constraints, we selected six representative search spaces. Nonetheless, we chose the search spaces represent all underlying classification models in the experiment. We also always picked the ones with the least points to focus on the low data regime performance; see Appendix [.T] for more details. The results of our tests in Figure 2 demonstrate that NAP and OptFormer outperform all other baselines. Surprisingly. although NAP uses a much smaller architecture than OptFormer (around 15 million parameters vs 250 million for OptFormer) and trains much less data (around 80k original points on average versus more than 3 million points for OptFormer), its regret performance statistically similar to that of the OptFormer after 100 steps. Moreover, on the same GPU, to perform the same inference, NAP only uses 2% of OptFormer's 's compute time and around 40% of its memory usage.
Tuning MIP Solvers Apart from HPO-B, we consider another real-worid example of hyperparameter tuning that requires finding the optimal parameters of MIP solvers. We use the open-source SCIP solver 45) and the Benchmark suite from the MIPLib2017 [46] that consists of a collection of 240 problems. The objective to find a set of hyperparameters so that SCIP can solve MIP instances in minimal time. Our high-dimensional search space comprises 135 hyperparameters with mixed types, including boolean, integers, categories and real numbers. We train our model on data collected from BO traces on 103 MIPs and test a held-out set of 42 instances. Our results in Figure 2 demonstrate that NAP is capable of outperforming all other basclines reaching low regret about an order of magnitude faster than FSBO [5] Figure 2 further demonstrates the importance of end-to-end training where NAP again outperforms NP-EI.
4.2 Sequence Optimisation Experiments
Now, we demonstrate NAP's abilities beyond hyperparameter tuning tasks in two real-world combi- natorial black-box optimisation problems.
Antibody CDRH3-Sequence Optimisation This experiment focuses on finding antibodies that can bind to target antigens. Antigens are proteins, i.c., sequences of amino acids that fold into a 3D shape giving them specific chemical properties. A protein region called CDRH3 is decisive in the antibody's 's ability to bind to target antigen. Following the work in 47 we represent CDRH3s as string of 11 characters, each character being the code for a different amino acid an alphabet of cardinality 22. The goal is to find the optimal CDRH3 that minimises the binding energy towards a specific antigen. Binding energies can be computed using state-of-the-ar simulation software like Absolut! 48) We collected datasets of CDRH3 sequences and their respective binding energies (with Absolut!) across various antigens from the protein database bank [49]. We then formed a transfer scenario across antigens where meta-learn on 109 datasets, validate 16, and test NAP on 32 new antigens. Our results in Figure 2 but can also outperform all other baselines in combinatorial domains.
indicate that NAP is not limited to hyperparameter tuning tasks
Electronic Design Automation (EDA) Logic synthesis (L.S) is an essential step in the EDA pipeline of the chip design process. At the beginning of LS, we represent the circuit as an AIG (an And-Inverter- Graph representation of Boolean functionn) and seek to map it to a netlist of technology-dependent gates (e.g.. 6-input logic gates in FPGA mapping). The goal in LS is to find sequence of graph transformations such that the resulting netlist meets an objective that trades off the number of gates (area) and the size of the longest directed path (delay) in the netlist. We perform sequence of logic synthesis operators dubbed a synthesis flow to optimise the AIG.
Following [50]. we consider length 20 LS flows and allow an alphabet of 11 such operators, e.g., (refactor, resub, ... balance) as implemented in the open-source ABC library (5T). We collected datasets for 43 different circuits. Each dataset consisted of 500 sequences (collected via a Genetic algorithm optimizer) and their associated area and delay. Additionally, we applied the well-known heuristic sequence resyn2 on each circuit to get a reference area and delay. For this task, the black-box takes a sequence as input and returns the sum of area and delay ratios with respect to the reference ones, as detailed in Appendix B. We train all methods on 30 circuits from OpenABC [50] validate on 4 and test on 9. Our results Figure again demonstrate that NAP outperforms all other baselines by significant margin.
8

---

HP0-8
MIP
Antibody
FDA
Average Regret
10-
10
10
10-1
10
10-2
10-²
10.7
10-3
0
23
50
75
100
10
10
10 0
100
200
300
10
100
200
300
R5 GP-E1 MetaBO
FS80
NP-EI
OptForer-B
NAP
Figure 2: Average regret vs. BO iterations with initial points. (Left) Results on search spaces on the HPO-B benchmark. (Middle-left) Results tuning SCIP for solving 42 different MIPs. (Middle- right) Antibody CDR3 sequence optimisation on 32 test datasets corresponding to 32 different antigens. (Right) Logic synthesis operator sequence optimisation on 9 test datasets corresponding to 9 different circuits. For each method, error bars show confidence intervals computed across runs on HPO-B and 10 runs all the others.
5 Related Work
Meta-Learning Paradigms Meta-learning aims to learn how to quickly solve a new task based on the experience gained from solving or observing data from related tasks (15). To achieve this goal, we can follow two main directions. In the first one, the meta-learner uses source data to learn good initialisation of its model parameters, such that it is able to make accurate predictions after only few optimisation steps on the new target task [52]. The second one is more ambitious as it endeavours to directly learn, from the related tasks, general rule predict a helpful quantity suggest the next point (black-box value, acquisition function value, etc.) from a context of observations (i.e. our history (())). Works following that direction, which include ours, can rely on different types of models to learn this general rule, such as NPs [TO,, recurrent neural networks [53], HyperNetworks [[6] or transformers [12 Orthogonal to these two directions is the learning of soyper-posterre the datasets to meta- train on. In that line of research, Rothfuss et [54] suggest the use of Stein Variational Gradient Descent (SVGD) to estimate uncertainty with multiple models better, and Hsieh al. [14] extend the use of SVGD to meta-learn AFs. We consider those last two works as an orthogonal direction to our, as NAP could also benefit from SVGD updates.
on
Learning a Meta-Model Chen al. 553 and TV al. [56] train an RNN to predict what should be the next suggestion in BO instead of predicting the acquisition function value. We do not compare to their method given the lack of available implementation. Still, we compare to the outperforming approach developed by Chen et al. [41] based on a transformer architecture designed to do meta-BO on hyperparameter tuning problems. Their OptFormer is trained on related tasks over different search spaces to output the next point to evaluate and predict its value. The next point is sequentially decoded, one token at time, and exploits the hyperparameters' names or descriptions improve generalisation across tasks. Contrary to NAP, OptFormer is not designed to predict acquisition value at any point and does not meet the two properties 3.2 and 3.3 and therefore needs much more training data to predict the proper sequence of tokens. We note that NAP does not rely variable descriptions, making easily deployable on various tasks and still very competitive in the hyperparameter optimisation context.
Rather than learning an entire predictive model from scratch, prior works [5| learn deep kernel surrogates, i.e. warping functions mapping the input space with a neural network before it is given to GP. Learning only the input transformation allows the authors to rely on the closed-form posterior prediction capacity of standard GP models. To perform the transfer, Feurer et al. (6) rely on an ensemble of GPs. Iwata [57] is an approach close to FSBO 5 as it learns a meta-model by meta-training Deep Kernel GP. Notably, it does in end-to-end fashion using RL to propagate gradients through the acquisition function and the GP back to the deep kernel. It does not, however, learn meta-acquisition.
Learning a Meta- Aeti-iiiuuiiioonn Funetion Hsieh al. 14 and Volpp et al. [13 choose to perform transfer through the acquisition function. They use directly GP surrogates and define the acquisition as a neural network that meta-trained on related source tasks. They first pre-train GP surrogates on all source tasks and fix their kernel parameters. They then rely on RL training to meta-leam the

---

neural acquisition function that takes as inputs the posterior mean and variance of the GP surrogate (which is itself trained online at test time). At test time, they allow for update in the GP but keep the weights of the neural acquisition fixed.
In summary, the methods meta-learning AFs do not do in end-to-end fashion and still rely on trained GP surrogates. While these methods are principled and competitive, they suffer from the cost of inverting the GP kernel matrix (cubic in the number of observations). In comparison, NAP can make predictions through a simple forward pass. On the other hand, the methods that learn meta-mode meea--odel,, either use a Deep Kernel GP (suffering the same cost) or have to learn large model from scratch, costing a lot of budgets for collecting data beforehand, as well a pre-training time. Both of them use standard acquisition functions, missing the potential benefits of doing transfer in acquisition between tasks.
The performance of some of those algorithms HPO-B is presented in Appendix Shhowiin-sshowing Showing that despite learning an architecture from scratch, NAP achieves a lower regret. For more detailed survey on transfer and meta-learning in BO, we refer the reader to Bai al.
6 Conclusion
We proposed the first end-to-end training protocol to meta-learn acquisition functions in meta-BO. Our method predicts acquisition values directly from history of points with meta-reinforcement learning and auxiliary losses. We demonstrated new state-of-the-an results compared to popular baselines a wide range of benchmarks.
Limitations & Future Work Our architecture suffers from the usual quadratic complexity of the transformer in terms of the number of tokens which limits the budget of BO steps. Nevertheless, can still handle around 5000 steps, which is enough for most BO scenarios. Another limitation of our architecture is that we need train a new model for each search space. In future, plan to enable our method to leverage meta-training from multiple search spaces and investigate how we could design data-driven BO-specific augmentation further mitigate meta-overfitting (58)
Acknowledgements
We would like to thank Prof. Frank Hutter and his team for their constructive feedback and for the availability of their benchmark code and datasets. We also would like to thanks Massimiliano Patacchiola for his comments during the writing phase of the paper.
This work is supported by the CSTT on Generalisable Robot Learning via Machine Learning Models 2100332-GB and by the 2030 "New Generation of AI" Major Project of China under grant No. 20227D001 16408.
References
[1] Jasper Snoek, Hugo Larochelle, and Ryan P. Adams. Practical bayesian optimization of machine learning algorithms. In Peter L. Bartlett, Fernando C. N. Pereira, Christopher J. C. Burges, Léon Bottou, and Kilian Q. Weinberger, editors, Advances in Neural Information Processing Systems 25: 26th Annual Conference on Neural Information Processing Systems 2012. Proceedings of meeting held December 3-6, 2012, Lake Tahoe, Nevada, United States, pages 2968, 2012. URL
neurips cc/
[2] Nate Gruver, Samuel Stanton, Polina Kirichenko, Mare Finzi, Phillip Maffettone, Vivek Myers, Emily Delaney, Peyton Greenside, and Andrew Gordon Wilson. Effective surrogate models for protein design with bayesian optimization. In ICML Workshop on Computational Biology, 2021.
[3] Fabio Muratore, Christian Filers, Michael Gienger, and Jan Peters. Data-efficient domain randomization with bayesian optimization. IEFE Robotics and Automation Letters, 6(2): 911-918, 2021.
10

---

[4] Tianyi Bai, Yang Li, Yu Shen, Xinyi Zhang, Wentao Zhang, and Bin Cui. Transfer learning for bayesian optimization: A survey, 2023.
[5] Martin Wistuba and Josif Grabocka. Few-shot bayesian optimization with deep kernel surro- gates. In 9th International Conference on Learning Representations, ICLR 2021, Virtual Event, Austria, May 3-7, 2021. OpenReview.net, 2021. URL /ttpss//opppeeeeiiiin wttp:/iwwwww.w........eeeeee net/forum?
id id=bxxgv5C3sYc
[6] Matthias Feurer, Benjamin Letham, and Eytan Bakshy. Scalable meta-learning for bayesian optimization using
gaussian process ensembles. In AutoML. Workshop at
ICML, volume 7, 2018.
[7] Massimiliano Patacchiola, Jack Turner Elliot J. Crowley, Michael O' Boyle, and Amos J Storkey. Bayesian meta-learning for the few-shot setting via deep kernels. In H. Larochelle, M. Ranzato, R. Hadsell, M.F. Balcan, and H. Lin, editors, Advances in Neu- ral Information Processing Systems, volume 33, pages 65108-1618.. Curran Associates. Inc., 2020. URL httpp///////ooooiiigg www.edddiiinnnnnuipippeeee neurips..... wwt.cc/pppppp... files/ //tp://///////////////////////// pptppppppp///////// 0096666662277$044000007777772--3
pdf
[8] Zi Wang, George Dahl, Kevin Swersky, Chansoo Lee, Zachary Nado, Justin Gilmer, Jasper Snoek, and Zoubin Ghahramani. Pre-trained gaussian processes for bayesian optimization, 2023.
[9] Wenlin Chen, Austin Tripp. and José Miguel Hemmández-Lobato. Meta-learning adaptive deep kernel gaussian processes for molecular property prediction. In The Eleventh International Conference on Learning Representations, 2023. URL https://opeereviiee wwwp://w.eeeffforrreoeeo
1d=KXRShOsdVTP)
[10] Marta Garnelo, Dan Rosenbaum, Christopher Maddison, Tiago Ramalho, David Saxton, Murray Shanahan, Yee Whye Teh, Danilo Jimenez Rezende, and S. M. Ali Eslami. Conditional neural processes. In Jennifer G. Dy and Andreas Krause, editors, Proceedings of the 35th International Conference on Machine Learning, ICML 2018. Stockholmsmässsan, Stockholm, Sweden, July 10-15, 2018, volume 80 of Proceedings of Machine Learning Research, pages 1690-1699. PMLR, 2018. URL /ttp::tt///prroeeddingg mlr www.spressssvi htrp://s........eeee//o////a/aaa oaapreeee//////g/eallllaalaaaaa gato://areeell.a.eg..mo
[11] Samuel Müller, Noah Hollmann, Sebastian Ainda-Arangee Josif Grabocka, and Frank Hutter. Transformers can do bayesian inference. In The Tenth International Conference on Learning Representations, ICLR 2022, Virtual Event, April 25-29, 2022. OpenReview.net, 2022. URL. https://openrevveex net. formm -dKgglgKb-KKKKF
[12] Tung Nguyen and Aditya Grover. Transformer neural processes: Uncertainty- aware meta learning via sequence modeling.
Kamalika Chaudhuri, Stefanie Jegelka, Le Song, Csaba Szepesvári, Gang Niu, and Sivan Sabato, editors, International Conference on Ma- chine Learning. ICMI. 2022, 17-23 July 2022, Baltimore, Maryland, USA, volume 162 of Proceedings of Machine Learning Research, pages 16569-16594, PMLR, 2022. URL https://prooeeddngg: mlr pptps//v1///ggueen.iiio htmll
[13] Michael Volpp. Lukas P. Fröhlich, Kirsten Fischer. Andreas Doerr, Stefan Falkner, Frank Hutter, and Christian Daniel. Meta-learning acquisition functions for transfer learning in bayesian optimization. In 8th International Conference on Learning Representations, ICLR 2020, Addis Ababa, Ethiopia, April 26-30, 2020. OpenReview.net, 2020. URL. htttp:://pperreiieee pet/fforum?i=-ryeppp..
[14] Bing-Jing Hsieh, Ping- Chun Hsieh, and Xi Liu. Reinforeed few-shot acquisition function learning for bayesian optimization. In M. Ranzato, A. Beygelzimer, Y. Dauphin, P.S. Liang, and Wortman Vaughan, editors, Advances in Neural Information Processing Systems, volume 34, pages 7718-7731. Curran Associates, Inc., 2021. URL htt.h:ttt:///tcttm pttpttpp:///pooeeedneggeee neurips
[15] Timothy M. Hospedales, Antreas Antoniou, Paul Micaelli, and Amos J. Storkey. Meta-learning in neural networks: A survey. IEEE Trans. Puttern Anal. Mach. Intell., 14(9):5149-5166 2022. doi: 20T0////AAM.222113077777
https:/ :/dt:i:///..i.. http//dii.ogg//////// orgg/10gg.0... 11199/....111/PPAAAA 2021
3079209

---

[16] David Ha, Andrew M. Dai, and Quoe V. Le. Hypemetworks. In 5th International Conference on Learning Representations, ICLR 2017, Toulon, France, April 24-26. 2017, Conference Track Proceedings. OpenReview.net, 2017. URL htttps://openreeiiiieie net/ net/forum?id- rkpACellx
[17] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. In Isabelle Guyon, Ulrike von Luxburg, Samy Bengio, Hanna M. Wallach, Rob Fergus, V. N. Vishwanathan, and Roman Garnett, editors, Advances in Neural Information Processing Systems 30: Annual Conference on Neural Information Processing Systems 2017, December 4.9, 2017, Long Beach, CA, USA, pages 5998-6008, 2017. URL https: httpps///pooeeeeenggg... neuripe ctteriis.cccpppepp cttccc//pppp pwppp/appppaaappappappppppppaaap 2017 201r//20//////////
#ffe222564/dde1111b003ololloaaa htt.://sss..cccaaaaaaaaaaataaaa. Abstract.htmm
[18] Emilio Parisotto, Francis Song, Jack Rae, Razvan Pascanu, Caglar Gulcehre, Siddhant Jayaku- mar, Max Jaderberg, Raphaël Lopez Kaufman, Aidan Clark, Seb Noury, Matthew Botvinick, Nicolas Heess, and Raia Hadsell. Stabilizing transformers for reinforcement learning. In Hal Daumé IIII and Aarti Singh, editors, Proceedings of the 37th International Conference Ma- chine Learning, volume 119 of Proceedings of Machine Learning Research, pages 7487-7498. mlr prw.s/ /1199
PMLR, 13-18 Jul 2020. URL
html
[19]
Max Jaderberg, Volodymyr Mnih, Wojeiech Marian Czarnecki, Tom Schaul, Joel Z. Leibo, David Silver, and Koray Kavukcuoglu. Reinforcement learning with unsupervised auxiliary tasks. In 5th International Conference on Learning Representations, ICLR 2017, Toulon, France, April 24-26, 2017, Conference Track Proceedings. OpenReview.nel, 2017. URL htt//forummiddSJyyPD555.
[20] Carl Edward Rasmussen and Christopher KI Williams. Gaussian Processes for Machine Learning, volume 2. MIT press Cambridge, MA, 2006.
[21] Jonas Mockus, Vytautas Tiesis, and Antanas Zilinskas. The application of Bayesian methods for seeking the extremum. Towards Global Optimization, 2(117-129):2, 1978.
[22] Thomas Bäck and Hans-Paul Schwefel. An overview of evolutionary algorithms for parameter optimization. Evolutionary computation, 1(1):1-23, 1993.
[23] James Bergstra and Yoshua Bengio. Random search for hyper- parameter optimization. Journal of machine learning research, 13(2), 2012.
[24] Matthias Feurer, Jost Tobias Springenberg, and Frank Hutter. Initializing bayesian hyperparam- eter optimization via meta-learning. In Blai Bonet and Sven Koenig, editors, Proceedings of the Twenty-Ninth AAAI Conference on Artificial Intelligence, January 25-30, 2015, Austin, Texas, USA, pages 1 128-11335 AAAI Press, 2015. BRL/http://wwwr aaai. org/ ocs/ index. php/ AAAI/AAAT15,
http://ppeeeeeeeeeeeeeeeee AATTAAAA155/ppprr///////////
[25] Martin Wistuba, Nicolas Schilling, and Lars Schmidt-Thieme. Scalable gaussian process-based transfer surrogates for hyperparameter optimization. Mach. Learn., 107(1):43-78, 2018. doi: WrRLhttps:to //doi http://ooo.or//////////// 10. 10077//109994007--6664
00.1007/1109944-17-66666
[26] Valerio Perrone and Huibin Shen. Learning search spaces for bayesian optimization: Another view of hyperparameter transfer learning. In Hanna M. Wallach, Hugo Larochelle, Alina Beygelzimer, Florence d' d'Allhé-Bur Buc, Emily B. Fox, and Roman Garnett, editors, Advances in Neural Information Processing Systems 32: Annual Conference on Neural Information Processing Systems 2019, NeurIPS 2019. December 8-14, 2019, Vancouver. BC, Canada, pages 12751-12761, 2019. htRLsttps:: httttss:://roceeddiiii neur cc. wptppcpppppee///1/../..//////// http:/ppaa//21//asall attaar/884bb88555aaaaaaaaaaaaee. htt.:///eeecccaaa---bbrrrtttttta Abstract.html
[27] Pavel Izmailov, Alexander Novikov, and Dmitry Kropotov. Scalable gaussian processes with billions of inducing inputs via tensor train decomposition. In Amos J. Storkey and Fernando Pérez-Cruz, editors, International Conference on Artificial Intelligence and Statis- fics, AISTATS 2018, 9-11 April 2018, Playa Blanca, Lanzarote, Canary Islands, Spain, vol- ume 84 of Proceedings of Machine Learning Research, pages 726-735. PMLR, 2018. URL http://proceeddnng mlr. www.ppresssseee http://////mmmmllv11..a html
12

---

[28] Edwin V. Bonilla, Kian Ming Adam Chai, and Christopher K. 1. Williams. Multi- task gaussian process prediction. In John C. Platt, Daphne Koller, Yoram Singer, and Sam T. Roweis, editors, Advances in Neural Information Processing Systems 20, Pro- ceedings of the Twenty-Firs Annual Conference on Neural Information Processing Sys- tems, Vancouver, British Columbia, Canada, December 3-6, 2007, pages 153-160. Curran Associates, Inc., 2007. URL httpp::/prooceedingg wtw.eeeediinnn.neuuiispss.. neurips cc/ pwwpp//000 / www.bbb.b..bbttt....h.....
[29] Marta Garnelo, Dan Rosenbaum, Christopher Maddison, Tiago Ramalho, David Saxton, Murray Shanahan, Yee Whye Teh, Danilo Jimenez Rezende, and S. M. Ali Eslami, Conditional neural processes. In Jennifer G. Dy and Andreas Krause, editors, Proceedings of the 35th International Conference on Machine Learning, ICMI. 2018, Stockholmsmüsxxnn Stockholm, Sweden, July 10-15, 2018, volume 80 of Proceedings of Machine Learning Research, pages 1690-1699. PMLR, 2018. WtLthttt://prett
mlr, wwwps/vv0/
[30] Jonathan Gordon, Wessel Bruinsma, Andrew Y. K. Foong, James Requeima, Yann Dubois, and Richard E. Turner Convolutional conditional neural processes. In 8th International Conference on Learning Representations, ICLR 2020, Addis Ababa, Ethiopia, April 26-30, 2020. OpenReview.net, 2020. URL
[31] Hyunjik Kim, Andriy Mnih, Jonathan Schwarz, Marta Garelo,, S. M. Ali Eslami, Dan Rosen- baum, Oriol Vinyals, and Yee Whye Teh. Attentive neural processes. 7th International Conference Learning Representations, ICLR 2019, New Orleans, LA, USA, May 6-9, 2019. OpenReview.net, 2019. URI. http:///.//////eaaaaaae/e/////ea
[32] Richard S. Sutton and Andrew G. Barto. Reinforcement learning an introduction. Adaptive computation and machine learning. MIT Press, 1998. ISBN 978- 978-0-262-119988-6 URL https
wwww.worllcaat /
[33] John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov, Proximal policy optimization algorithms. CoRR, abs/1707.06347,, 2017. URL attp://arxiv org/
abs/ 1tte/1077.000 06347
[34] Niranjan Srinivas, Andreas Krause, Sham M. Kakade, and Matthias W. Seeger. Gaussian process optimization in the bandit setting: No regret and experimental design. In Johannes Fürnkranz and Thorsten Joachims, editors, Proceedings of the 27th International Conference on Machine Learning (ICML-10), June 21-24, 2010, Haifa. Israel, pages 1015-1022. Omnipress, URLhtttss //ccml..mmmmmm/mm /ct///ooeeeeccee/10///ee//// /etppprrr///...... pdf
2010.
[35] Jonathan Ho and Stefano Ermon. Generative adversarial imitation learning. Advances in neural information processing systems, 29, 2016.
[36] Haoran Tang, Rein Houthooft, Davis Foote, Adam Stooke, Open Xi Chen, Yan Duan, John Schulman, Filip DeTurck, and Pieter Abbeel. # exploration: A study of count-based exploration for deep reinforcement learning. Advances in neural information processing systems, 30, 2017.
[37] Andrew Y Ng, Daishi Harada, and Stuart Russell. Policy invariance under reward transforma- tions: Theory and application reward shaping. In ICML, volume 99, pages 278-287. Citeseer, 1999.
[38] Paul F Christiano, Jan Leike, Tom Brown, Miljan Martic, Shane Legg, and Dario Amodei. Deep reinforcement learning from human preferences. Advances in neural information processing systems, 30, 2017.
[39] Xingyu Lin, Harjatin Baweja, George Kantor, and David Held. Adaptive auxiliary task weighting for reinforcement learning. Advances in neural information processing systems, 32, 2019.
[40] Pablo Hernandez-Leew Bilal Kartal, and Matthew E Taylor. Agent modeling as auxiliary task for deep reinforcement learning. In Proceedings of the AAAI conference on artificial intelligence and interactive digital entertainment, volume 15, pages 31 31-37, 2019.
13

---

[41] Yutian Chen, Xingyou Song, Chansoo Lee, Zi Wang, Richard Zhang, David Dohan, Kazuya Kawakami, Greg Kochanski, Arnaud Doucet, Marc Aurelio Ranzato, Sagi Perel, and Nando de Freitas. Towards learning universal hyperparameter optimizers with transformers. In S. Koyejo, S. Mohamed, A. Agarwal, D. Belgrave, K. Cho, and Oh, editors, Advances in Neural Information Processing Systems, volume 35, pages 32053-32068. Curran Associates, neurips.eppppppp cc/ cf6601106feee772eeccc//111111222
Inc., 2022. URL https: URtpttttt: httpattppppppp..ceeennea.eeeee
ccppppppppcpppppeeeeeppepepppppe pappppappp____ www.ss/ppppee//222222 Cone--Cooeeeeece .dd
[42]
Juho Lee, Yoonho Lee, Jungtaek Kim, Adam R. Kosiorek, Seungjin Choi, and Yee Whye Teh. Set transformer: A framework for attention based permutation sermutation-ivvariinn invariant neural networks. In Kamalika Chaudhuri and Ruslan Salakhutdinov, editors, Proceedings of the 36th International Conference on Machine Learning, ICMI. 2019, 9-15 June 2019, Long Beach, California, USA, volume 97 of Proceedings of Machine Learning Research, pages 3744-3733 PMLR, 2019. URL http:://ppoootoo http://proceeddings mlr wwwpppressssieee vaapsss/v////eeeee v97//ee19/1lle 1ee19d.eeddee html
[43] Kate Rakelly, Aurick Zhou, Chelsea Finn, Sergey Levine, and Deirdre Quillen. Efficient off-policy meta-reinforreement learning via probabilistic context variables. In Kamalika Chaud- huri and Ruslan Salakhutdinov, editors, Proceedings of the 36th International Conference on Machine Learning. volume 97 of Proceedings of Machine Learning Research, pages 5331- 5340. PMLR, 09 15 Jun 2019. URL.attps://proceediin mmr wpwpss//9//raelll//ttattt html
[44] Sebastian Pineda-Arango, Hadi S. Jomaa, Martin Wistuba, and Josif Grabocka. HPO-B: A large- scale reproducible benchmark for black-box HPO based openml. In Joaquin Vanschoren and Sai-Kit Yeung, editors, Proceedings of the Neural Information Processing Systems Track on Datasets and Benchmarks I, NeurIPS Datasets and Benchmarks 2021, December 2021, virtual. 2021. URL https: http:://///aattttteteeeeee www.hbarkhhhbb....e.......eee.ee - new.reeipdnnn...uiip...... .ttm
[45] Ksenia Bestuzheva, Mathieu Besançon, Wei-Kun Chen, Antonia Chmiela, Tim Donkiewicz, Jasper van Doormmalen, Leon Eifler, Oliver Gaul, Gerald Gamrath, Ambros Gleixner, et al. The scip optimization suite 8.0. arXiv preprini
2021.
[46] Ambros M. Gleixner, Gregor Hendel, Gerald Gamrath, Tobias Achterberg, Michael Bastubbe, Timo Berthold, Philipp Christophel, Kati Jarck, Thorsten Koch, Jeff T. Linderoth, Marco Lübbecke, Hans D. Mittelmann, Derya B. Özyurt, Ted K. Ralphs, Domenico Salvagnin, and Yuji Shinano. MIPLIB 2017: data-driven compilation of the 6th iixeeeinteger programming library. Math. Program. Comput., 13(3):443 490, 2021. doi: 10.1007/s1 http:p://aal.orgg/...00////a/// 02005822222000004---99
URL httth::t///:ooii.o org 101111...00011110000
[47] Asif Khan, Alexander I. Cowen- Antoine Grosnit, Derrick-Goh-Xie Deik, Philippe A. Robert, Victor Greiff, Eva Smorodina, Puneet Rawat, Rahmad Akbar, Kamil Dreczkowski, Rasul Tutunov, Dany Bou-Amman Jun Wang, Amos Storkey, and Haitham Bou-Ammar. Toward real- world automated antibody design with combinatorial bayesian optimization. Cell Reports Meth- ods, page 100374, 2023. ISSN 2667-2375. doi: http:///oi.org/1000116// wttp::....ccmthh.cmcc.. 202.. 100374. URL BRRLhttps: https: www.ssscccccccccccccccmmmmm com/ http:///ww.aaaoaaaooooo/oooaaaaa
[48] Philippe A. Robert, Rahmad Akbar, Robert Frank, Milena Pavlović, Michael Widrich, Igor Snapkov, Andrei Slabodkin, Maria Chernigovskaya, Lonneke Scheffer, Eva Smorodina, Puneet Rawat, Brij Bhushan Mehta, Mai Ha Vu, Ingvild Froberg Mathisen, Aurél Prósz, Krzysztof Abram, Alex Olar, Enkelejda Miho, Dag Trygve Tryslew Haug. Fridtjof Lund-Johansen. Sepp Hochreiter, Ingrid Hobæk Haff, Günter Klambauer, Geir Kjetil Sandve, and Victor Greiff. Unconstrained generation of synthetic antibody-antigan structures to guide machine learning methodology for antibody specificity prediction. Nature Computational Science, 2412):845 865, 2022. ISSN 2662-66222222 8457. doi: 10. 0t0/://44388002200333333 com/ http//rticcess//m sarttccees/s33588 s43588 022 00372-4
URL https: h/wpss://www www.uraauree.e.
[49] Helen M Berman, John Westbrook, Zukang Feng, Gary Gilliland, Talapady N Bhat, Helge Weissig, Ilya Shindyalov, and Philip Bourne. The protein data bank, Nucleic acids research, 28( (8(1):235-242 2000.
14

---

[50] Animesh Basak Chowdhury, Benjamin Tan, Ramesh Karri, and Siddharth Garg. Openabe-da A large-scale dataset for machine learning guided integrated circuit synthesis. 2021.
[51] Robert Brayton and Alan Mishchenko. Abc: An academic - verification tool. In Tayssir Touili, Byron Cook, and Paul Jackson, editors, Computer Aided Verification, pages 24 40, Berlin, Heidelberg, 2010. Springer Berlin Heidelberg. ISBN 978-3-642- 1429-64...66...
[52] Chelsea Finn, Pieter Abbeel, and Sergey Levine. Model-aggostic meta-learning for fast adapta- tion of deep networks. In Doina Precup and Yee Whye Teh, editors, Proceedings of the 34th International Conference on Machine Learning, ICML 2017, Sydney, NSW, Australia, 6-11 August 2017, volume 70 of Proceedings of Machine Learning Research, pages 1126-1135. PMLR, 2017. URL
mlr. wal...press/v70// httt0/finn17a.htaa
[53] Yan Duan, Marcin Andrychowicz, Bradly C. Stadie, Jonathan Ho, Jonas Schneider, Ilya Sutskever, Pieter Abbeel, and Wojciech Zaremba. One-shot imitation learning. In Isabelle Guyon, Ulrike von Luxburg, Samy Bengio, Hanna M. Wallach, Rob Fergus, S. V. N. Vish- wanathan, and Roman Garnett, editors, Advances in Neural Information Processing Systems 30: Annual Conference on Neural Information Processing Systems 2017, December 4-9, 2017, Long Beach, CA, USA, pages I 1087-1098, 2017. URL http::tttt:::/ceeeett ieapeeeeddinnn.n.u.iiioe nwwrinurip.... ec/ paper/ / 2017 http//ahhhhhaaaaaa
html
[54] Jonas Rothfuss, Vincent Fortuin, Martin Josifoski, and Andreas Krause. PACOH: opyes-o meta-learning with sue-guaraattes. In Marina Meila and Tong Zhang, editors, Proceedings of the 38th International Conference Machine Learning, ICML 2021. 18-24 July 2021, Virtual Event, volume 139 of Proceedings of Machine Learning Research, pages 9116-9126 16. PMLR, 2021. TRRLhttp: htt.:ttt:/prrceeedingeee mlr
wwwhfoossha.a....o html
[55] Yutian Chen, Matthew W. Hoffman, Sergio Gómez Colmenarejo. Misha Denil, Timothy P. Lillicrap. Matt Botvinick, and Nando de Freitas. Learning to learn without gradient descent by gradient descent. In Doina Precup and Yee Whye Teh, editors, Proceedings of the 34th International Conference on Machine Learning, volume 70 of Proceedings of Machine Learning Research, pages 748-756. PMLR, 06-11 Aug 2017. URL https: pttpp://procceednng mlr press/v700 v h000/chen176/ hhmmll
[56) Vishnu TV, Pankaj Malhotra, Jyoti Narwariya, Lovekesh Vig, and Gautam Shroff. Meta- learning for black-box optimization. In Ulf Brefeld, Élisa Fromont, Andreas Hotho, Arno J. Knobbe, Marloes Maathuis, and Céline Robardet, editors, Machine Learning and Knowledge Discovery in Databases European Conference, ECML PKDD 2019, Würzburg, Germany, September 16-20, 2019, Proceedings, Part II, volume 11907 of Lecture Notes in Computer Science, pages 366-381. Springer. 2019. doi: 0007/998-3-000-4600 [/doi.org/11 1007 0 3- 080 46147-8_22 8_22
22. URL https
[57] Tomaharu Iwaa.. End-to-end learning of deep kernel acquisition functions for bayesian opti- mization, 2021.
[58] Huaxiu Yao, Long-Kai Huang, Linjun Zhang, Ying Wei, Li Tian, James Zou, Junzhou Huang, and Zhenhui Li. Improving generalization in meta-learning via task augmentation. In Marina Meila and Tong Zhang, editors, Proceedings of the 38th International Conference on Machine Learning, ICML 2021, 18-24 July 2021, Virtual Event, volume 139 of Proceedings of Machine Learning Research, pages 11887-1 11897. PMLR, 2021. URL http://prooceddings mlr pross/v139/yao21bb html
[59] V. L. Goncharov. Sur la distribution des cycles dans les permutations. Acad. Sei. URSS (N.S.), 35:267-269, 1944.
[60] Katharina Eggensperger, Philipp Müller, Neeratyoy Mallik, Matthias Feurer, René Sass, Aaron Klein, Noor H. Awad, Marius Lindauer, and Frank Hutter. Hpobench: A collection of re- producible multi-fidelity benchmark problems for HPO. In Joaquin Vanschoren and Sai-Kit Yeung, editors, Proceedings of the Neural Information Processing Systems Track Datasets and Benchmarks 1, NeurlPS Datasets and Benchmarks 2021, December 2021, virtual, 2021. /ttp://tt///etttteseaaaaaaaaaatn
setaatttssst---enhccaae www.searbnhhhhhheeeeeeee proceeddiooooeeeeeeooooo. nerrssneeriiie deurips... cc/ wtw.:pp....pppppeeeeeeeee /app///2////////////// 9xbbscelO00333385555ccccccca htt.:r/eeeebbeebttttttrt----oooo round2. html
15


Reviews:
[{"review_id": "l69IVgQKTr", "reviewer": "Reviewer_DqJs", "summary": "By combining a Transformer-based architecture and reinforcement learning objective, this paper proposes the first end-to-end framework for meta-Bayesian optimization.\nThe proposed framework is claimed to resolve the inefficiency in the existing two-stage approaches.", "questions": "- Is there any factor that necessitates the use of RL?", "limitations": "Yes", "rating": 6, "confidence": 2, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "- The meta-Bayesian optimization is an important topic that can be useful in various domains.\n- The presented motivation appears to be compelling, while the proposition of employing a Transformer-based architecture seems reasonable.\n- The experiments include practical problem settings.", "weaknesses": "- I am not sure if RL is really the optimal way to achieve an end-to-end meta-BO framework. I think this paper lacks a justification for why RL is a necessary component.\n- Although this paper claims to tackle the inefficiencies from two-stage approaches, introducing RL makes training much harder. The authors add an auxiliary task to solve this issue, but due to this, the proposed method does not seem to reduce the complexity and the objective discrepancy in the previous two-stage approaches.\n\nPlease note that I do not have much expertise in the relevant fields.\nI currently do not see any major flaw in this work, but I am also not fully convinced.\nComments from other reviewers and further discussion may largely change my score.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for their review and for acknowledging the strengths of our work and the relevance of our experiments. We hope that we can further motivate the use of RL and auxiliary loss in our response below.\n\n>Is there any factor that necessitates the use of RL?\n\nTo answer your first point, we would like to underline that the necessity of using RL comes from the fact that we are learning acquisition functions. As we do not have true labels for how good it is to select a point (it depnds on the other points seen so far and their values), we cannot rely on supervised learning and thus choose to rely on RL. The NP-EI baseline - trained only using a supervised loss and EI acquisition - shows that there is value in meta-learning the acquisition with RL as NAP outperforms it on every task.\n\n>Although this paper claims to tackle the inefficiencies from two-stage approaches, introducing RL makes training much harder. The authors add an auxiliary task to solve this issue, but due to this, the proposed method does not seem to reduce the complexity and the objective discrepancy in the previous two-stage approaches.\n\nEven in a two-stage approach we would need to train using RL for the second stage. Indeed, we can pretrain a neural process surrogate with supervised loss but to meta-learn a neural acquisition we must introduce RL. We add the auxiliary loss to help tackle the sparsity of the RL reward, which introduces a useful inductive bias. Even though this auxiliary loss is indeed not exactly optimising the same objective as in the downstream task, we argue that (see appendix C.1.) training end-to-end enables NAP to be updated simultaneously from both signals hence reducing the objective discrepancy. We use an end-to-end architecture so we can backpropagate information both from the acquisition (RL) and the auxiliary loss (supervised) through the whole network, which is not possible in a two-stage approach (see Figure 3 in appendix)."}, {"review_id": "PztLvmOtB8", "reviewer": "Reviewer_P5Ji", "summary": "In this work, they developed the first end-end transformer based architecture for meta Bayesian Optimization and demonstrated empirically state-of-the-art regret minimization on hyperparameter optimization, antibody and chip design problems. They propose using a novel transformer architecture based Neural Process to learn the acquisition function and train it E2E compared to prior work in meta-learning acquisition functions. They identified logarithmic reward sparsity patterns in RL and introduced an auxiliary loss maximizing log-likelyhood of making the correct predictions on their labeled source task datasets as an inductive bias to stabilize training.", "questions": "Did you tune the weighting of the auxiliary loss?\n\nHow were the hyperparameters chosen for the implementation if no hyperparameter search was conducted.", "limitations": "Discussed limitations limiting to 5000 BO steps due to their use of the transformer network.", "rating": 5, "confidence": 2, "soundness": 3, "presentation": 4, "contribution": 2, "strengths": "Demonstrates strong empirical results on their tasks and ablation experiments are included in the appendix demonstrating the importance of their inclusions.\n\nProposes novel combination of transformer network and E2E learning of the acquisition function for Bayesian optimization with a new formulation and inclusion of auxiliary loss.\n\nWell written and well positioned with existing work.", "weaknesses": "Somewhat limited novelty due to prior separately have utilized transformers for Bayesian optimization and other prior work having meta-learned acquisition functions.\n\nSomewhat limited experimental evidence since no hyperparameter search was conducted on the proposed method or baseline. Results were only included on four tasks.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for their positive comments and for underlining the clarity of the writing. We thank them for raising interesting points that we will try to explain below.\n\n> Somewhat limited novelty due to prior separately have utilized transformers for Bayesian optimization and other prior work having meta-learned acquisition functions.\n\nAs discussed in related work, prior work indeed exists where authors use RL to meta-learn neural acquisition functions [1,2]. However they rely on Gaussian Process models and by consequence their method is not end-to-end trainable.  Prior work also exists where authors use transformers but their goal is to meta-learn a surrogate model [3,4] which can in turn be used along with an off-the-shelf acquisition function. Again, these methods do not learn a single architecture that directly predicts acquisition function values from a history of observations. One of our contributions resides in the ability for NAP to be trained end-to-end for acquisition function value prediction. At test time, we show that this end-to-end training allows to achieve better performance, making our contributions significant.\n\n>Results were only included on four tasks.\n\nRegarding the concerns about experimental evidence, we believe that we show the performance of NAP on a panel of experiments, ranging from hyperparameter optimisation in low dimension (HPO-B) to high dimension (MIP) and from continuous spaces (HPO-B) to combinatorial spaces (Antibody, EDA) and even mixed-type spaces (MIP). This gives, in our opinion, a good overview of the practical applications where BO in general can be used and where meta-BO methods such as NAP can also be used. Please also note that the **HPO-B experiment is actually a collection of 6 tasks** (see appendix) and is an established benchmark for meta-learning [5,6].\n\n>How were the hyperparameters chosen for the implementation if no hyperparameter search was conducted.\n\nThank you for raising the topic of hyperparameters.\nWe did not perform any hyperparameter tuning on the values provided in the table in appendix an in particular we did not tune the weight between the two losses, which is simply set to 1.0 in all experiments. Moreover, all hyperparameters are fixed across all the experiments for a fair comparison. We set the hyperparameters based on the values from the codebase of MetaBO [2] and PFN [3] because we consider that their values have been optimised by their authors.\nFinally, with this single set of hyperparameters, we were able to obtain better results on all experiments, compared to SOTA-baselines (FSBO [6], OptFormer-EI [5], GP-EI). \n\n>Somewhat limited experimental evidence since no hyperparameter search was conducted on the proposed method or baseline.\n\nWe do not conduct a full hyperparameter search for the reasons mentioned above, however we conduct an ablation study that can be seen as an analysis on the hyperparameter $\\lambda$ giving the relative weights of the two losses in the total loss (see appendix C.1.). If the weight of the auxiliary loss is zero, we have NAP-RL and if the loss of the RL loss is zero, we have NP-EI. We can see that simply with equal weighting ($\\lambda = 1$), we achieve better results.\n\n----------\n\n[1] Bing-Jing Hsieh, Ping-Chun Hsieh, and Xi Liu. Reinforced few-shot acquisition function learning for bayesian optimization. In M. Ranzato, A. Beygelzimer, Y. Dauphin, P.S. Liang, and J. Wortman Vaughan, editors, Advances in Neural Information Processing Systems, volume 34, pages 7718–7731. Curran Associates, Inc., 2021.\n\n[2] Michael Volpp, Lukas P. Fröhlich, Kirsten Fischer, Andreas Doerr, Stefan Falkner, Frank Hutter, and Christian Daniel. Meta-learning acquisition functions for transfer learning in bayesian optimization. In 8th International Conference on Learning Representations, ICLR 2020, Addis Ababa, Ethiopia, April 26-30, 2020.\n\n[3] Samuel Müller, Noah Hollmann, Sebastian Pineda-Arango, Josif Grabocka, and Frank Hutter. Transformers can do bayesian inference. In The Tenth International Conference on Learning Representations, ICLR 2022, Virtual Event, April 25-29, 2022.\n\n[4] Tung Nguyen and Aditya Grover. Transformer neural processes: Uncertainty-aware meta learning via sequence modeling. In Kamalika Chaudhuri, Stefanie Jegelka, Le Song, Csaba Szepesvári, Gang Niu, and Sivan Sabato, editors, International Conference on Machine Learning, ICML 2022, 17-23 July 2022, Baltimore, Maryland, USA, volume 162 of Proceedings of Machine Learning Research, pages 16569–16594. PMLR, 2022.\n\n[5] Yutian Chen, Xingyou Song, Chansoo Lee, Zi Wang, Richard Zhang, David Dohan, Kazuya Kawakami, Greg Kochanski, Arnaud Doucet, Marc Aurelio Ranzato, Sagi Perel, and Nando de Freitas. Towards learning universal hyperparameter optimizers with transformers. In\nS. Koyejo, S. Mohamed, A. Agarwal, D. Belgrave, K. Cho, and A. Oh, editors, Advances in Neural Information Processing Systems, volume 35, pages 32053–32068. Curran Associates, Inc., 2022.\n\n[6] Martin Wistuba and Josif Grabocka. Few-shot bayesian optimization with deep kernel surrogates. In 9th International Conference on Learning Representations, ICLR 2021, Virtual Event, Austria, May 3-7, 2021."}, {"review_id": "t16HpcBNAx", "reviewer": "Reviewer_K11E", "summary": "This work proposes an approach to meta-Bayesian optimization (BO) that features a single transformer neural process architecture for both the meta-surrogate function and meta-acquisition function, which are respectively trained via standard neural process maximum likelihood and model-free meta-reinforcement learning (in the style of RL^2) frameworks. The authors demonstrate the empirical benefits of such an approach over prior meta-BO methods on four suites of black-box optimization problems.", "questions": "- How is your framing of the acquisition function as a policy with order-invariant processing of the current optimization trace related to ideas in context-based/prototype-based meta-reinforcement learning [A]? It seems like the former is a specific instance of the latter where the state space for each MDP is singleton. \n\n[A] Rakelly & Zhou et al., Efficient Off-Policy Meta-Reinforcement Learning via Probabilistic Context Variables, ICML 2019.", "limitations": "Yes.", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "### Originality\nMy background is in meta-learning and reinforcement learning, not Bayesian optimization. It appears that the proposed approach is novel for meta-BO in that it makes minimal modelling choices, opting for a completely black-box neural process for the meta-surrogate function as opposed to e.g. a Gaussian process, and a completely black-box acquisition function as opposed to e.g. expected improvement (EI).\n\n### Quality\nThe empirical evaluation seems reasonably thorough (though again, I am not an expert in BO), with experiments based on multiple data sources and a healthy pool of competing methods. I also appreciated the described engineering effort in adapting previous methods to handle datasets for which they were not originally designed for.  \n\n### Clarity\nI'm not sure that Lemma 3.1 adds much substance in favor of the paper's design decisions. Instead, perhaps we can more loosely but intuitively argue that typical acquisition functions such as EI depend on the surrogate function's predictions of the underlying objective function (as it well should, to avoid no free lunch), but this dependency is dropped with a completely end-to-end black box acquisition function. The neural process meta-surrogate training can then be justified as softly reintroducing this dependency a la multi-task learning (multi in the sense of surrogate function modeling and acquisition function policy learning, not multiple BO tasks) via a shared architecture. \n\n### Significance\nI imagine the demonstrated empirical benefit of an end-to-end solution in a domain where handcrafted solution components have been standard will have considerable significance.", "weaknesses": "- A minor point -- it is unusual to see one maximizing a \"loss\" (Eq. 3 and Alg. 1).", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for their comment, for the remark on extending baselines to search spaces they were not designed for originally, and for raising an interesting point regarding meta-RL, We will answer point by point below.\n\n>I'm not sure that Lemma 3.1 adds much substance in favor of the paper's design decisions. Instead, perhaps we can more loosely but intuitively argue that typical acquisition functions such as EI depend on the surrogate function's predictions of the underlying objective function (as it well should, to avoid no free lunch), but this dependency is dropped with a completely end-to-end black box acquisition function. The neural process meta-surrogate training can then be justified as softly reintroducing this dependency a la multi-task learning (multi in the sense of surrogate function modeling and acquisition function policy learning, not multiple BO tasks) via a shared architecture.\n\nRegarding clarity, the lemma 3.1 is a formal explanation of the sparsity of the reward used in RL, specifically linked to the use for BO.\nThis issue is well-known in many RL problems but we wanted to make it clear that it is also the case in a meta-BO setting. This in itself, does justify the use of an auxiliary loss, but we also agree with the intuition you mentioned for the choice of the chosen auxiliary loss.\n\n>How is your framing of the acquisition function as a policy with order-invariant processing of the current optimization trace related to ideas in context-based/prototype-based meta-reinforcement learning [A]? It seems like the former is a specific instance of the latter where the state space for each MDP is singleton.\n\nIf we want to match the history order-invariance of our policy with the tuple order-invariance of the mentioned paper, the tuple $t_0$, ..., $t_{n-1}$ without the reward should be defined like $t_0 = (s=\\{\\}, a=x_0, s'=f(x_0))$ and $t_i = (s=x_{i-1}, a=x_i, s'=f(x_i))$.\nWe believe this is what you meant by a singleton state-space.\nIf it is doable in practice, it does not form a standard MDP because the reward function cannot be defined based on a single tuple.\n\nInstead, a better instantiation would define the state in the tuple as a history of collected points $(x_i, f(x_i))$, however, in such case, the history order-invariance is not necessarily preserved through tuple order-invariance.\n\nThis is an interesting point however, and we are going to include the mentioned paper in the related work section."}, {"review_id": "R0fE4HTN16", "reviewer": "Reviewer_6XK8", "summary": "The authors present a hyperparameter optimization method that is based on a transformer. In contrast to other recent work on HPO with transformers, this method does not rely on an acquisition function but instead trains an end-to-end model that outputs the acquisition scores directly.", "questions": "This is a complex method that probably introduce several hyperparameters itself. How do you set them? How robust are they? Do you think there is a chance that this method can be directly applied to a different task or will it require significant work to get it running?\n\nWhy is NP-EI missing for HPO-B?", "limitations": "The authors do not address the limitations that come with the complexity of this method, i.e., additional hyperparameters.", "rating": 5, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "**Diversity of benchmark tasks**\n\nExperiments on 4 different types of tasks, comparison against state-of-the-art methods. Demonstrate some improvements over baselines.\n\n**Clear Motivation**\n\nUse of transformers have shown promising results. End-to-end learning is always a good idea. The authors combine these previously existing ideas.", "weaknesses": "**Complexity of the Method**\n\nThe authors combine transformers, reinforcement learning and neural processes. There are most likely a lot of knobs and using this method effectively might be very difficult. Meta-BO is a prime example of a method that adds quite a bit of complexity and it turned out that it did not generalize to any other setup than considered by the authors.\n\n**Novelty**\n\nThis is neither the first paper to use transfer learning, transformers, RL, or end-to-end learning for HPO. This is a paper that combines all of it to a new method.\nA paper not discussed in this work: Bing-Jing Hsieh, Ping-Chun Hsieh, Xi Liu: Reinforced Few-Shot Acquisition Function Learning for Bayesian Optimization. NeurIPS 2021: 7718-7731\nIt also trains an end-to-end model and the authors demonstrated some improvements over FSBO.\n\n**Some Overclaiming of Contributions**\n\nIn multiple locations, the authors claim to present the \" first end-to-end training protocol for meta-BO\". This is not the case, see reference above and MetaBO.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for their time and for stressing the importance of end-to-end training. We try to answer the questions regarding complexity, novelty and hyperparameter tuning below.\n\n>The authors combine transformers, reinforcement learning and neural processes. There are most likely a lot of knobs and using this method effectively might be very difficult. Meta-BO is a prime example of a method that adds quite a bit of complexity and it turned out that it did not generalize to any other setup than considered by the authors.\n\n>This is a complex method that probably introduce several hyperparameters itself. How do you set them? How robust are they? Do you think there is a chance that this method can be directly applied to a different task or will it require significant work to get it running?\n\nAs this comment and question relate to similar topics, we answer them together. Our method involves a good amount of pretraining on source task data but this is not unlike other meta-BO methods. In our experiments we give the same budget for pretraining to NAP and MetaBO. The hyperparameters used are all provided in Table 2 in appendix and they remain the same across our experiments.  We set their values based on the ones found in the repository of the MetaBO [1] baseline and did not change them further. We also fix an equal weight between the RL loss and the supervised auxiliary loss. To preserve fair comparison between all methods and because we assume the hyperparameters set by the authors of each baseline are already optimised, we do not tune ours. In that regard, we believe that this makes NAP more portable to new tasks as the hyperparameters have not been tuned for each specific task.\n\nWe acknowledge the difficulty of applying MetaBO in other domains. However, we believe that it comes from hypotheses made by the authors (that the parameters of the GPs are fixed during training and testing).\nBy tackling 9 heterogeneous tasks (6 in HPO-B), we show that NAP is not suffering from the same problem and achieve good performance across tasks.\n\n>Novelty: This is neither the first paper to use transfer learning, transformers, RL, or end-to-end learning for HPO. This is a paper that combines all of it to a new method. A paper not discussed in this work: Bing-Jing Hsieh, Ping-Chun Hsieh, Xi Liu: Reinforced Few-Shot Acquisition Function Learning for Bayesian Optimization. NeurIPS 2021: 7718-7731 It also trains an end-to-end model and the authors demonstrated some improvements over FSBO.\n\n>Some Overclaiming of Contributions: In multiple locations, the authors claim to present the \" first end-to-end training protocol for meta-BO\". This is not the case, see reference above and MetaBO.\n\nThese two remarks are also linked so we answer them together. We refer to the work by Hsieh et al. on a few occasions in our paper (ref [14] in bib, [2] in this comment) but thank you for giving us a chance to expand on the differences between their approach and ours. We will also add these arguments in the Related Work section.\n\nFirst, we would like to outline that the setting in [2] is slightly different from our setting in that it aims to do *adaptation*, i.e. few-shot learning; which means that the authors meta-learn a neural acquisition on some source task data as well as data generated from a prior, but then allow their model to have access to test task observations to fine tune. Moreover, as PACOH [3], they rely on ensemble-method with Stein Variational Gradient Descent (SVGD). We consider these prior works as orthogonal directions as they can be combined with NAP: we could also use SVGD with multiple transformers or to finetune NAP at test time.\n\nSecond, we can argue that FSAF is not end-to-end differentiable because it still relies on Gaussian Process surrogates (also the case for the MetaBO [1] baseline) and hence is classified as a two stage approach. In their paper in section 3.1 the authors rely on the posterior mean and variance of a GP for state-action representation of each point.\n\nWhat we are claiming is to be the first **end-to-end differentiable** method for meta-BO that specifically generalises neural processes to learn acquisition functions directly from the raw observations. We will make that distinction clearer in the paper, in abstract and introduction when talking about contributions.\n\n>NP-EI missing for HPO-B\n\nThank you for pointing that out, we added NP-EI as a baseline in the HPO-B experiment, please see figures of the uploaded rebuttal PDF. \n\n----------\n\n[1] Michael Volpp, Lukas P. Fröhlich, Kirsten Fischer, Andreas Doerr, Stefan Falkner, Frank Hutter, and Christian Daniel. Meta-learning acquisition functions for transfer learning in bayesian optimization. In 8th International Conference on Learning Representations, ICLR 2020, Addis Ababa, Ethiopia, April 26-30, 2020.\n\n[2] Bing-Jing Hsieh, Ping-Chun Hsieh, and Xi Liu. Reinforced few-shot acquisition function learning for bayesian optimization. In M. Ranzato, A. Beygelzimer, Y. Dauphin, P.S. Liang, and J. Wortman Vaughan, editors, Advances in Neural Information Processing Systems, volume 34, pages 7718–7731. Curran Associates, Inc., 2021.\n\n[3] Jonas Rothfuss, Vincent Fortuin, Martin Josifoski, and Andreas Krause. PACOH: bayes-optimal meta-learning with pac-guarantees. In Marina Meila and Tong Zhang, editors, Proceedings of the 38th International Conference on Machine Learning, ICML 2021, 18-24 July 2021, Virtual Event, volume 139 of Proceedings of Machine Learning Research, pages 9116–9126. PMLR, 2021.\n"}, {"review_id": "3ARQiOdvRF", "reviewer": "Reviewer_Mquj", "summary": "This paper proposed an end-to-end transformer-based framework for meta-Bayesian optimization (meta-BO). It formulated the meta-BO as a RL problem, defining a MDP in which a policy can be trained to solve the meta-BO problem. To help the training of RL algorithm, this paper also proposed an inductive bias via an auxiliary loss. The experiments had demonstrated the effectiveness of the proposed method.", "questions": "Please refer to weakness.", "limitations": "Yes.", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. This paper deals with a good problem, which has a lot of applications in real world. The formulation of MDP is also reasonable.\n2. This paper is well written, and also easy to follow.\n3. The proposed inductive bias via an auxiliary loss Section 3.3. is insightful. \n", "weaknesses": "1. The experiments is insufficient. In the current experiments, the author only provided the evaluation results of the proposed method NAP and other baselines on several benchmarks. However, it would be better to do some ablation studies to further analyze NAP. For example, it should be important to investigate the effect of the auxiliary loss Eq(3).\n2. The applications are limited. RL is known to be sample inefficient, especially when the state/action spaces are of high dimensionality. Furthermore, NAP utilized a transformer to extract historic information, which is also relatively hard to train. Therefore, NAP may be implicitly restricted into situations when x is of low dimension, limiting its applications.\n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for their positive comments on the applicability of our work and we try to answer their concerns below.\n\n>The experiments is insufficient. In the current experiments, the author only provided the evaluation results of the proposed method NAP and other baselines on several benchmarks. However, it would be better to do some ablation studies to further analyze NAP. For example, it should be important to investigate the effect of the auxiliary loss Eq(3).\n\nWe would like to highlight that we did an ablation study in Section C.1 of the Appendix.\nIn Table 3 we show for each version of NAP, the components that are present or absent. Figure 4 shows the regret results on another experiment, an HPO experiment on XGBoost hyperparameters. We show that NAP-RL, trained only through the RL loss, performs worse than NAP, emphasising the importance of the auxiliary loss for downstream performance.\n\n>The applications are limited. RL is known to be sample inefficient, especially when the state/action spaces are of high dimensionality. Furthermore, NAP utilized a transformer to extract historic information, which is also relatively hard to train. Therefore, NAP may be implicitly restricted into situations when x is of low dimension, limiting its applications.\n\nNote that we pre-train NAP with RL, as such it is **offline** and therefore does not impact the **online** sample efficiency of NAP at test time as we only use online observations to select the next query.\nIn BO in general, the metric that matters most is online sample efficiency because we assume that we are in a setting where the black-box objective is costly and evaluating it is the main bottleneck.\n\nFurthermore, we show that even with a dataset of rather limited size, we can pre-train NAP e.g. on the MIP experiment where the search space is of high dimension (135), and are still able to learn useful information transferable to new tasks that leads to best performance. We do acknowledge some limitations, for instance that NAP is restricted to be trained and tested on a search space of similar dimension (discussed in Limitations section) and we would like to extend it to be dimension-agnostic in a future work. Finally, we acknowledge the difficulty of training a transformer with RL signals only (underlined in the ablation), hence our use of the auxiliary loss."}, {"review_id": "JYBeUBTfJP", "reviewer": "Reviewer_dCjX", "summary": "The authors propose Neural Acquisition Process, a novel method for Bayesian optimization by meta-learning a function that jointly performs the surrogate and acquisition steps. The function, i.e. a transformer, is able to work as a policy that predicts the action **a** given a state **s**, where the action is the hyperparameter to observe. The state comprises the history of the observed samples. Also, the transformer can predict probabilistic the performance of a hyperparameter. They use reinforcement learning to train the policy and, additionally, they use negative log-likelihood of the predicted performance for meta-tasks as an auxiliary loss. The authors test the method on 4 different benchmarks and delineate its difference from previous work, demonstrating the relevance of their approach.", "questions": "* Why is there no confidence bar for the MIP dataset?", "limitations": "No important limitation is detected.", "rating": 7, "confidence": 4, "soundness": 4, "presentation": 4, "contribution": 3, "strengths": "* They present a novel and effective method for pre-training a transformer to perform Bayesian optimization. The method is well-founded and all the parts make sense from an engineering perspective. For instance, the use of auxiliary losses and reinforcement learning appears as a very important and interesting approach.\n* They establish a difference from prior work in terms of some parameters such as history order invariance, F values, number of tokens, etc.\n* The authors provide a strong experimental protocol, comparing to relevant baselines in a broad set of datasets and benchmarks, which makes the empirical results very strong.\n", "weaknesses": "* The method might be overkill for small search spaces and nonexpensive functions. However, many black-box functions nowadays, such as deep neural networks, are expensive to evaluate, thus this approach might be highly relevant.\n* The authors do not present a comparison against time. How does the method perform in terms of time?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for their time and their remarks as well as underlining the soundness of our experimental protocol. We try to reply to their concerns below.\n\n>The method might be overkill for small search spaces and nonexpensive functions. However, many black-box functions nowadays, such as deep neural networks, are expensive to evaluate, thus this approach might be highly relevant.\n\nIndeed, for small search spaces and non-expensive black-box functions, our method might be overkill, and so would regular BO to a certain extent because if querying the objective is cheap, sample efficiency becomes less important than time efficiency. If the objective becomes very expensive to evaluate, then black-box solvers become useful. We keep our experiments limited to this case, in which sample efficiency is most important as the main bottleneck in terms of cost and time is querying the objective, and this cost outweighs the cost of any model fitting or pretraining done offline or online. We conduct experiments in search spaces that can be considered small (e,g, in HPO-B) but also in larger ones, e.g. Antibody design and EDA in combinatorial space and MIP in high-dimension and mixed space (up to 135 dimensions).\n\n>The authors do not present a comparison against time. How does the method perform in terms of time?\n\nWe have to distinguish the time during testing and the time during pretraining. Methods that make use of GP surrogates are bound to be slower in terms of time compared to NAP and NP-EI that only use forward passes in the network to decide what is the next query point. At each step of BO at test time, for example FSBO [1] or MetaBO [2] have to fit a GP model. However, pretraining time is more important for NAP, and MetaBO compared with NP-EI as the latter doesn't use RL which is computationally more costly during offline pretraining. They also take more time to pretrain than FSBO that meta-learns a feature extractor for a GP (deep kernel). The GP-EI baseline has no pretraining as it is a classical BO method with no amount of meta-learning. Finally, we mention a more detailed comparison with Optformer-EI [3] in appendix B.2. showing that NAP is faster to train, evaluate and uses far less memory.\n\nIt is natural that there is a tradeoff between pretraining cost and regret performance at test time, but as the most important metric in our setting is sample efficiency at test time, we did not show the results versus time.\nDuring testing, we observed a rough estimate of 10x speedup compared to GPs-based methods that requires refitting. We will add those numbers to the appendix for reference. \n\n>Question: Why is there no confidence bar for the MIP dataset?\n\nThe error bars are visible if you open the PDF file in Acrobat Reader. We also found that they are visible when opening it with Firefox. But indeed, we are not sure why they are sadly not visible when opening the PDF with Chrome. We are working on finding where this issue comes from.\n\n----------\n\n[1] Martin Wistuba and Josif Grabocka. Few-shot bayesian optimization with deep kernel surrogates. In 9th International Conference on Learning Representations, ICLR 2021, Virtual Event, Austria, May 3-7, 2021.\n\n[2] Michael Volpp, Lukas P. Fröhlich, Kirsten Fischer, Andreas Doerr, Stefan Falkner, Frank Hutter, and Christian Daniel. Meta-learning acquisition functions for transfer learning in bayesian optimization. In 8th International Conference on Learning Representations, ICLR 2020, Addis Ababa, Ethiopia, April 26-30, 2020.\n\n[3] Yutian Chen, Xingyou Song, Chansoo Lee, Zi Wang, Richard Zhang, David Dohan, Kazuya Kawakami, Greg Kochanski, Arnaud Doucet, Marc Aurelio Ranzato, Sagi Perel, and Nando de Freitas. Towards learning universal hyperparameter optimizers with transformers. In S. Koyejo, S. Mohamed, A. Agarwal, D. Belgrave, K. Cho, and A. Oh, editors, Advances in Neural Information Processing Systems, volume 35, pages 32053–32068. Curran Associates, Inc., 2022."}]

Author rebuttal:
We thank all reviewers for their time reading our paper and writing reviews. We are very appreciative of all positive comments made about our work, on clarity, applicability of the method, quality of experiments and baselines implementation. We further thank all reviewers for raising interesting points and questions about motivation, novelty and hyperparameters. We address each of the reviews separately in their own threads but we summarize here the main remarks.

### Novelty 

We discuss prior works that make use of transformers for BO as well as prior work using RL to learn acquisition functions. However, to the best of our knowledge, we are the first to propose an **end-to-end differentiable** method that specifically learns acquisition values with RL and is based on a transformer neural process architecture. We will clarify this claim in the main paper. We also discuss orthogonal approaches that learn a better prior from the source tasks data for pretraining, as well as using RL for learning acquisitions in order to do better adaptation (fine tuning) at test time. We will also extend our discussion of these research directions and stress their differences with ours.

### Hyperparameters

We will make it clearer in the main paper that the hyperparameters of NAP were not tuned. This is only fair as we did not tune them for the baselines either, considering that the ones set by the authors in their respective code repositories were already optimised. Furthermore, we used the same  values for all of the experiments, avoiding task-specific hyperparameter tuning that could limit generalisation and applicability of our method. In particular we used an equal weight between the RL loss and the auxiliary loss in all experiments ($\lambda = 1$). An ablation for that particular parameter has been carried out and results are shown in Appendix C.1.

### Extra experiment

We added the NP-EI baseline to the HPO-B experiment as noted by one reviewer, the update plots are in the PDF document added in this rebuttal.

Meta-review: This paper proposes a meta-learning method of Bayesian optimization. With the proposed method, transformer-based neural processes are used to model acquisition functions, and they are trained in an end-to-end fashion based on reinforcement learning. The effectiveness of the proposed method is demonstrated in their extensive experiments using different types of tasks with proper baselines. Meta-Bayesian optimization is an important task. The design of the proposed method is reasonable and effective. The auxiliary loss used in the proposed method is interesting. This paper is well written. It would be good to add more discussion on related work on meta-learning of Bayesian optimization, e.g., End-to-End Learning of Deep Kernel Acquisition Functions for Bayesian Optimization, arXiv, 2111.00639, 2021, to clarify the contribution of this paper. The additional experimental results submitted in the author response strengthen the paper.
Decision: Accept (poster)