Title: End-to-End Meta-Bayesian Optimisation with Transformer Neural Processes
Venue: NeurIPS 2023
Accepted: yes
Keywords: meta-learning

bayesian optimisation

neural process

transformer

end-to-end

reinforcement learning
Abstract: Meta-Bayesian optimisation (meta-BO) aims to improve the sample efficiency of Bayesian optimisation by leveraging data from related tasks. While previous methods successfully meta-learn either a surrogate model or an acquisition function independently, joint training of both components remains an open challenge. This paper proposes the first end-to-end differentiable meta-BO framework that generalises neural processes to learn acquisition functions via transformer architectures. We enable this end-to-end framework with reinforcement learning (RL) to tackle the lack of labelled acquisition data. Early on, we notice that training transformer-based neural processes from scratch with RL is challenging due to insufficient supervision, especially when rewards are sparse. We formalise this claim with a combinatorial analysis showing that the widely used notion of regret as a reward signal exhibits a logarithmic sparsity pattern in trajectory lengths. To tackle this problem, we augment the RL objective with an auxiliary task that guides part of the architecture to learn a valid probabilistic model as an inductive bias. We demonstrate that our method achieves state-of-the-art regret results against various baselines in experiments on standard hyperparameter optimisation tasks and also outperforms others in the real-world problems of mixed-integer programming tuning, antibody design, and logic synthesis for electronic design automation.

End-to-End Meta-Bayesian Optimisation with Transformer Neural Processes

Abstract

Meta-Bayesian optimisation (meta-BO) aims to improve the sample efficiency of Bayesian optimisation by leveraging data from related tasks. While previous methods successfully meta-learn either a surrogate model or an acquisition function independently, joint training of both components remains an open challenge. This paper proposes the first end-to-end differentiable meta-BO framework that generalises neural processes to learn acquisition functions via transformer architectures. We enable this end-to-end framework with reinforcement learning (RL) to tackle the lack of labelled acquisition data. Early on, we notice that training transformer-based neural processes from scratch with RL is challenging due to insufficient supervision, especially when rewards are sparse. We formalise this claim with a combinatorial analysis showing that the widely used notion of regret as a reward signal exhibits a logarithmic sparsity pattern in trajectory lengths. To tackle this problem, we augment the RL objective with an auxiliary task that guides part of the architecture to learn a valid probabilistic model as an inductive bias. We demonstrate that our method achieves state-of-the-art regret results against various baselines in experiments on standard hyperparameter optimisation tasks and also outperforms others in the real-world problems of mixed-integer programming tuning, antibody design, and logic synthesis for electronic design automation.

1 Introduction

Bayesian optimisation (BO) techniques are sample-efficient sequential model-based solvers optimising expensive-to-evaluate black-box objectives. Traditionally, BO methods operate in isolation focusing on one task at a time, a setting that led to numerous successful applications, including but not limited to hyperparameter tuning [1], drug and chip design [2], and robotics [3]. Although successful, focusing on only one objective in isolation may increase sample complexities on new tasks since standard BO algorithms work tabula-rasa as they start optimising from scratch, ignoring previously observed black-box functions.

To improve sample efficiency on new target tasks, meta-BO makes use of data collected on related source tasks [4] and attempts to transfer knowledge in between. Those methods are primarily composed of two parts: 1) a (meta) surrogate model predicting the function to optimise and 2) a (meta) acquisition function (AF) estimating how good a queried location is. Regarding surrogate

Matthieu Zimmer ∗

Haitham Bou Ammar

modelling, prior work relies on Gaussian processes (GPs) to perform transfer across tasks due to their data efficiency [5-9], or focuses on deep neural networks [10-12] gaining from their representation flexibility. As for acquisition functions, previous works assumed a fixed GP and trained a neural network to perform transfer [13, 14].

Although successful, those methods rely on the assumption that the two steps described above can be handled independently, potentially missing benefits from considering both steps together. Therefore, this paper advocates for an end-to-end training protocol for meta-BO where we train a model predicting AF values directly from observed data, without relying on a GP surrogate.

As shown in [10, 12, 15, 16], Neural processes (NP) are a good candidate for meta-learning due to their structural properties, thus we propose to use a new transformer-based NP to model the acquisition function directly [17].

Nevertheless, the lack of labelled AF data prohibits a supervised learning protocol. Here, we rely on reward functions to assess the goodness of AFs and formulate a new reinforcement learning (RL) problem. Our RL formulation attempts to learn an optimal policy that selects new evaluation probes by minimising per-task cumulative regrets.

Early on, we faced very unstable training with minimal learning due to the combination of RL with transformers [18]. Furthermore, we notice that this problem amplifies when the reward function is sparse, which is the case in our setting, as we formally show in Section 3.2. To mitigate this difficulty, we introduce an inductive bias in our transformer architecture via an auxiliary loss [19] that guides a part of the network to learn a valid probabilistic model, effectively specialising as a neural process with gradient updates. Having developed our framework, we demonstrate state-of-the-art regret performance in hyperparameter optimisation of real-world problems and combinatorial sequence optimisation for antibody and chip design (see Section 4). Those solid empirical results show the value of end-to-end training in meta-BO to further improve sample complexity.

Contributions We summarise our contributions as follows: i) developing the first end-to-end transformer-based architecture for meta-BO predicting acquisition function values, ii) identifying logarithmic reward sparsity patterns hindering end-to-end learning with RL, iii) introducing NP-based inductive biases for successful model updates, and iv) demonstrating new state-of-the-art regret results on a wide range of traditional (hyperparameter tuning) and non-traditional (chip and antibody design) real-world benchmarks.

2 Background

2.1 Bayesian Optimisation

In BO, we sequentially maximise an expensive-to-evaluate black-box function f ( x ) for variables x ∈ X . BO techniques operate in two steps. First, based on previously collected evaluation data, we fit a probabilistic surrogate model to emulate f ( x ) allowing us to make probabilistic predictions of the function's behaviour on unobserved input points. Given the first step's probabilistic predictions, the second step in BO optimises an AF that trades off exploration and exploitation to find a new input query, x new ∈ X , for evaluation. Both the model and acquisition choices play a critical role in the success of BO. Many works adopt GPs as surrogate models due to their sample efficiency and practical uncertainty estimation [20]. Moreover, on the acquisition side, the common practice is to optimise one from a set of widely adopted AFs such as Expected Improvement (EI) [21].

2.2 Transfer in Bayesian Optimisation

Transfer techniques in BO [4, 13] aim to improve the optimisation of a newly observed target blackbox function by reusing knowledge from past experiences gathered in related source domains, i.e. solve max x ∈X f Target ( x ) by leveraging information from K source black-boxes f 1 ( x ) , . . . , f K ( x ) . We assume this information is available to our agent via D 1 , . . . , D K such that each dataset D k = {⟨ x ( k ) i , y ( k ) i ⟩} n ( k ) i =1 consists of n ( k ) (noisy) evaluations of f k ( x ) for all k ∈ [1 : K ] . Following the meta-learning literature [4, 5, 13], we impose no additional assumptions on the process that collects D 1 , . . . , D K , allowing for a diverse set of source optimisation algorithms, including but not limited to BO, genetic and evolutionary algorithms [22], and sampling-based strategies [23]. Many algorithms that use source data to improve performance on target domains exist. We categorise those based

on the part they customise within the BO pipeline, i.e., by registering if they affect initial points [24, 25], search spaces [26], surrogate models [5] or AFs [13, 14]. Since our work devises end-to-end meta-BO pipelines, we now elaborate on critical surrogate models needed for the rest of the paper and leave a detailed presentation of related work to Section 5.

Although GPs [20] are a prominent tool for single-task BO, high computational complexity [27], among other drawbacks (e.g. smoothness assumptions or limited scalability in terms of dimensions), can limit their application to transfer scenarios. Of course, one can generalise GPs to support transfer using multi-task kernels [28] or GP ensembles [6], for example. However, recent trends demonstrated superior performance to such extensions when using deep networks (e.g., neural processes) to meta-learn probabilistic surrogates [5, 13].

Neural Processes A popular method for effective meta-learning consists of directly inputting context points (i.e., available data to adjust to a target task) for a model to adapt to unseen tasks [15]. We can accomplish such a goal by relying on neural processes (NPs), a class of models that combine the flexibility of deep neural networks with the properties of stochastic processes [10]. Given an observation set of input-output pairs D obs and a set of n pred locations x ( pred ) at which we desire to make predictions, an NP parameterised by θ outputs a distribution p θ ( ·| x (pred) , D obs ) that approximates the true posterior over the labels y ( pred ) . Among the different parameterisations of p θ ( · ) , e.g., in [29-31], transformer architectures [17] recently emerged as compelling choices of NPs [11, 12]. In this work, we also adopt a transformer architecture inspired by this architecture's broad software support and state-of-the-art supervised learning results as reported in [11].

2.3 Reinforcement Learning

In Section 3, we attempt to learn an end-to-end model that predicts acquisition values. Of course, we cannot fit the parameters of such a model with supervised learning due to the lack of labelled acquisition data. RL [32] is a viable alternative for learning from delayed and non-differentiable reward signals in those cases. In RL, we formalise problems as Markov decision processes (MDP): M = ⟨S , A , P , R , γ ⟩ , where S and A denote the state and action spaces, P : S×A×S → [0 , 1] the state transition model, R the reward function dictating the optimisation goal, and γ ∈ [0 , 1) a discount factor specifying the degree to which rewards are discounted over time. A policy π : S × A → [0 , 1] is an action-selection rule that is defined as a probability distribution over state-action pairs, where π ( a t | s t ) represents the probability of selecting action a t in state s t . An RL agent aims to find an optimal policy π ⋆ that maximises (discounted) expected returns. For determining π ⋆ , we use the state-of-the-art proximal policy optimisation (PPO) algorithm [33].

3 End-to-End Meta-Bayesian Optimisation

While transfer techniques in BO saw varying degrees of success in many applications, current approaches lack end-to-end differentiability, where model learning and acquisition discovery arise as two separate steps. Specifically, the surrogate model gradients hardly affect the acquisition network, and the acquisition's network gradients fail to back-propagate to the surrogate's updates.

Enabling end-to-end transfer frameworks in which we learn the surrogate and acquisition jointly hold the promise for more scalable and easier-to-deploy algorithms that are more robust to input data or task changes. Following such a framework, we also expect more accurate predictions that can lead to better regret results (see Section 4) since we optimise the entire transfer pipeline, including the intermediate probabilistic model and AF representations. Additionally, end-to-end training techniques allow us to mitigate the need for domain-specific expertise and permit stable implementations that benefit fully from GPU and computing hardware.

The most straightforward way to enable end-to-end training in Bayesian optimisation is to introduce a deep network that acquires search variables and historical evaluations of black-box functions as inputs and outputs acquisition values after a set of nonlinear transformations. Of course, it is challenging to fit the weights of such a network due, in part, to a lack of labelled acquisition data where search variables and history of evaluations are inputs and acquisition values are labels.

3.1 Reinforcement Learning for End-to-End Training

Our approach utilises RL to fit the network's parameters θ from minimal supervision, circumventing the need for labelled acquisition data. To formalise the RL problem, we introduce an MDP where:

State : s t = [ H t , t, T ] (history, BO time-step & budget) Action : a t = x t (choice of new probe), with H t = {⟨ x 1 , y 1 ⟩ , . . . , ⟨ x t -1 , y t -1 ⟩} denoting the history of black-box evaluations up-to the current time-step t . Adding the current BO step t and the maximum budget T in our state variable s t helps balance exploration versus exploitation trade-offs as noted in [34]. Our MDP's transition function is straightforward, updating H t by appending newly evaluated points, i.e., H t +1 = H t ∪ {⟨ x t , y t ⟩} and incrementing the time variable t . Regarding rewards, we follow the well-established literature [13, 14] and define r t = max 1 ≤ ℓ ≤ t y ℓ to correspond to simple regret. Given such an MDP, our agent attempts to find a parameterised policy π θ which, when conditioned on s t , proposes a new probe x t that minimises cumulative regret (i.e., the sum of total discounted simple regrets).

Extensions to Multi-Task Reinforcement Learning The above MDP describes an RL configuration that learns θ in a single BO task. We now extend this formulation to multi-task scenarios allowing for a meta-learning setup. To do so, we introduce a set of MDPs M 1 , . . . , M K with K being the total number of available tasks. This paper considers same-domain multi-task learning scenarios. As such, we assume that all MDPs share the same state and action spaces and leave cross-domain extensions as an interesting avenue for future work. We define each MDP, M k , as previously introduced such that:

Moreover, for each task k , the transition model updates task-specific histories with H ( k ) t +1 = H ( k ) t ⊔ {⟨ x ( k ) t , y ( k ) t ⟩} and increments t ( k ) . Contrary to the single task setup, we now seek a policy π θ which performs well on average across K tasks:

where p tasks denotes the task distribution. Furthermore, the per-task history distribution p π θ ( H ( k ) T ( k ) ) is jointly parameterised by θ and defined as:

with µ 0 ( x ( k ) 1 ) denoting an initial (action) distribution from which we sample the first decision x ( k ) 1 . While it appears that off-the-shelf PPO can tackle the problem in Equation 1, Section 3.2 details difficulties associated with such an RL formulation, noting that in BO situations, the sparsity of reward signals can impede the learning of the network's weights θ . Before presenting those arguments, we now differentiate from the closest prior art, clarifying critical MDP differences.

Connection & Differences to [13] Although we are the first to propose end-to-end multi-task MDPs for meta-BO, others have also used RL to discover AFs, for example. Our parameterisation architecture and MDP definition significantly differ from the prior art, particularly from [13], the closest method to our work. Our approach uses a transformer-based deep network model to parameterise the whole pipeline (see Section 3.4). In contrast, the work in [13] assumes a pre-trained fixed Gaussian process surrogate and uses multi-layer perceptrons only to discover acquisitions. Our state variable requires historical information from which we jointly learn a probabilistic model and an acquisition end-to-end instead of requiring posterior means and variances of a Gaussian process model. Hence, our framework is more flexible, allowing us to model non-smooth black-box functions while overcoming some drawbacks of GP surrogates, like training and inference times.

3.2 Limitations of Regret Rewards in End-to-End Training

To define Equation 1, we followed the well-established literature of meta-BO [4, 13] and utilised simple regret reward functions. Although this choice is reasonable, we face challenges in applying such rewards in end-to-end training. Apart from difficulties associated with end-to-end training

of deep architectures [18], our RL algorithm is subject to additional complexities when estimating gradients from Equation 1 due to the sparsity of the reward function. To better understand this problem, we start by noticing that for a reward component r ( k ) t to contribute to the cumulative summation ∑ t γ t -1 r ( k ) t , we need to observe a function value y ( k ) t that outperforms all values we have seen so far, i.e., y ( k ) t > max 1 ≤ ℓ<t y ( k ) ℓ . During the early training stages of RL, we can quantify the average number of such informative events (when y ( k ) t > max 1 ≤ ℓ<t y ( k ) ℓ ) by a combinatorial argument that frames this calculation as a calculation of the number of cycles in a permutation of T ( k ) elements, leading us to the following lemma.

Lemma 3.1. Consider a task with a horizon length (budget) T , and define r t = max 1 ≤ ℓ ≤ t y t the simple regret as introduced in Equation 1. For a history H T , let m H denote the total number of informative rewards, i.e. the number of steps t at which y t > max 1 ≤ ℓ<t y ℓ . Under a random policy π θ , the number of informative events is logarithmic in T such that: E H∼ p π θ [ m H ] = O (log T ) , where p π θ is induced by π θ as in Equation 2.

We defer the proof of Lemma 3.1 to Appendix A due to space constraints. Here, we note that this result implies that the information contained in one sampled trajectory is sparse at the beginning of RL training when the policy acts randomly. Of course, this increases the difficulty of estimating informative gradients of Equation 1 when updating θ . One can argue that the sparsity described in Lemma 3.1 only holds under random policies during the early stages of RL training and that sparsity patterns decrease as policies improve. Interestingly, simple regret rewards do not necessarily confirm this intuition. To realise this, consider the other end of the RL training spectrum in which policies have improved to near optimality such that π θ → π θ ⋆ . Because π θ has been trained to maximise regret, it will seek to suggest the optimal point of the current task, as early as possible in the BO trajectory. Consequently, the policy is encouraged to produce trajectories with even sparser rewards during later training stages, further complicating the problem of informative gradient estimates of Equation 1.

3.3 Inductive Biases and Auxiliary Tasks

Learning from sparse reward signals is a well-known difficulty in the reinforcement learning literature [32]. Many solutions, from imitation learning, [35] to exploration bonuses [36], improve reward signals to reduce agent-environment interactions and enhance gradient updates. Others [37] attempt to define more informative rewards from prior knowledge or via human interactions [38]. Unfortunately, both of those approaches are hard to use in BO. Indeed, manually engineering black-box-specific rewards is notoriously difficult and requires domain expertise and extensive knowledge of the source and target black-box functions we wish to optimise. Furthermore, learning from human feedback is data-intensive, conflicting with the goal of sample-efficient optimisation.

Another prominent direction demonstrating significant gains is the introduction of auxiliary tasks (losses) within RL that allows agents to discover relevant inductive biases leading to impressive empirical successes [19, 39, 40]. Inspired by those results, we propose introducing an inductive bias in our method via an auxiliary supervised loss. Since we have at our disposal the collected source tasks datasets on which we are training our architecture D (1) , . . . , D ( K ) , we augment our objective such that our RL agent maximises not only rewards but also the likelihood of making correct predictions on these labelled datasets.

Supervised Auxiliary Loss Consider a source task k and consider that we split its corresponding dataset into an observed set and a prediction set D ( k ) = D ( k ) obs ⊔ D ( k ) perd . We define the auxiliary loss to be exactly the log-likelihood of functions values y ( pred ) at predicted locations x ( pred ) given observations D obs :

Interestingly, this part of our model specialises as a neural process (Section 2.2) with D ( k ) obs being the history H ( k ) t and x ( pred ) being the input points at which we wish to make predictions. To represent p θ ( · ) , we use a head of our transformer to predict multi-modal Riemannian (or bar plot probability density function) posteriors as [11, 41]. We compute Equation 3 on random iid splits of D ( k ) and not directly on trajectories generated by the policy. This is because as the policy improves, the trajectories

it generates are composed of less diverse (non-iid.) points. Indeed as training progresses, π θ becomes better and therefore finds the optimal point in D ( k ) more rapidly. To fully take advantage of the labelled data at our disposal, we evaluate this auxiliary loss on iid. data, i.e. splits of D ( k ) into D ( k ) obs and D ( k ) pred sampled uniformly at random. It is sensible from a BO standpoint as well since we want the introduced inductive bias to encourage our network to maximise the likelihood not only in a neighbourhood of the optimiser but also in the rest of the dataset.

3.4 Neural Acquisition Processes

Wenowintroduce in more detail our transformer architecture and note some of its important properties. We call this architecture the neural acquisition process (NAP) because it is a new type of NPs jointly predicting AF values and distribution over actions. Similarly to other NP [11, 12], it takes a context and queried locations as input. We parameterise it by θ and denote the acquisition prediction by α θ ( x (pred) , H , t, T ) .

Since α θ outputs real numbers, we still need to define how we can obtain a valid probability distribution over the action space to form a policy π θ ( ·| s t ) . Defining such a distribution over the whole action space is hard if A is continuous. Hence, similarly to Volpp et al. [13], we evaluate the policy π θ only on the finite set of locations x (pred) for a given task 1 . Therefore, we have:

We now have all the necessary components to define the full objective combining Equation 1 and 3: J ( θ ) = J ( π θ )+ λ L ( θ ) , where λ is a hyperparameter to balance between the two objectives. We summarise the full training algorithm in Alg. 1 detailing each of the components.

Finally, we study some desirable properties of NAPs, explain how we can achieve them, and why they are important in the context of BO.

Algorithm 1 Neural Acquisition Process training. Require: Source tasks training data {D ( k ) } K k =1 , initial parameters θ , budgets T ( k ) ≡ T , discount factor γ , learning rate η for each epoch do select task k and dataset D ( k ) , set H 0 = {∅} for t = 1 , . . . , T do x t ∼ π θ ( ·| s t ) ▷ predict action y t = f ( k ) ( x t ) ▷ execute action r t = y ∗ ≤ t ▷ collect reward H t +1 ←H t ∪ { ( x t , y t ) } ▷ update hist. end for R = ∑ T t =1 γ t r t ▷ cumul. reward D → D obs ⊔ D pred ▷ split source data L = p θ ( y ( pred ) | x ( pred ) , D obs ) ▷ aux. loss θ ← θ + η ( ∇ θ R + ∇ θ L ) ▷ update θ end for

Property 3.2 (History-order invariance) . An NP g is history-order invariant if for any choice of permutation function ψ that changes the order of the points in history, g ( x , ψ ( H )) = g ( x , H ) .

Unlike vanilla transformers, we do not use a positional encoding. It allows NAP to treat the history H as a set instead of an ordered sequence [42] and to be invariant to the order of the history. To form an ⟨ x , y ⟩ pair in this set, we sum the embedding of x and y [11], whereas for queried locations we only use the embedding of x for a token (see Fig. 1, left). It is important for BO since the order in which we collect the points is not relevant for assessing how promising is a new queried location. Previous meta-RL algorithms [43] also shown the importance of relying on order-invariance.

Property 3.3 (Query independence) . A NP g is query independent if for any choice of n queried locations x (pred) = ( x (pred) 1 , . . . , x (pred) n ) , we have g ( x (pred) , H ) = ( g ( x (pred) 1 , H ) , . . . , g ( x (pred) n , H )) .

In NAP, every token in the history can access each other through the self-attention mask, whereas the elements of x (pred) can only attend to themselves and to tokens in the history H (see Fig. 1, right). Because the tokens inside x (pred) cannot access each other and we do not use positional encoding, NAP is query independent, which is important to make consistent predictions of AF values for BO as they should not depend on the other queried locations. Additionally, NAP is fully differentiable enabling end-to-end training but also optimisation of the queried locations via gradient ascent for continuous action spaces. In Table 1, we highlight the differences with other state-of-the-art models regarding those properties.

x

1

y

1

+

x

2

y

2

+

x

3

y

3

+

x

(pred)

1

x

(pred)

2

t, T

α

θ

(

x

(pred)

1

)

α

θ

(

x

(pred)

2

)

Self-attention layer (

×

n)

Embedding

x

1

, y

1

x

1

, y

1

x

2

, y

2

x

2

, y

2

x

3

, y

3

x

3

, y

3

x

(pred)

1

x

(pred)

1

x

(pred)

2

x

(pred)

2

4 Experiments

We conduct experiments on hyperparameter optimisation (HPO) and sequence optimisation tasks to assess NAP's efficiency. Regarding HPO, we run our algorithm on datasets from the HPO-B benchmark [44] and in real-world settings of tuning hyperparameters for Mixed-Integer Programming (MIP) solvers. For sequence optimisation, we test NAP on combinatorial black-box problems from antibody design and synthesis flow optimisation for electronic design and automation (EDA).

Baselines We compare our method against popular meta-learning baselines, including few-shot Bayesian optimisation (FSBO) [5], MetaBO [13] as well as OptFormers [41] when applicable. Moreover, we show how classical GP-based BO, equipped with an adequate kernel 2 and an EI acquisition function, which we title GP-EI, performs across domains. We first fit a GP on the metatraining datasets to enable a fair comparison. We initialise the GP model with those learnt kernel parameters at test time. This way, the GP baseline can benefit from the information provided in the source tasks. We also explore training a neural process directly on source task datasets and then using a fixed EI acquisition. For that, we introduce NP-EI that combines the same base architecture from [11] with EI. Additionally, we contrast NAP against random search (RS).

Following the standard practice in meta-BO, we report our results in terms of normalised regrets for easier comparison across all tasks. We attempt to re-implement all baselines across all empirical domains, extending previous implementations as needed, e.g., developing MetaBO [13] and FSBO [5] versions for combinatorial and mixed spaces to enable a fair comparison in MIP solver tuning, antibody and EDA design tasks.

Remark on OptFormer We re-run OptFormer with an EI AF on hyperparameter tuning tasks by extending the implementation in [41] to the discrete search space version of HPO-B. However, the lack of a complete open-source implementation and interpretable meta-data in the other benchmarks (e.g., in antibody design and EDA) prohibited successful execution.

4.1 Hyperparameter Optimisation Results

Hyperparameter Optimisation Benchmarks We experiment on the HPO-B benchmark [44], which contains datasets of (classification) model hyperparameters and the models' corresponding accuracies across multiple types and search spaces. Due to resource constraints, we selected six representative search spaces. Nonetheless, we chose the search spaces to represent all underlying classification models in the experiment. We also always picked the ones with the least points to focus on the low data regime performance; see Appendix C.1 for more details. The results of our tests in Figure 2 demonstrate that NAP and OptFormer outperform all other baselines. Surprisingly, although NAP uses a much smaller architecture than OptFormer (around 15 million parameters vs 250 million for OptFormer) and trains on much less data (around 80k original points on average versus more than 3 million points for OptFormer), its regret performance is statistically similar to that of the OptFormer after 100 steps. Moreover, on the same GPU, to perform the same inference, NAP only uses 2% of OptFormer's compute time and around 40% of its memory usage.

Tuning MIP Solvers Apart from HPO-B, we consider another real-world example of hyperparameter tuning that requires finding the optimal parameters of MIP solvers. We use the open-source SCIP solver [45] and the Benchmark suite from the MIPLib2017 [46] that consists of a collection of 240 problems. The objective is to find a set of hyperparameters so that SCIP can solve MIP instances in minimal time. Our high-dimensional search space comprises 135 hyperparameters with mixed types, including boolean, integers, categories and real numbers. We train our model on data collected from BO traces on 103 MIPs and test on a held-out set of 42 instances. Our results in Figure 2 demonstrate that NAP is capable of outperforming all other baselines reaching low regret about an order of magnitude faster than FSBO [5]. Figure 2 further demonstrates the importance of end-to-end training where NAP again outperforms NP-EI.

4.2 Sequence Optimisation Experiments

Now, we demonstrate NAP's abilities beyond hyperparameter tuning tasks in two real-world combinatorial black-box optimisation problems.

Antibody CDRH3-Sequence Optimisation This experiment focuses on finding antibodies that can bind to target antigens. Antigens are proteins, i.e., sequences of amino acids that fold into a 3D shape giving them specific chemical properties. A protein region called CDRH3 is decisive in the antibody's ability to bind to a target antigen. Following the work in [47], we represent CDRH3s as a string of 11 characters, each character being the code for a different amino acid in an alphabet of cardinality 22. The goal is to find the optimal CDRH3 that minimises the binding energy towards a specific antigen. Binding energies can be computed using state-of-the-art simulation software like Absolut! [48]. We collected datasets of CDRH3 sequences and their respective binding energies (with Absolut!) across various antigens from the protein database bank [49]. We then formed a transfer scenario across antigens where we meta-learn on 109 datasets, validate on 16, and test NAP on 32 new antigens. Our results in Figure 2 indicate that NAP is not limited to hyperparameter tuning tasks but can also outperform all other baselines in combinatorial domains.

Electronic Design Automation (EDA) Logic synthesis (LS) is an essential step in the EDA pipeline of the chip design process. At the beginning of LS, we represent the circuit as an AIG (an And-InverterGraph representation of Boolean functions) and seek to map it to a netlist of technology-dependent gates (e.g., 6-input logic gates in FPGA mapping). The goal in LS is to find a sequence of graph transformations such that the resulting netlist meets an objective that trades off the number of gates (area) and the size of the longest directed path (delay) in the netlist. We perform a sequence of logic synthesis operators dubbed a synthesis flow to optimise the AIG.

Following [50], we consider length 20 LS flows and allow an alphabet of 11 such operators, e.g., { refactor , resub , . . . , balance } as implemented in the open-source ABC library [51]. We collected datasets for 43 different circuits. Each dataset consisted of 500 sequences (collected via a Genetic algorithm optimizer) and their associated area and delay. Additionally, we applied the well-known heuristic sequence resyn2 on each circuit to get a reference area and delay. For this task, the black-box takes a sequence as input and returns the sum of area and delay ratios with respect to the reference ones, as detailed in Appendix B.1. We train all methods on 30 circuits from OpenABC [50], validate on 4 and test on 9. Our results in Figure 2 again demonstrate that NAP outperforms all other baselines by a significant margin.

HPO-B

MIP

Antibody

EDA

gret

10-1

10-1

10-1

10-1

Regr

verage

10-2

10-2

10-2

10-2

10-3

10-3

0

25

50

75

100

100

101

102

0

100

200

300

10-3

0

100

200

300

RS

GP-EI

MetaBO

FSBO

NP-EI

OptFormer-El

NAP

5 Related Work

Meta-Learning Paradigms Meta-learning aims to learn how to quickly solve a new task based on the experience gained from solving or observing data from related tasks [15]. To achieve this goal, we can follow two main directions. In the first one, the meta-learner uses source data to learn a good initialisation of its model parameters, such that it is able to make accurate predictions after only few optimisation steps on the new target task [52]. The second one is more ambitious as it endeavours to directly learn, from the related tasks, a general rule to predict a helpful quantity to suggest the next point (black-box value, acquisition function value, etc.) from a context of observations (i.e. our history H ( k ) ). Works following that direction, which include ours, can rely on different types of models to learn this general rule, such as NPs [10], recurrent neural networks [53], HyperNetworks [16] or transformers [12, 11]. Orthogonal to these two directions is the learning of a hyper-posterior on the datasets to meta-train on. In that line of research, Rothfuss et al. [54] suggest the use of Stein Variational Gradient Descent (SVGD) to estimate uncertainty with multiple models better, and Hsieh et al. [14] extend the use of SVGD to meta-learn AFs. We consider those last two works as an orthogonal direction to our, as NAP could also benefit from SVGD updates.

Learning a Meta-Model Chen et al. [55] and TV et al. [56] train an RNN to predict what should be the next suggestion in BO instead of predicting the acquisition function value. We do not compare to their method given the lack of available implementation. Still, we compare to the outperforming approach developed by Chen et al. [41] based on a transformer architecture designed to do meta-BO on hyperparameter tuning problems. Their OptFormer is trained on related tasks over different search spaces to output the next point to evaluate and predict its value. The next point is sequentially decoded, one token at a time, and exploits the hyperparameters' names or descriptions to improve generalisation across tasks. Contrary to NAP, OptFormer is not designed to predict acquisition value at any point and does not meet the two properties 3.2 and 3.3, and therefore needs much more training data to predict the proper sequence of tokens. We note that NAP does not rely on variable descriptions, making it easily deployable on various tasks and still very competitive in the hyperparameter optimisation context.

Rather than learning an entire predictive model from scratch, prior works [5, 7-9] learn deep kernel surrogates, i.e. a warping functions mapping the input space with a neural network before it is given to a GP. Learning only the input transformation allows the authors to rely on the closed-form posterior prediction capacity of standard GP models. To perform the transfer, Feurer et al. [6] rely on an ensemble of GPs. Iwata [57] is an approach close to FSBO [5] as it learns a meta-model by meta-training a Deep Kernel GP. Notably, it does so in an end-to-end fashion using RL to propagate gradients through the acquisition function and the GP back to the deep kernel. It does not, however, learn a meta-acquisition.

Learning a Meta-Acquisition Function Hsieh et al. [14] and Volpp et al. [13] choose to perform transfer through the acquisition function. They use directly GP surrogates and define the acquisition as a neural network that is meta-trained on related source tasks. They first pre-train GP surrogates on all source tasks and fix their kernel parameters. They then rely on RL training to meta-learn the

neural acquisition function that takes as inputs the posterior mean and variance of the GP surrogate (which is itself trained online at test time). At test time, they allow for update in the GP but keep the weights of the neural acquisition fixed.

In summary, the methods meta-learning AFs do not do so in an end-to-end fashion and still rely on trained GP surrogates. While these methods are principled and competitive, they suffer from the cost of inverting the GP kernel matrix (cubic in the number of observations). In comparison, NAP can make predictions through a simple forward pass. On the other hand, the methods that learn a meta-model, either use a Deep Kernel GP (suffering the same cost) or have to learn a large model from scratch, costing a lot of budgets for collecting data beforehand, as well a pre-training time. Both of them use standard acquisition functions, missing the potential benefits of doing transfer in acquisition between tasks.

The performance of some of those algorithms on HPO-B is presented in Appendix-C showing that despite learning an architecture from scratch, NAP achieves a lower regret. For a more detailed survey on transfer and meta-learning in BO, we refer the reader to Bai et al. [4].

6 Conclusion

We proposed the first end-to-end training protocol to meta-learn acquisition functions in meta-BO. Our method predicts acquisition values directly from a history of points with meta-reinforcement learning and auxiliary losses. We demonstrated new state-of-the-art results compared to popular baselines in a wide range of benchmarks.

Limitations & Future Work Our architecture suffers from the usual quadratic complexity of the transformer in terms of the number of tokens which limits the budget of BO steps. Nevertheless, it can still handle around 5000 steps, which is enough for most BO scenarios. Another limitation of our architecture is that we need to train a new model for each search space. In future, we plan to enable our method to leverage meta-training from multiple search spaces and investigate how we could design data-driven BO-specific augmentation to further mitigate meta-overfitting [58].

References

Jasper Snoek, Hugo Larochelle, and Ryan P. Adams. Practical bayesian optimization of machine learning algorithms. In Peter L. Bartlett, Fernando C. N. Pereira, Christopher J. C. Burges, Léon Bottou, and Kilian Q. Weinberger, editors, Advances in Neural Information Processing Systems 25: 26th Annual Conference on Neural Information Processing Systems 2012. Proceedings of a meeting held December 3-6, 2012, Lake Tahoe, Nevada, United States , pages 2960-2968, 2012. URL https://proceedings.neurips.cc/paper/2012/hash/ 05311655a15b75fab86956663e1819cd-Abstract.html .

Nate Gruver, Samuel Stanton, Polina Kirichenko, Marc Finzi, Phillip Maffettone, Vivek Myers, Emily Delaney, Peyton Greenside, and Andrew Gordon Wilson. Effective surrogate models for protein design with bayesian optimization. In ICML Workshop on Computational Biology , 2021.

Fabio Muratore, Christian Eilers, Michael Gienger, and Jan Peters. Data-efficient domain randomization with bayesian optimization. IEEE Robotics and Automation Letters , 6(2): 911-918, 2021.

Tianyi Bai, Yang Li, Yu Shen, Xinyi Zhang, Wentao Zhang, and Bin Cui. Transfer learning for bayesian optimization: A survey, 2023.

Martin Wistuba and Josif Grabocka. Few-shot bayesian optimization with deep kernel surrogates. In 9th International Conference on Learning Representations, ICLR 2021, Virtual Event, Austria, May 3-7, 2021 . OpenReview.net, 2021. URL https://openreview.net/forum? id=bJxgv5C3sYc .

Matthias Feurer, Benjamin Letham, and Eytan Bakshy. Scalable meta-learning for bayesian optimization using ranking-weighted gaussian process ensembles. In AutoML Workshop at ICML , volume 7, 2018.

Massimiliano Patacchiola, Jack Turner, Elliot J. Crowley, Michael O' Boyle, and Amos J Storkey. Bayesian meta-learning for the few-shot setting via deep kernels. In H. Larochelle, M. Ranzato, R. Hadsell, M.F. Balcan, and H. Lin, editors, Advances in Neural Information Processing Systems , volume 33, pages 16108-16118. Curran Associates, Inc., 2020. URL https://proceedings.neurips.cc/paper_files/paper/2020/file/ b9cfe8b6042cf759dc4c0cccb27a6737-Paper.pdf .

Zi Wang, George E. Dahl, Kevin Swersky, Chansoo Lee, Zachary Nado, Justin Gilmer, Jasper Snoek, and Zoubin Ghahramani. Pre-trained gaussian processes for bayesian optimization, 2023.

Wenlin Chen, Austin Tripp, and José Miguel Hernández-Lobato. Meta-learning adaptive deep kernel gaussian processes for molecular property prediction. In The Eleventh International Conference on Learning Representations , 2023. URL https://openreview.net/forum? id=KXRSh0sdVTP .

Marta Garnelo, Dan Rosenbaum, Christopher Maddison, Tiago Ramalho, David Saxton, Murray Shanahan, Yee Whye Teh, Danilo Jimenez Rezende, and S. M. Ali Eslami. Conditional neural processes. In Jennifer G. Dy and Andreas Krause, editors, Proceedings of the 35th International Conference on Machine Learning, ICML 2018, Stockholmsmässan, Stockholm, Sweden, July 10-15, 2018 , volume 80 of Proceedings of Machine Learning Research , pages 1690-1699. PMLR, 2018. URL http://proceedings.mlr.press/v80/garnelo18a.html .

Samuel Müller, Noah Hollmann, Sebastian Pineda-Arango, Josif Grabocka, and Frank Hutter. Transformers can do bayesian inference. In The Tenth International Conference on Learning Representations, ICLR 2022, Virtual Event, April 25-29, 2022 . OpenReview.net, 2022. URL https://openreview.net/forum?id=KSugKcbNf9 .

Tung Nguyen and Aditya Grover. Transformer neural processes: Uncertainty-aware meta learning via sequence modeling. In Kamalika Chaudhuri, Stefanie Jegelka, Le Song, Csaba Szepesvári, Gang Niu, and Sivan Sabato, editors, International Conference on Machine Learning, ICML 2022, 17-23 July 2022, Baltimore, Maryland, USA , volume 162 of Proceedings of Machine Learning Research , pages 16569-16594. PMLR, 2022. URL https://proceedings.mlr.press/v162/nguyen22b.html .

Michael Volpp, Lukas P. Fröhlich, Kirsten Fischer, Andreas Doerr, Stefan Falkner, Frank Hutter, and Christian Daniel. Meta-learning acquisition functions for transfer learning in bayesian optimization. In 8th International Conference on Learning Representations, ICLR 2020, Addis Ababa, Ethiopia, April 26-30, 2020 . OpenReview.net, 2020. URL https://openreview. net/forum?id=ryeYpJSKwr .

Bing-Jing Hsieh, Ping-Chun Hsieh, and Xi Liu. Reinforced few-shot acquisition function learning for bayesian optimization. In M. Ranzato, A. Beygelzimer, Y. Dauphin, P.S. Liang, and J. Wortman Vaughan, editors, Advances in Neural Information Processing Systems , volume 34, pages 7718-7731. Curran Associates, Inc., 2021. URL https://proceedings.neurips. cc/paper/2021/file/3fab5890d8113d0b5a4178201dc842ad-Paper.pdf .

Timothy M. Hospedales, Antreas Antoniou, Paul Micaelli, and Amos J. Storkey. Meta-learning in neural networks: A survey. IEEE Trans. Pattern Anal. Mach. Intell. , 44(9):5149-5169, 2022. doi: 10.1109/TPAMI.2021.3079209. URL https://doi.org/10.1109/TPAMI.2021. 3079209 .

David Ha, Andrew M. Dai, and Quoc V. Le. Hypernetworks. In 5th International Conference on Learning Representations, ICLR 2017, Toulon, France, April 24-26, 2017, Conference Track Proceedings . OpenReview.net, 2017. URL https://openreview.net/forum?id= rkpACe1lx .

Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. In Isabelle Guyon, Ulrike von Luxburg, Samy Bengio, Hanna M. Wallach, Rob Fergus, S. V. N. Vishwanathan, and Roman Garnett, editors, Advances in Neural Information Processing Systems 30: Annual Conference on Neural Information Processing Systems 2017, December 4-9, 2017, Long Beach, CA, USA , pages 5998-6008, 2017. URL https://proceedings.neurips.cc/paper/2017/hash/ 3f5ee243547dee91fbd053c1c4a845aa-Abstract.html .

Emilio Parisotto, Francis Song, Jack Rae, Razvan Pascanu, Caglar Gulcehre, Siddhant Jayakumar, Max Jaderberg, Raphaël Lopez Kaufman, Aidan Clark, Seb Noury, Matthew Botvinick, Nicolas Heess, and Raia Hadsell. Stabilizing transformers for reinforcement learning. In Hal Daumé III and Aarti Singh, editors, Proceedings of the 37th International Conference on Machine Learning , volume 119 of Proceedings of Machine Learning Research , pages 7487-7498. PMLR, 13-18 Jul 2020. URL https://proceedings.mlr.press/v119/parisotto20a. html .

Max Jaderberg, Volodymyr Mnih, Wojciech Marian Czarnecki, Tom Schaul, Joel Z. Leibo, David Silver, and Koray Kavukcuoglu. Reinforcement learning with unsupervised auxiliary tasks. In 5th International Conference on Learning Representations, ICLR 2017, Toulon, France, April 24-26, 2017, Conference Track Proceedings . OpenReview.net, 2017. URL https://openreview.net/forum?id=SJ6yPD5xg .

Carl Edward Rasmussen and Christopher KI Williams. Gaussian Processes for Machine Learning , volume 2. MIT press Cambridge, MA, 2006.

Jonas Mockus, Vytautas Tiesis, and Antanas Zilinskas. The application of Bayesian methods for seeking the extremum. Towards Global Optimization , 2(117-129):2, 1978.

Thomas Bäck and Hans-Paul Schwefel. An overview of evolutionary algorithms for parameter optimization. Evolutionary computation , 1(1):1-23, 1993.

James Bergstra and Yoshua Bengio. Random search for hyper-parameter optimization. Journal of machine learning research , 13(2), 2012.

Matthias Feurer, Jost Tobias Springenberg, and Frank Hutter. Initializing bayesian hyperparameter optimization via meta-learning. In Blai Bonet and Sven Koenig, editors, Proceedings of the Twenty-Ninth AAAI Conference on Artificial Intelligence, January 25-30, 2015, Austin, Texas, USA , pages 1128-1135. AAAI Press, 2015. URL http://www.aaai.org/ocs/index.php/ AAAI/AAAI15/paper/view/10029 .

Martin Wistuba, Nicolas Schilling, and Lars Schmidt-Thieme. Scalable gaussian process-based transfer surrogates for hyperparameter optimization. Mach. Learn. , 107(1):43-78, 2018. doi: 10.1007/s10994-017-5684-y. URL https://doi.org/10.1007/s10994-017-5684-y .

Valerio Perrone and Huibin Shen. Learning search spaces for bayesian optimization: Another view of hyperparameter transfer learning. In Hanna M. Wallach, Hugo Larochelle, Alina Beygelzimer, Florence d'Alché-Buc, Emily B. Fox, and Roman Garnett, editors, Advances in Neural Information Processing Systems 32: Annual Conference on Neural Information Processing Systems 2019, NeurIPS 2019, December 8-14, 2019, Vancouver, BC, Canada , pages 12751-12761, 2019. URL https://proceedings.neurips.cc/paper/2019/hash/ 6ea3f1874b188558fafbab78e8c3a968-Abstract.html .

Pavel Izmailov, Alexander Novikov, and Dmitry Kropotov. Scalable gaussian processes with billions of inducing inputs via tensor train decomposition. In Amos J. Storkey and Fernando Pérez-Cruz, editors, International Conference on Artificial Intelligence and Statistics, AISTATS 2018, 9-11 April 2018, Playa Blanca, Lanzarote, Canary Islands, Spain , volume 84 of Proceedings of Machine Learning Research , pages 726-735. PMLR, 2018. URL http://proceedings.mlr.press/v84/izmailov18a.html .

Edwin V. Bonilla, Kian Ming Adam Chai, and Christopher K. I. Williams. Multitask gaussian process prediction. In John C. Platt, Daphne Koller, Yoram Singer, and Sam T. Roweis, editors, Advances in Neural Information Processing Systems 20, Proceedings of the Twenty-First Annual Conference on Neural Information Processing Systems, Vancouver, British Columbia, Canada, December 3-6, 2007 , pages 153-160. Curran Associates, Inc., 2007. URL https://proceedings.neurips.cc/paper/2007/hash/ 66368270ffd51418ec58bd793f2d9b1b-Abstract.html .

Marta Garnelo, Dan Rosenbaum, Christopher Maddison, Tiago Ramalho, David Saxton, Murray Shanahan, Yee Whye Teh, Danilo Jimenez Rezende, and S. M. Ali Eslami. Conditional neural processes. In Jennifer G. Dy and Andreas Krause, editors, Proceedings of the 35th International Conference on Machine Learning, ICML 2018, Stockholmsmässan, Stockholm, Sweden, July 10-15, 2018 , volume 80 of Proceedings of Machine Learning Research , pages 1690-1699. PMLR, 2018. URL http://proceedings.mlr.press/v80/garnelo18a.html .

Jonathan Gordon, Wessel P. Bruinsma, Andrew Y. K. Foong, James Requeima, Yann Dubois, and Richard E. Turner. Convolutional conditional neural processes. In 8th International Conference on Learning Representations, ICLR 2020, Addis Ababa, Ethiopia, April 26-30, 2020 . OpenReview.net, 2020. URL https://openreview.net/forum?id=Skey4eBYPS .

Hyunjik Kim, Andriy Mnih, Jonathan Schwarz, Marta Garnelo, S. M. Ali Eslami, Dan Rosenbaum, Oriol Vinyals, and Yee Whye Teh. Attentive neural processes. In 7th International Conference on Learning Representations, ICLR 2019, New Orleans, LA, USA, May 6-9, 2019 . OpenReview.net, 2019. URL https://openreview.net/forum?id=SkE6PjC9KX .

Richard S. Sutton and Andrew G. Barto. Reinforcement learning - an introduction . Adaptive computation and machine learning. MIT Press, 1998. ISBN 978-0-262-19398-6. URL https: //www.worldcat.org/oclc/37293240 .

John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. CoRR , abs/1707.06347, 2017. URL http://arxiv.org/ abs/1707.06347 .

Niranjan Srinivas, Andreas Krause, Sham M. Kakade, and Matthias W. Seeger. Gaussian process optimization in the bandit setting: No regret and experimental design. In Johannes Fürnkranz and Thorsten Joachims, editors, Proceedings of the 27th International Conference on Machine Learning (ICML-10), June 21-24, 2010, Haifa, Israel , pages 1015-1022. Omnipress, 2010. URL https://icml.cc/Conferences/2010/papers/422.pdf .

Jonathan Ho and Stefano Ermon. Generative adversarial imitation learning. Advances in neural information processing systems , 29, 2016.

Haoran Tang, Rein Houthooft, Davis Foote, Adam Stooke, OpenAI Xi Chen, Yan Duan, John Schulman, Filip DeTurck, and Pieter Abbeel. # exploration: A study of count-based exploration for deep reinforcement learning. Advances in neural information processing systems , 30, 2017.

Andrew Y Ng, Daishi Harada, and Stuart Russell. Policy invariance under reward transformations: Theory and application to reward shaping. In ICML , volume 99, pages 278-287. Citeseer, 1999.

Paul F Christiano, Jan Leike, Tom Brown, Miljan Martic, Shane Legg, and Dario Amodei. Deep reinforcement learning from human preferences. Advances in neural information processing systems , 30, 2017.

Xingyu Lin, Harjatin Baweja, George Kantor, and David Held. Adaptive auxiliary task weighting for reinforcement learning. Advances in neural information processing systems , 32, 2019.

Pablo Hernandez-Leal, Bilal Kartal, and Matthew E Taylor. Agent modeling as auxiliary task for deep reinforcement learning. In Proceedings of the AAAI conference on artificial intelligence and interactive digital entertainment , volume 15, pages 31-37, 2019.

Yutian Chen, Xingyou Song, Chansoo Lee, Zi Wang, Richard Zhang, David Dohan, Kazuya Kawakami, Greg Kochanski, Arnaud Doucet, Marc Aurelio Ranzato, Sagi Perel, and Nando de Freitas. Towards learning universal hyperparameter optimizers with transformers. In S. Koyejo, S. Mohamed, A. Agarwal, D. Belgrave, K. Cho, and A. Oh, editors, Advances in Neural Information Processing Systems , volume 35, pages 32053-32068. Curran Associates, Inc., 2022. URL https://proceedings.neurips.cc/paper_files/paper/2022/file/ cf6501108fced72ee5c47e2151c4e153-Paper-Conference.pdf .

Juho Lee, Yoonho Lee, Jungtaek Kim, Adam R. Kosiorek, Seungjin Choi, and Yee Whye Teh. Set transformer: A framework for attention-based permutation-invariant neural networks. In Kamalika Chaudhuri and Ruslan Salakhutdinov, editors, Proceedings of the 36th International Conference on Machine Learning, ICML 2019, 9-15 June 2019, Long Beach, California, USA , volume 97 of Proceedings of Machine Learning Research , pages 3744-3753. PMLR, 2019. URL http://proceedings.mlr.press/v97/lee19d.html .

Kate Rakelly, Aurick Zhou, Chelsea Finn, Sergey Levine, and Deirdre Quillen. Efficient off-policy meta-reinforcement learning via probabilistic context variables. In Kamalika Chaudhuri and Ruslan Salakhutdinov, editors, Proceedings of the 36th International Conference on Machine Learning , volume 97 of Proceedings of Machine Learning Research , pages 53315340. PMLR, 09-15 Jun 2019. URL https://proceedings.mlr.press/v97/rakelly19a. html .

Sebastian Pineda-Arango, Hadi S. Jomaa, Martin Wistuba, and Josif Grabocka. HPO-B: A largescale reproducible benchmark for black-box HPO based on openml. In Joaquin Vanschoren and Sai-Kit Yeung, editors, Proceedings of the Neural Information Processing Systems Track on Datasets and Benchmarks 1, NeurIPS Datasets and Benchmarks 2021, December 2021, virtual , 2021. URL https://datasets-benchmarks-proceedings.neurips.cc/paper/2021/ hash/ec8956637a99787bd197eacd77acce5e-Abstract-round2.html .

Ksenia Bestuzheva, Mathieu Besançon, Wei-Kun Chen, Antonia Chmiela, Tim Donkiewicz, Jasper van Doornmalen, Leon Eifler, Oliver Gaul, Gerald Gamrath, Ambros Gleixner, et al. The scip optimization suite 8.0. arXiv preprint arXiv:2112.08872 , 2021.

Ambros M. Gleixner, Gregor Hendel, Gerald Gamrath, Tobias Achterberg, Michael Bastubbe, Timo Berthold, Philipp Christophel, Kati Jarck, Thorsten Koch, Jeff T. Linderoth, Marco E. Lübbecke, Hans D. Mittelmann, Derya B. Özyurt, Ted K. Ralphs, Domenico Salvagnin, and Yuji Shinano. MIPLIB 2017: data-driven compilation of the 6th mixed-integer programming library. Math. Program. Comput. , 13(3):443-490, 2021. doi: 10.1007/s12532-020-00194-3. URL https://doi.org/10.1007/s12532-020-00194-3 .

Asif Khan, Alexander I. Cowen-Rivers, Antoine Grosnit, Derrick-Goh-Xin Deik, Philippe A. Robert, Victor Greiff, Eva Smorodina, Puneet Rawat, Rahmad Akbar, Kamil Dreczkowski, Rasul Tutunov, Dany Bou-Ammar, Jun Wang, Amos Storkey, and Haitham Bou-Ammar. Toward realworld automated antibody design with combinatorial bayesian optimization. Cell Reports Methods , page 100374, 2023. ISSN 2667-2375. doi: https://doi.org/10.1016/j.crmeth.2022.100374. URL https://www.sciencedirect.com/science/article/pii/S2667237522002764 .

Philippe A. Robert, Rahmad Akbar, Robert Frank, Milena Pavlovi´ c, Michael Widrich, Igor Snapkov, Andrei Slabodkin, Maria Chernigovskaya, Lonneke Scheffer, Eva Smorodina, Puneet Rawat, Brij Bhushan Mehta, Mai Ha Vu, Ingvild Frøberg Mathisen, Aurél Prósz, Krzysztof Abram, Alex Olar, Enkelejda Miho, Dag Trygve Tryslew Haug, Fridtjof Lund-Johansen, Sepp Hochreiter, Ingrid Hobæk Haff, Günter Klambauer, Geir Kjetil Sandve, and Victor Greiff. Unconstrained generation of synthetic antibody-antigen structures to guide machine learning methodology for antibody specificity prediction. Nature Computational Science , 2(12):845-865, 2022. ISSN 2662-8457. doi: 10.1038/s43588-022-00372-4. URL https://www.nature. com/articles/s43588-022-00372-4 .

Helen M Berman, John Westbrook, Zukang Feng, Gary Gilliland, Talapady N Bhat, Helge Weissig, Ilya N Shindyalov, and Philip E Bourne. The protein data bank. Nucleic acids research , 28(1):235-242, 2000.

Animesh Basak Chowdhury, Benjamin Tan, Ramesh Karri, and Siddharth Garg. Openabc-d: A large-scale dataset for machine learning guided integrated circuit synthesis, 2021.

Robert Brayton and Alan Mishchenko. Abc: An academic industrial-strength verification tool. In Tayssir Touili, Byron Cook, and Paul Jackson, editors, Computer Aided Verification , pages 24-40, Berlin, Heidelberg, 2010. Springer Berlin Heidelberg. ISBN 978-3-642-14295-6.

Chelsea Finn, Pieter Abbeel, and Sergey Levine. Model-agnostic meta-learning for fast adaptation of deep networks. In Doina Precup and Yee Whye Teh, editors, Proceedings of the 34th International Conference on Machine Learning, ICML 2017, Sydney, NSW, Australia, 6-11 August 2017 , volume 70 of Proceedings of Machine Learning Research , pages 1126-1135. PMLR, 2017. URL http://proceedings.mlr.press/v70/finn17a.html .

Yan Duan, Marcin Andrychowicz, Bradly C. Stadie, Jonathan Ho, Jonas Schneider, Ilya Sutskever, Pieter Abbeel, and Wojciech Zaremba. One-shot imitation learning. In Isabelle Guyon, Ulrike von Luxburg, Samy Bengio, Hanna M. Wallach, Rob Fergus, S. V. N. Vishwanathan, and Roman Garnett, editors, Advances in Neural Information Processing Systems 30: Annual Conference on Neural Information Processing Systems 2017, December 4-9, 2017, Long Beach, CA, USA , pages 1087-1098, 2017. URL https://proceedings.neurips.cc/ paper/2017/hash/ba3866600c3540f67c1e9575e213be0a-Abstract.html .

Jonas Rothfuss, Vincent Fortuin, Martin Josifoski, and Andreas Krause. PACOH: bayes-optimal meta-learning with pac-guarantees. In Marina Meila and Tong Zhang, editors, Proceedings of the 38th International Conference on Machine Learning, ICML 2021, 18-24 July 2021, Virtual Event , volume 139 of Proceedings of Machine Learning Research , pages 9116-9126. PMLR, 2021. URL http://proceedings.mlr.press/v139/rothfuss21a.html .

Yutian Chen, Matthew W. Hoffman, Sergio Gómez Colmenarejo, Misha Denil, Timothy P. Lillicrap, Matt Botvinick, and Nando de Freitas. Learning to learn without gradient descent by gradient descent. In Doina Precup and Yee Whye Teh, editors, Proceedings of the 34th International Conference on Machine Learning , volume 70 of Proceedings of Machine Learning Research , pages 748-756. PMLR, 06-11 Aug 2017. URL https://proceedings.mlr. press/v70/chen17e.html .

Vishnu TV, Pankaj Malhotra, Jyoti Narwariya, Lovekesh Vig, and Gautam Shroff. Metalearning for black-box optimization. In Ulf Brefeld, Élisa Fromont, Andreas Hotho, Arno J. Knobbe, Marloes H. Maathuis, and Céline Robardet, editors, Machine Learning and Knowledge Discovery in Databases - European Conference, ECML PKDD 2019, Würzburg, Germany, September 16-20, 2019, Proceedings, Part II , volume 11907 of Lecture Notes in Computer Science , pages 366-381. Springer, 2019. doi: 10.1007/978-3-030-46147-8\_22. URL https: //doi.org/10.1007/978-3-030-46147-8_22 .

Tomaharu Iwata. End-to-end learning of deep kernel acquisition functions for bayesian optimization, 2021.

Huaxiu Yao, Long-Kai Huang, Linjun Zhang, Ying Wei, Li Tian, James Zou, Junzhou Huang, and Zhenhui Li. Improving generalization in meta-learning via task augmentation. In Marina Meila and Tong Zhang, editors, Proceedings of the 38th International Conference on Machine Learning, ICML 2021, 18-24 July 2021, Virtual Event , volume 139 of Proceedings of Machine Learning Research , pages 11887-11897. PMLR, 2021. URL http://proceedings.mlr. press/v139/yao21b.html .

V. L. Goncharov. Sur la distribution des cycles dans les permutations. Acad. Sci. URSS (N.S.) , 35:267-269, 1944.

Katharina Eggensperger, Philipp Müller, Neeratyoy Mallik, Matthias Feurer, René Sass, Aaron Klein, Noor H. Awad, Marius Lindauer, and Frank Hutter. Hpobench: A collection of reproducible multi-fidelity benchmark problems for HPO. In Joaquin Vanschoren and Sai-Kit Yeung, editors, Proceedings of the Neural Information Processing Systems Track on Datasets and Benchmarks 1, NeurIPS Datasets and Benchmarks 2021, December 2021, virtual , 2021. URL https://datasets-benchmarks-proceedings.neurips.cc/paper/2021/hash/ 93db85ed909c13838ff95ccfa94cebd9-Abstract-round2.html .

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe authors propose Neural Acquisition Process, a novel method for Bayesian optimization by meta-learning a function that jointly performs the surrogate and acquisition steps. The function, i.e. a transformer, is able to work as a policy that predicts the action **a** given a state **s**, where the action is the hyperparameter to observe. The state comprises the history of the observed samples. Also, the transformer can predict probabilistic the performance of a hyperparameter. They use reinforcement learning to train the policy and, additionally, they use negative log-likelihood of the predicted performance for meta-tasks as an auxiliary loss. The authors test the method on 4 different benchmarks and delineate its difference from previous work, demonstrating the relevance of their approach.\n\nSTRENGTHS:\n* They present a novel and effective method for pre-training a transformer to perform Bayesian optimization. The method is well-founded and all the parts make sense from an engineering perspective. For instance, the use of auxiliary losses and reinforcement learning appears as a very important and interesting approach.\n* They establish a difference from prior work in terms of some parameters such as history order invariance, F values, number of tokens, etc.\n* The authors provide a strong experimental protocol, comparing to relevant baselines in a broad set of datasets and benchmarks, which makes the empirical results very strong.\n\nWEAKNESSES:\n* The method might be overkill for small search spaces and nonexpensive functions. However, many black-box functions nowadays, such as deep neural networks, are expensive to evaluate, thus this approach might be highly relevant.\n* The authors do not present a comparison against time. How does the method perform in terms of time?\n\nQUESTIONS:\n* Why is there no confidence bar for the MIP dataset?\n\nLIMITATIONS:\nNo important limitation is detected.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 4, "DATE": "2023-07-03"}

comments: SUMMARY:
This paper proposed an end-to-end transformer-based framework for meta-Bayesian optimization (meta-BO). It formulated the meta-BO as a RL problem, defining a MDP in which a policy can be trained to solve the meta-BO problem. To help the training of RL algorithm, this paper also proposed an inductive bias via an auxiliary loss. The experiments had demonstrated the effectiveness of the proposed method.

STRENGTHS:
1. This paper deals with a good problem, which has a lot of applications in real world. The formulation of MDP is also reasonable.
2. This paper is well written, and also easy to follow.
3. The proposed inductive bias via an auxiliary loss Section 3.3. is insightful.

WEAKNESSES:
1. The experiments is insufficient. In the current experiments, the author only provided the evaluation results of the proposed method NAP and other baselines on several benchmarks. However, it would be better to do some ablation studies to further analyze NAP. For example, it should be important to investigate the effect of the auxiliary loss Eq(3).
2. The applications are limited. RL is known to be sample inefficient, especially when the state/action spaces are of high dimensionality. Furthermore, NAP utilized a transformer to extract historic information, which is also relatively hard to train. Therefore, NAP may be implicitly restricted into situations when x is of low dimension, limiting its applications.

QUESTIONS:
Please refer to weakness.

LIMITATIONS:
Yes.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2023-07-05

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe authors present a hyperparameter optimization method that is based on a transformer. In contrast to other recent work on HPO with transformers, this method does not rely on an acquisition function but instead trains an end-to-end model that outputs the acquisition scores directly.\n\nSTRENGTHS:\n**Diversity of benchmark tasks**\n\nExperiments on 4 different types of tasks, comparison against state-of-the-art methods. Demonstrate some improvements over baselines.\n\n**Clear Motivation**\n\nUse of transformers have shown promising results. End-to-end learning is always a good idea. The authors combine these previously existing ideas.\n\nWEAKNESSES:\n**Complexity of the Method**\n\nThe authors combine transformers, reinforcement learning and neural processes. There are most likely a lot of knobs and using this method effectively might be very difficult. Meta-BO is a prime example of a method that adds quite a bit of complexity and it turned out that it did not generalize to any other setup than considered by the authors.\n\n**Novelty**\n\nThis is neither the first paper to use transfer learning, transformers, RL, or end-to-end learning for HPO. This is a paper that combines all of it to a new method.\nA paper not discussed in this work: Bing-Jing Hsieh, Ping-Chun Hsieh, Xi Liu: Reinforced Few-Shot Acquisition Function Learning for Bayesian Optimization. NeurIPS 2021: 7718-7731\nIt also trains an end-to-end model and the authors demonstrated some improvements over FSBO.\n\n**Some Overclaiming of Contributions**\n\nIn multiple locations, the authors claim to present the \" first end-to-end training protocol for meta-BO\". This is not the case, see reference above and MetaBO.\n\nQUESTIONS:\nThis is a complex method that probably introduce several hyperparameters itself. How do you set them? How robust are they? Do you think there is a chance that this method can be directly applied to a different task or will it require significant work to get it running?\n\nWhy is NP-EI missing for HPO-B?\n\nLIMITATIONS:\nThe authors do not address the limitations that come with the complexity of this method, i.e., additional hyperparameters.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 5, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-05"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis work proposes an approach to meta-Bayesian optimization (BO) that features a single transformer neural process architecture for both the meta-surrogate function and meta-acquisition function, which are respectively trained via standard neural process maximum likelihood and model-free meta-reinforcement learning (in the style of RL^2) frameworks. The authors demonstrate the empirical benefits of such an approach over prior meta-BO methods on four suites of black-box optimization problems.\n\nSTRENGTHS:\n### Originality\nMy background is in meta-learning and reinforcement learning, not Bayesian optimization. It appears that the proposed approach is novel for meta-BO in that it makes minimal modelling choices, opting for a completely black-box neural process for the meta-surrogate function as opposed to e.g. a Gaussian process, and a completely black-box acquisition function as opposed to e.g. expected improvement (EI).\n\n### Quality\nThe empirical evaluation seems reasonably thorough (though again, I am not an expert in BO), with experiments based on multiple data sources and a healthy pool of competing methods. I also appreciated the described engineering effort in adapting previous methods to handle datasets for which they were not originally designed for.  \n\n### Clarity\nI'm not sure that Lemma 3.1 adds much substance in favor of the paper's design decisions. Instead, perhaps we can more loosely but intuitively argue that typical acquisition functions such as EI depend on the surrogate function's predictions of the underlying objective function (as it well should, to avoid no free lunch), but this dependency is dropped with a completely end-to-end black box acquisition function. The neural process meta-surrogate training can then be justified as softly reintroducing this dependency a la multi-task learning (multi in the sense of surrogate function modeling and acquisition function policy learning, not multiple BO tasks) via a shared architecture. \n\n### Significance\nI imagine the demonstrated empirical benefit of an end-to-end solution in a domain where handcrafted solution components have been standard will have considerable significance.\n\nWEAKNESSES:\n- A minor point -- it is unusual to see one maximizing a \"loss\" (Eq. 3 and Alg. 1).\n\nQUESTIONS:\n- How is your framing of the acquisition function as a policy with order-invariant processing of the current optimization trace related to ideas in context-based/prototype-based meta-reinforcement learning [A]? It seems like the former is a specific instance of the latter where the state space for each MDP is singleton. \n\n[A] Rakelly & Zhou et al., Efficient Off-Policy Meta-Reinforcement Learning via Probabilistic Context Variables, ICML 2019.\n\nLIMITATIONS:\nYes.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-07"}

comments: SUMMARY:
In this work, they developed the first end-end transformer based architecture for meta Bayesian Optimization and demonstrated empirically state-of-the-art regret minimization on hyperparameter optimization, antibody and chip design problems. They propose using a novel transformer architecture based Neural Process to learn the acquisition function and train it E2E compared to prior work in meta-learning acquisition functions. They identified logarithmic reward sparsity patterns in RL and introduced an auxiliary loss maximizing log-likelyhood of making the correct predictions on their labeled source task datasets as an inductive bias to stabilize training.

STRENGTHS:
Demonstrates strong empirical results on their tasks and ablation experiments are included in the appendix demonstrating the importance of their inclusions.

Proposes novel combination of transformer network and E2E learning of the acquisition function for Bayesian optimization with a new formulation and inclusion of auxiliary loss.

Well written and well positioned with existing work.

WEAKNESSES:
Somewhat limited novelty due to prior separately have utilized transformers for Bayesian optimization and other prior work having meta-learned acquisition functions.

Somewhat limited experimental evidence since no hyperparameter search was conducted on the proposed method or baseline. Results were only included on four tasks.

QUESTIONS:
Did you tune the weighting of the auxiliary loss?

How were the hyperparameters chosen for the implementation if no hyperparameter search was conducted.

LIMITATIONS:
Discussed limitations limiting to 5000 BO steps due to their use of the transformer network.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2023-07-09

comments: SUMMARY:
By combining a Transformer-based architecture and reinforcement learning objective, this paper proposes the first end-to-end framework for meta-Bayesian optimization.
The proposed framework is claimed to resolve the inefficiency in the existing two-stage approaches.

STRENGTHS:
- The meta-Bayesian optimization is an important topic that can be useful in various domains.
- The presented motivation appears to be compelling, while the proposition of employing a Transformer-based architecture seems reasonable.
- The experiments include practical problem settings.

WEAKNESSES:
- I am not sure if RL is really the optimal way to achieve an end-to-end meta-BO framework. I think this paper lacks a justification for why RL is a necessary component.
- Although this paper claims to tackle the inefficiencies from two-stage approaches, introducing RL makes training much harder. The authors add an auxiliary task to solve this issue, but due to this, the proposed method does not seem to reduce the complexity and the objective discrepancy in the previous two-stage approaches.

Please note that I do not have much expertise in the relevant fields.
I currently do not see any major flaw in this work, but I am also not fully convinced.
Comments from other reviewers and further discussion may largely change my score.

QUESTIONS:
- Is there any factor that necessitates the use of RL?

LIMITATIONS:
Yes
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2023-07-27