SlowGuess/NeurIPS_2023_OCR huggingface dataset:
We now introduce in more detail our transformer architecture and note some of its important properties. We call this architecture the neural acquisition process (NAP) because it is a new type of NPs jointly predicting AF values and distribution over actions. Similarly to other NP [11, 12], it takes a context and queried locations as input. We parameterise it by  $\theta$  and denote the acquisition prediction by  $\alpha_{\theta}(\pmb{x}^{(\mathrm{pred})},\mathcal{H},t,T)$ .
Since  $\alpha_{\theta}$  outputs real numbers, we still need to define how we can obtain a valid probability distribution over the action space to form a policy  $\pi_{\theta}(\cdot |s_t)$ . Defining such a distribution over the whole action space is hard if  $\mathcal{A}$  is continuous. Hence, similarly to Volpp et al. [13], we evaluate the policy  $\pi_{\theta}$  only on the finite set of locations  $\pmb{x}^{(\mathrm{pred})}$  for a given task<sup>1</sup>. Therefore, we have:
$$
\pi_ {\boldsymbol {\theta}} \left(\boldsymbol {x} _ {t} ^ {(\text {p r e d})} | \boldsymbol {s} _ {t}\right) \propto \frac {e ^ {\alpha_ {\boldsymbol {\theta}} \left(\boldsymbol {x} _ {t} ^ {(\text {p r e d})} , \mathcal {H} _ {t} , t , T\right)}}{\sum_ {i} ^ {n _ {\text {p r e d}}} e ^ {\alpha_ {\boldsymbol {\theta}} \left(\boldsymbol {x} _ {i} ^ {(\text {p r e d})} , \mathcal {H} _ {t} , t , T\right)}}.
$$
We now have all the necessary components to define the full objective combining Equation 1 and 3:  $\mathcal{J}(\pmb {\theta}) = J(\pi_{\pmb{\theta}}) + \lambda \mathcal{L}(\pmb {\theta})$  , where  $\lambda$  is a hyperparameter to balance between the two objectives. We summarise the full training algorithm in Alg. 1 detailing each of the components.
Finally, we study some desirable properties of NAPs, explain how we can achieve them, and why they are important in the context of BO.
Algorithm 1 Neural Acquisition Process training.
Require: Source tasks training data  $\{\mathcal{D}^{(k)}\}_{k = 1}^{K}$ , initial parameters  $\theta$ , budgets  $T^{(k)} \equiv T$ , discount factor  $\gamma$ , learning rate  $\eta$
for each epoch do
select task  $k$  and dataset  $\mathcal{D}^{(k)}$ , set  $\mathcal{H}_0 = \{\emptyset\}$
$x_{t}\sim \pi_{\pmb{\theta}}(\cdot |\pmb{s}_{t})$ $\triangleright$  predict action
$y_{t} = f^{(k)}(x_{t})$ $\triangleright$  execute action
$r_t = y_{<t}^*$ $\triangleright$  collect reward
$\mathcal{H}_{t + 1}\leftarrow \mathcal{H}_t\cup \{(x_t,y_t)\} \triangleright$  update hist.
[ R = \sum_{t=1}^{T} \gamma^t r_t ]  
 $\triangleright$  cumul. reward
$\mathcal{D}\rightarrow \overline{\mathcal{D}_{\mathrm{obs}}}\sqcup \mathcal{D}_{\mathrm{pred}}$  > split source data
$\mathcal{L} = p_{\pmb{\theta}}(\pmb{y}^{(\mathrm{pred})}|\pmb{x}^{(\mathrm{pred})}, \mathcal{D}_{\mathrm{obs}})$ $\triangleright$  aux. loss
$\pmb {\theta}\gets \pmb {\theta} + \eta (\nabla_{\theta}R + \nabla_{\theta}\mathcal{L})$  update  $\pmb{\theta}$
Property 3.2 (History-order invariance). An NP  $g$  is history-order invariant if for any choice of permutation function  $\psi$  that changes the order of the points in history,  $g(\pmb{x},\psi (\mathcal{H})) = g(\pmb{x},\mathcal{H})$ .
Unlike vanilla transformers, we do not use a positional encoding. It allows NAP to treat the history  $\mathcal{H}$  as a set instead of an ordered sequence [42] and to be invariant to the order of the history. To form an  $\langle x,y\rangle$  pair in this set, we sum the embedding of  $x$  and  $y$  [11], whereas for queried locations we only use the embedding of  $x$  for a token (see Fig. 1, left). It is important for BO since the order in which we collect the points is not relevant for assessing how promising is a new queried location. Previous meta-RL algorithms [43] also shown the importance of relying on order-invariance.
Table 1: We compare the properties of different transformer architectures.  $L$  is the number of tokens needed to encode the meta-data, and  $D$  denotes the dimensionality of  $\mathcal{X}$  .
History-order inv.	Query ind.	AF values	Tokens
NAP (ours)	✓	✓	✓	t + n_pred
TNPs [12]	✓	X	X	t + n_pred
OptFormer [41]	X	X	X	L + (D + 2)(t + n_pred)
PFN [11]	✓	✓	X	t + n_pred
<sup>1</sup>With a continuous action space, one could optimise  $\pmb{x}^{(\mathrm{pred})}$  to maximise the NAP outputs before forming the distribution over the discrete actions.
Property 3.3 (Query independence). A NP  $g$  is query independent if for any choice of  $n$  queried locations  $\pmb{x}^{(pred)} = (\pmb{x}_1^{(pred)},\dots,\pmb{x}_n^{(pred)})$ , we have  $g(\pmb{x}^{(pred)},\mathcal{H}) = (g(\pmb{x}_1^{(pred)},\mathcal{H}),\dots,g(\pmb{x}_n^{(pred)},\mathcal{H}))$ .
In NAP, every token in the history can access each other through the self-attention mask, whereas the elements of  $\pmb{x}^{(\mathrm{pred})}$  can only attend to themselves and to tokens in the history  $\mathcal{H}$  (see Fig. 1, right). Because the tokens inside  $\pmb{x}^{(\mathrm{pred})}$  cannot access each other and we do not use positional encoding, NAP is query independent, which is important to make consistent predictions of AF values for BO as they should not depend on the other queried locations. Additionally, NAP is fully differentiable enabling end-to-end training but also optimisation of the queried locations via gradient ascent for continuous action spaces. In Table 1, we highlight the differences with other state-of-the-art models regarding those properties.
Figure 1: Our proposed NAP architecture (left) and an example of the masks applied during inference (right). We apply independent embedding on  $\boldsymbol{x}_i, y_i, t$  and  $T$ . The colored squares mean that the tokens on the left can attend the tokens on the top in the self-attention layer.
We conduct experiments on hyperparameter optimisation (HPO) and sequence optimisation tasks to assess NAP's efficiency. Regarding HPO, we run our algorithm on datasets from the HPO-B benchmark [44] and in real-world settings of tuning hyperparameters for Mixed-Integer Programming (MIP) solvers. For sequence optimisation, we test NAP on combinatorial black-box problems from antibody design and synthesis flow optimisation for electronic design and automation (EDA).
Baselines We compare our method against popular meta-learning baselines, including few-shot Bayesian optimisation (FSBO) [5], MetaBO [13] as well as OptFormers [41] when applicable. Moreover, we show how classical GP-based BO, equipped with an adequate kernel  ${}^{2}$  and an EI acquisition function, which we title GP-EI, performs across domains. We first fit a GP on the metatraining datasets to enable a fair comparison. We initialise the GP model with those learnt kernel parameters at test time. This way, the GP baseline can benefit from the information provided in the source tasks. We also explore training a neural process directly on source task datasets and then using a fixed EI acquisition. For that, we introduce NP-EI that combines the same base architecture from [11] with EI. Additionally, we contrast NAP against random search (RS).
Following the standard practice in meta-BO, we report our results in terms of normalised regrets for easier comparison across all tasks. We attempt to re-implement all baselines across all empirical domains, extending previous implementations as needed, e.g., developing MetaBO [13] and FSBO [5] versions for combinatorial and mixed spaces to enable a fair comparison in MIP solver tuning, antibody and EDA design tasks.
Remark on OptFormer We re-run OptFormer with an EI AF on hyperparameter tuning tasks by extending the implementation in [41] to the discrete search space version of HPO-B. However, the lack of a complete open-source implementation and interpretable meta-data in the other benchmarks (e.g., in antibody design and EDA) prohibited successful execution.
2We adapt the GP kernel to each task depending on the nature of the optimisation variable, e.g., Matérn in continuous domains, categorical for combinatorial search space, and mixed when having both.
4.1 Hyperparameter Optimisation Results
Hyperparameter Optimisation Benchmarks We experiment on the HPO-B benchmark [44], which contains datasets of (classification) model hyperparameters and the models' corresponding accuracies across multiple types and search spaces. Due to resource constraints, we selected six representative search spaces. Nonetheless, we chose the search spaces to represent all underlying classification models in the experiment. We also always picked the ones with the least points to focus on the low data regime performance; see Appendix C.1 for more details. The results of our tests in Figure 2 demonstrate that NAP and OptFormer outperform all other baselines. Surprisingly, although NAP uses a much smaller architecture than OptFormer (around 15 million parameters vs 250 million for OptFormer) and trains on much less data (around 80k original points on average versus more than 3 million points for OptFormer), its regret performance is statistically similar to that of the OptFormer after 100 steps. Moreover, on the same GPU, to perform the same inference, NAP only uses  $2\%$  of OptFormer's compute time and around  $40\%$  of its memory usage.
Tuning MIP Solvers Apart from HPO-B, we consider another real-world example of hyperparameter tuning that requires finding the optimal parameters of MIP solvers. We use the open-source SCIP solver [45] and the Benchmark suite from the MIPLib2017 [46] that consists of a collection of 240 problems. The objective is to find a set of hyperparameters so that SCIP can solve MIP instances in minimal time. Our high-dimensional search space comprises 135 hyperparameters with mixed types, including boolean, integers, categories and real numbers. We train our model on data collected from BO traces on 103 MIPs and test on a held-out set of 42 instances. Our results in Figure 2 demonstrate that NAP is capable of outperforming all other baselines reaching low regret about an order of magnitude faster than FSBO [5]. Figure 2 further demonstrates the importance of end-to-end training where NAP again outperforms NP-EI.
4.2 Sequence Optimisation Experiments
Now, we demonstrate NAP's abilities beyond hyperparameter tuning tasks in two real-world combinatorial black-box optimisation problems.
Antibody CDRH3-Sequence Optimisation This experiment focuses on finding antibodies that can bind to target antigens. Antigens are proteins, i.e., sequences of amino acids that fold into a 3D shape giving them specific chemical properties. A protein region called CDRH3 is decisive in the antibody's ability to bind to a target antigen. Following the work in [47], we represent CDRH3s as a string of 11 characters, each character being the code for a different amino acid in an alphabet of cardinality 22. The goal is to find the optimal CDRH3 that minimises the binding energy towards a specific antigen. Binding energies can be computed using state-of-the-art simulation software like Absolut! [48]. We collected datasets of CDRH3 sequences and their respective binding energies (with Absolut!) across various antigens from the protein database bank [49]. We then formed a transfer scenario across antigens where we meta-learn on 109 datasets, validate on 16, and test NAP on 32 new antigens. Our results in Figure 2 indicate that NAP is not limited to hyperparameter tuning tasks but can also outperform all other baselines in combinatorial domains.
Electronic Design Automation (EDA) Logic synthesis (LS) is an essential step in the EDA pipeline of the chip design process. At the beginning of LS, we represent the circuit as an AIG (an And-Inverter-Graph representation of Boolean functions) and seek to map it to a netlist of technology-dependent gates (e.g., 6-input logic gates in FPGA mapping). The goal in LS is to find a sequence of graph transformations such that the resulting netlist meets an objective that trades off the number of gates (area) and the size of the longest directed path (delay) in the netlist. We perform a sequence of logic synthesis operators dubbed a synthesis flow to optimise the AIG.
Following [50], we consider length 20 LS flows and allow an alphabet of 11 such operators, e.g., {refactor, resub, ..., balance} as implemented in the open-source ABC library [51]. We collected datasets for 43 different circuits. Each dataset consisted of 500 sequences (collected via a Genetic algorithm optimizer) and their associated area and delay. Additionally, we applied the well-known heuristic sequence resyn2 on each circuit to get a reference area and delay. For this task, the black-box takes a sequence as input and returns the sum of area and delay ratios with respect to the reference ones, as detailed in Appendix B.1. We train all methods on 30 circuits from OpenABC [50], validate on 4 and test on 9. Our results in Figure 2 again demonstrate that NAP outperforms all other baselines by a significant margin.
Figure 2: Average regret vs. BO iterations with 5 initial points. (Left) Results on 6 search spaces on the HPO-B benchmark. (Middle-left) Results tuning SCIP for solving 42 different MIPs. (Middle-right) Antibody CDR3 sequence optimisation on 32 test datasets corresponding to 32 different antigens. (Right) Logic synthesis operator sequence optimisation on 9 test datasets corresponding to 9 different circuits. For each method, error bars show confidence intervals computed across 5 runs on HPO-B and 10 runs on all the others.
Meta-Learning Paradigms Meta-learning aims to learn how to quickly solve a new task based on the experience gained from solving or observing data from related tasks [15]. To achieve this goal, we can follow two main directions. In the first one, the meta-learner uses source data to learn a good initialisation of its model parameters, such that it is able to make accurate predictions after only few optimisation steps on the new target task [52]. The second one is more ambitious as it endeavours to directly learn, from the related tasks, a general rule to predict a helpful quantity to suggest the next point (black-box value, acquisition function value, etc.) from a context of observations (i.e. our history  $\mathcal{H}^{(k)}$ ). Works following that direction, which include ours, can rely on different types of models to learn this general rule, such as NPs [10], recurrent neural networks [53], HyperNetworks [16] or transformers [12, 11]. Orthogonal to these two directions is the learning of a hyper-posterior on the datasets to meta-train on. In that line of research, Rothfuss et al. [54] suggest the use of Stein Variational Gradient Descent (SVGD) to estimate uncertainty with multiple models better, and Hsieh et al. [14] extend the use of SVGD to meta-learn AFs. We consider those last two works as an orthogonal direction to our, as NAP could also benefit from SVGD updates.
Learning a Meta-Model Chen et al. [55] and TV et al. [56] train an RNN to predict what should be the next suggestion in BO instead of predicting the acquisition function value. We do not compare to their method given the lack of available implementation. Still, we compare to the outperforming approach developed by Chen et al. [41] based on a transformer architecture designed to do meta-BO on hyperparameter tuning problems. Their OptFormer is trained on related tasks over different search spaces to output the next point to evaluate and predict its value. The next point is sequentially decoded, one token at a time, and exploits the hyperparameters' names or descriptions to improve generalisation across tasks. Contrary to NAP, OptFormer is not designed to predict acquisition value at any point and does not meet the two properties 3.2 and 3.3, and therefore needs much more training data to predict the proper sequence of tokens. We note that NAP does not rely on variable descriptions, making it easily deployable on various tasks and still very competitive in the hyperparameter optimisation context.
Rather than learning an entire predictive model from scratch, prior works [5, 7-9] learn deep kernel surrogates, i.e. a warping functions mapping the input space with a neural network before it is given to a GP. Learning only the input transformation allows the authors to rely on the closed-form posterior prediction capacity of standard GP models. To perform the transfer, Feurer et al. [6] rely on an ensemble of GPs. Iwata [57] is an approach close to FSBO [5] as it learns a meta-model by meta-training a Deep Kernel GP. Notably, it does so in an end-to-end fashion using RL to propagate gradients through the acquisition function and the GP back to the deep kernel. It does not, however, learn a meta-acquisition.
Learning a Meta-Acquisition Function Hsieh et al. [14] and Volpp et al. [13] choose to perform transfer through the acquisition function. They use directly GP surrogates and define the acquisition as a neural network that is meta-trained on related source tasks. They first pre-train GP surrogates on all source tasks and fix their kernel parameters. They then rely on RL training to meta-learn the
neural acquisition function that takes as inputs the posterior mean and variance of the GP surrogate (which is itself trained online at test time). At test time, they allow for update in the GP but keep the weights of the neural acquisition fixed.
In summary, the methods meta-learning AFs do not do so in an end-to-end fashion and still rely on trained GP surrogates. While these methods are principled and competitive, they suffer from the cost of inverting the GP kernel matrix (cubic in the number of observations). In comparison, NAP can make predictions through a simple forward pass. On the other hand, the methods that learn a meta-model, either use a Deep Kernel GP (suffering the same cost) or have to learn a large model from scratch, costing a lot of budgets for collecting data beforehand, as well a pre-training time. Both of them use standard acquisition functions, missing the potential benefits of doing transfer in acquisition between tasks.
The performance of some of those algorithms on HPO-B is presented in Appendix-C showing that despite learning an architecture from scratch, NAP achieves a lower regret. For a more detailed survey on transfer and meta-learning in BO, we refer the reader to Bai et al. [4].
We proposed the first end-to-end training protocol to meta-learn acquisition functions in meta-BO. Our method predicts acquisition values directly from a history of points with meta-reinforcement learning and auxiliary losses. We demonstrated new state-of-the-art results compared to popular baselines in a wide range of benchmarks.
Limitations & Future Work Our architecture suffers from the usual quadratic complexity of the transformer in terms of the number of tokens which limits the budget of BO steps. Nevertheless, it can still handle around 5000 steps, which is enough for most BO scenarios. Another limitation of our architecture is that we need to train a new model for each search space. In future, we plan to enable our method to leverage meta-training from multiple search spaces and investigate how we could design data-driven BO-specific augmentation to further mitigate meta-overfitting [58].
We would like to thank Prof. Frank Hutter and his team for their constructive feedback and for the availability of their benchmark code and datasets. We also would like to thanks Massimiliano Patacchiola for his comments during the writing phase of the paper.
This work is supported by the CSTT on Generalisable Robot Learning via Machine Learning Models 2100332-GB and by the 2030 "New Generation of AI" - Major Project of China under grant No. 2022ZD0116408.
[1] Jasper Snoek, Hugo Larochelle, and Ryan P. Adams. Practical bayesian optimization of machine learning algorithms. In Peter L. Bartlett, Fernando C. N. Pereira, Christopher J. C. Burges, León Bottou, and Kilian Q. Weinberger, editors, Advances in Neural Information Processing Systems 25: 26th Annual Conference on Neural Information Processing Systems 2012. Proceedings of a meeting held December 3-6, 2012, Lake Tahoe, Nevada, United States, pages 2960-2968, 2012. URL https://proceedings.neurips.cc/paper/2012/bit/05311655a15b75fab86956663e1819cd-Abstract.html.
[2] Nate Gruver, Samuel Stanton, Polina Kirichenko, Marc Finzi, Phillip Maffettone, Vivek Myers, Emily Delaney, Peyton Greenside, and Andrew Gordon Wilson. Effective surrogate models for protein design with bayesian optimization. In ICML Workshop on Computational Biology, 2021.
[3] Fabio Muratore, Christian Eilers, Michael Gienger, and Jan Peters. Data-efficient domain randomization with bayesian optimization. IEEE Robotics and Automation Letters, 6(2): 911-918, 2021.