FunBO: Discovering Acquisition Functions for Bayesian Optimization with FunSearch
Authors: Virginia Aglietti
Ira Ktena
Jessica Schrouff
Eleni Sgouritsa
Francisco Ruiz
Alan Malek
Alexis Bellot
Silvia Chiappa
Venue: ICML 2025 poster (2025)
Keywords: Bayesian Optimization
LLM
Acquisition Functions
Primary area: deep_learning->large_language_models
TL;DR: We propose FunBO, an LLM-based method that can be used to learn new acquisition functions for Bayesian Optimization that perform well across a variety of experimental settings.

Abstract: The sample efficiency of Bayesian optimization algorithms depends on carefully crafted acquisition functions (AFs) guiding the sequential collection of function evaluations. The best-performing AFs can vary significantly across optimization problems, often requiring ad-hoc and problem-specific choices. This work tackles the challenge of designing novel AFs that perform well across a variety of experimental settings. Based on FunSearch, a recent work using Large Language Models (LLMs) for discovery in mathematical sciences, we propose FunBO, an LLM-based method that can be used to learn new AFs written in computer code by leveraging access to a number of evaluations for a limited set of objective functions. We provide the analytic expression of all discovered AFs and evaluate them on various global optimization benchmarks and hyperparameter optimization tasks. We show how FunBO identifies AFs that generalize well both in and out of the training distribution of functions, thus outperforming established general-purpose AFs and achieving competitive performance against AFs that are customized to specific function types and are learned via transfer-learning algorithms.

FunBO: Discovering Acquisition Functions for Bayesian Optimization with FunSearch
Virginia Aglietti Ira Ktena
Jessica Schrouff" Alexis Bellot
Eleni Sgouritsa
Francisco J. R. Ruiz Alan Malek
Silvia Chiappa
Abstract
in robotics (Calandra et al., 2016) and recommending new molecules in drug design (Korovina et al., 2020). Two main components lic at the heart of any BO algorithm: surro- gate model and an acquisition function (AF). The surrogate model expresses assumptions about the objective function, e-g., its smoothness, and it is often given by Gaussian Process (GP) (Rasmussen & Williams, 2006). Based on the surrogate model, the AF determines the sequential collec- tion of function evaluations by assigning score to potential observation locations. BO's 's success heavily depends on the AF'S ability to efficiently balance exploitation (i.e. assigning high score to locations that are likely to yield optimal function values) and exploration (i.e. assigning a high score to regions with higher uncertainty about the objective func- tion in order to inform future decisions), thus leading to the identification of the optimum with the minimum number of evaluations.
The sample efficiency of Bayesian optimization algorithms depends on carefully crafted acquisi- tion functions (AFS) guiding the sequential collec- tion of function evaluations. The best-performing AFS can vary significantly across optimization problems, often requiring ad-hoc and problem- specific choices. This work tackles the chal- lenge of designing novel AFS that perform well across variety of experimental settings, Based Funsearch, recent work using Large Lan- guage Models (LLMS) for discovery in mathemat- ical sciences, we propose FunBO, an LLM-based method that can be used to learn new AFS written in computer code by leveraging access to a num- ber of evaluations for a limited set of objective functions. We provide the analytic expression of all discovered AFS and evaluate them various global optimization benchmarks and hyperparam- eter optimization tasks. We show how FunBO identifies AFS that generalize well both in and out of the training distribution of functions, thus out- performing established general-purpose AFS and achieving competitive performance against AFS that customized specific function types and are learned via transfer-learning algorithms.
Existing AFS aim to provide either mization strategies or approaches tailored to specific ob- jective types. For example, Expected Improvement (EI) (Saltanis, 1971; Mockus, 1974), Upper Confidence Bound (UCB) (Lai & Robbins, 1985) and Probability of Improve- ment (Pofi) (Kushner, 1962; 1964) are all widely adopted general general-purpose purpose AFS that can be used out-of-the-bool across
opti-
BO algorithms and objective functions. The performance of these AFS varies significantly across different types of black-box functions, making the AF choice ad-hoc, empir- ically driven, decision. There exists extensive literature alternative AFS outperforming EI, UCB and Poft, for in- stance entropy- (Wang & Jegelka, 2017) or knowledge- gradient (Saltanis, 1971; Frazier et al., 2008) optimizers, see Garnett (2023, Chapter 7) for review. However, while these functions are often interpretable, they are generally hard implement and expensive evaluate, partly defeat- ing the purpose of replacing the expensive original optimiza- tion with the optimization of much cheaper and faster to evaluate AF. In order to avoid the limitations of current Ass, several works have proposed self-adjusting the hyper- parameters known AFS in a data driven way throughout the optimization process (Benjamins et al., 2023; Ding et al., 2022), combining different AFS a portfolio and selecting them via online multi-arma armed bandit strategy (Hoffman
1. Introduction
Bayesian optimization (BO) (Kushner, 1962; 1964; Mockus, 1974; Jones et al., 1998) is methodology for optimizing complex and expensive-to-evvllute black-box functions that emerge in many scientific disciplines. Bo has been used across a wide variety of applications ranging from hyperpa- rameter tuning in machine learning (Bergstra et al., 2011; Snoek et al., 2012; Cho al., 2020) to designing policies
Now at GlaxoSmithKline. 'Google DeepMind, London, UK. Correspondence to: Virginia Aglictti <aglictti @ google.com>.
Proceedings of the 4nnd International Conference Machine Learning. Vancouver, Canada. PMLR 267, 2025. Copyright 2025 by the author(s).

---

FunBO: Discovering Acquisition Functions for Bayesian Optimization with FunScarch
al., 2011) or computing Pareto front over different AFS (Lyu et al., 2018). Other prior works (Hsieh al., 2021; Volpp et al., 2020; Wistuba & Grabocka, 2021) have instead proposed representing AFS via neural networks thus bypass- ing the need for an analytical representation and learning new AFS tailored to specific objectives by transferring infor- mation from set of related training functions via, e.g., rein- forcement learning or transformers. While such learned AFS can outperform general- purposse AFS, their generalization performance to objectives outside of the training distribution is often poor (see experimental section and discussion on generalization behavior in Volpp et al. (2020)). More re- cently, the concurrent work of Yao et al. (2024) investigated representing AFS in code for specific optimization settings where the experimentation budget is limited.
flows with minimal overhead; (iii) leveraging foundational knowledge embedded in LLMS. Having been trained on vast corpora of code and text, these models possess knowl- edge about both programming constructs and existing BO strategies, which FunBO utilizes to efficiently explore the program space and construct effective heuristics.
We extensively test FunBO on a range of optimization prob- lems including standard global optimization benchmarks and hyperparameter optimization (HPO) tasks. For each experiment, we report the explicit functional form of the discovered AFS and show that they generalize well to the optimization of functions both in and out of the training distribution, outperforming general-purposs AFS while com- paring favorably to function-speeifie ones.
Defining methodologies that automatically identify new AFs capable of outperforming general-purposs -prpose and function- specific alternatives, both in and out of the training distri- bution, remains a significant and unaddressed challenge. In this work we tackle this challenge by considering AFS rep- resented in computer code. Learning new AFS expressed in code presents three main difficulties: (i) the vast space of all possible programs makes exhaustive search infeasi- ble, (ii) efficiently exploring constrained space of possible programs requires scalable methods and (iii) there is no clear criterion for ensuring the validity and effectiveness of generated AFS.
2. Preliminaries
We consider an sspensive-to-evaluate black-box function f X' + R over the input space X C R2d for which we aim to identify the global minimum x = arg min fxx). We assume access to a set of auxiliary black-box and expensive- to-evaluate objective functions, 9 = {oy 1 with g,
X3 + TR, X, S TRd, for j = 1,....,, J, from which we can obtain a set of evaluations.
Bayesian optimization. Bо seeks to identify x² with the smallest number T of sequential evaluations of f given N initial observations D { Be with y/: = fxx,). BO relies on a probabilistic surrogate model for which in this work is set to a GP with prior distribution over any batch of input points X = {zI,..., .NN given by x(] = (M(m( (m)(),, ). Ko(X, X')) with prior mean mx) and kernel Ko X') with hyperparameters A. The pos- terior distribution p(f(D) is available in closed form via standard GP updates. At every step t in the optimization process, BO selects the next evaluation location by optimiz- ing an AF (--ID;) X + I, given the current posterior distribution p( (D:). with Dj denoting the function evalu- ations collected up to trial t (including D). A commonly used AF is the Expected Improvement (EI), which is de- fined as am (x) = (v* (=== + (a: (z)), where y² denotes the best function value observed in D., also called incumbent, = (y* (x)))))))/o (x]D)), & and & are the standard Normal density and distribution functions, and m(æ and a (2 mean and standard deviation computed at x e X. Other general -prrpose AFS proposed in the literature are: UCB (Orca (ax = m(x (z with hyperparame- ter 88, Poft (Oren (ar &0yy" m(= (D)))))))/a(a (D.)) and the posterior mean (MMAN(x/D2) = m (n(x (D)) (denoted
Contributions. We overcome these difficulties by formulat- ing the problem of learning novel AFS written in computer code an algorithm discovery problem and address it by ex- tending FunSearch (Romera-Paredee et al., 2023), recently proposed algorithm solving open problems in mathematical sciences via LLMS.. In particular, we introduce FunBO, a novel method that explores the large space of AFS written in computer code by taking an initial AF as input and, with a limited number of evaluations for a set of objective func- tions, iteratively modifying it to improve the performance of the resulting BO algorithm. We focus on Python programs but FunBO can be readily applied to languages supported by FunSearch.
Unlike existing algorithms, FunBO outputs code snippets corresponding to improved AFS. This approach offers sev- eral advantages: (i) interpretability, as the code-based AFS can be directly inspected, analyzed, and understood. This allows for clear comprehension of the logic behind the optimization strategy employed by the discovered AF, which contrasts with neural network-based AFS for which it is of- ten difficult to understand why they perform well or how they balance exploration and exploitation; (ii) deployability and simplicity. The APS discovered by PunBO are output as concise code snippets (e.g., Python functions) that can be readily integrated into existing Bo libraries and work-
(æx (D.) are the GP posterior
'e focus on noiseless observations but the method can be equivalently applied to noisy outcomes.
2

---

FunBO: Discovering Acquisition Functions for Bayesian Optimization with FunScarch
by MEAN h
the problem at hand (also called "problem specification") and scoring each h(-) according a predefined performance metric, e.g., the number of bins used in h(-)). The inputs of both h(-) (denoted by h hereinafter) and e(h. -) (denoted by e hereinafter), are problem specific. A description of h's inputs is provided in the function's 's docstring' together with an explanation of how the function itself is used within e.
Unlike general-purgee pprepose AFS, several works have proposed increasing the efficiency of BO for a specific optimization problem, say the optimization of f. by either udaptively se- lecting and/or adjusting known AFS in a data-driven manner (Benjamins et al., 2023) or by learning problem-specine AFS (Hsieh et al., 2021; Volpp et al., 2020: Wistuba & Grabocka, 2021). The learned AFS are trained on the set G. whose func- tions are assumed to be drawn from the same distribution or function class associated with ʃ. reflecting a meta-learning setup. "Function class" here refers to a set of functions with a shared structure and obtained by, e.g. applying scaling and translation transformations to their input and output values or evaluating the loss function of the same machine learning model, e.g., AdaBoost, on different data sets. For instance, Wistuba et al. (2018) learns AF that is weighted super- position of ETs by exploiting access to sufficiently large dataset for functions in G. Volpp et al. (2020) considered settings where the observations for functions in G are lim- ited and proposed MetaBO, a reinforcement learning based algorithm that learns a specialized neural AF, i.e., a neural network representing the AF. The neural AF takes as inputs set of potential locations (with a given d), the posterior mean and variance at those points, the trial t and the budget T and is trained using proximal policy optimization algo- rithm (Schulman et al., 2017). Similarly, Hsieh et al. (2021) proposed FSAF, an AF obtained via sew-shot adaptation of a learned AF using a small number of function instances in G.
Given these initial components, FunSearch prompts an LLM to propose an improved h, scores the proposals inputs, e.g., on different bin bin-packing instances, and adds them to a programs database. The programs database stores correct h functions* together with their respective scores. In order to encourage diversity of programs and enable explo- ration of different solutions, a population-based approach inspired by genetic algorithms (Tanese, 1989) is adopted for the programs database (DB). At a subsequent step, functions in the database are sampled to create new prompt, the LLM's proposals are scored and stored again. The process repeats for = I,..., T until a time budget T is reached and the heuristic with the highest score on a set of inputs is returned.
set of
3. FunBO
FunBO is a FunSearch- method for discovering novel AFs that increase BO efficiency by exploiting the set of aux- iliary objectives G. In particular, FunBO (i) uses the same prompt and DB structure as FunSearch, but (ii) proposes a new problem specification by viewing the learning of AFS as algorithm discovery problem, and (iii) introduces a novel initialization and evaluation pipeline that is used within the FunSearch structure. FunBO does not make assumptions about similarities between f and G, nor does it assume ac- cess to a large dataset for each function in g. Therefore, FunBO can be used to discover both general-purpose and function-specifiic AFS as well as to adapt AFS via few-shots. FunBO leverages the LLMS' ability to generate executable code to make the search for novel AFS automatic and scal- able, potentially leveraging the extensive LLMS" knowledge of Bo and AFs while delivering more interpretable AFs than those represented by neural networks. Furthermore, while FunSearch was only applied to problems that required evoly- ing functions with simple inputs (integers, floats or short tuples; with only one application taking as input a single ar- ray), FunBO explores a significantly more complex function space where programs take inputs multiple arrays. This demonstrates how the same formulation can be applied to problems of increasing complexity as long as an appropriate scoring mechanism is identified.
Note that, while general-purposs AFS are used to opti- mize objectives across function classes, learned AFS aim to achieve high performance for the single function class to which f and & belong. See Section A for an extensive discussion of related work.
Funsearch. FunSearch (Romera-Paredes et al., 2023) is a recently proposed evolutionary algorithm for searching in the functional space by combining& a pre-trained LLM used for generating new computer programs with efficient evaluator, which guards against hallucinations and scores fitness. An example problem that FunSearch tackles is the online bin packing problem (Coffman et al., 1984), where a set of items of various sizes arriving online needs to be packed into the smallest possible number of fixed size bins.
A set of heuristics has been designed for deciding which bin to assign incoming item to, e.g., "first fit." FunSearch aims to discover new heuristics that improve on existing ones by taking as inputs: (i) the computer code of an evolve function h(()) representing the initial heuristic to be improved by the LLM, c.g., "first fit" and (ii) an evaluate function e(h, -), also written in computer code, specifying
'wide variet on Python programs. "The definition of correct function is also problem specific. For instance, a program can be considered correct if it compiles.
We focus on AFS that can be evaluated in closed form given the posterior parameters of a GP surrogate model and exclude those whose computation involves approximations.
3

---

FunBO: Discovering Acquisition Functions for Bayesian Optimization with FunScarch
h, h.Gnny
Inputs: Gn. Gv, Nos. B.T
Evalustion
Sctup: Initialize is (Fig. 6). e (Fig. 7-8) and DB with Non islands. Assign
Protrained LLM
BD
each island.
W
while < T do
3D
City
1. Sample two programs from DB and create prompt (Fig. 2, right)
W
2. Get a batch of B samples from the LLM
3. For each correct h" in the batch compute sa- (9T:)
4. Add correct h" to DR and update it (see Appendix C)
5. Update step r = 7
Prompe
end
T
Proprara dititteee
Output: Return & in DB with score in the top 20th percentile for Gr. and highest score on Gv.
Lng angmasAAcao (ooh) (5h) st. top
OkpunDO
percentle
Figure 1. Left: The FunBO algorithm. Right: Graphical representation FunBO. The different FunBO component w.r.t. FunSearch Pareers-Paredes et al., 2023, Fig. 1) are highlighted in color.
Method overview. FunBO sequentially prompts LLM to improve initial AF expressed in code so to enhance the performance of the corresponding BO algorithm when optimizing objectives in G. At every step T of FunBO, an LLM's prompt is created by including the code for two AF instances generated and stored in a programs database (DB) at previous iterations. With this prompt, a number (B) of alternative AFS are sampled from the LLM and are evaluated based their average performance which acts as training dataset. The evaluation process for an AF, say h" at step t, on G1r gives a numeric score 5y- (G,,) that is used to store programs in DB and sample them for subsequent prompts. The "process" of prompt creation, LLM sampling. and AF scoring and storing repeats until a time budget T is reached. Out of the top performing" AFS on GT. the algorithm returns the AF performing the best, on average, in the optimization of Gv = (TG which acts as a validation dataset. When no validation functions are used (g = -GT)), the AF with the highest average performance on Gr. is returned. Each FunBO component highlighted in bold is described below more detail, along with the complete algorithm and graphical representation in Fig. 1. We denote the AF returned by FunBO as O'uunBO
defines different optimization strategies, see for instance the AF generated for one of our experiments in Fig. 3 (left).
Prompt. At every algorithm iteration, a prompt is con- structed by sampling two AFS, h; and hj, previously gen- crated and stored in DB. h2 and hy are sampled from DB in a way that favors higher scoring and shorter programs (see paragraph below for more details) and are sorted in the prompt in ascending order based on their scores $n, (GT.) and sh, (Grx), see the prompt skeleton' in Fig. 2 (bottom). The LLM is then asked to generate new AF representing an improved version of the last, higher scoring, program.
subset Gr C G,
Evaluation. As expected, the evaluation protocol is critical for the discovery of appropriate AFS. Indeed, a scoring mech- anism that captures small improvements in the proposed AF is needed to steer the I.L.M toward promising regions of the function space. Our novel evaluation setup, unlike the one used in FunSearch, entails performing a full Bo loop to eval- uate program fitness. In particular, each function generated by the LLM is (i) checked to verify it is correct, i.c., it com- piles and returns a numerical output: (ii) scored based on the average performance of BO algorithm using h" as an AF Gn. Evaluation is performed by running a full bo loop with heT for each function gj E GT. and computing a score that contains two terms: a term that rewards AFs finding values close to the true optimum, and a term that rewards AFS finding the optimum in fewer evaluations (often called
Initial AF. FunBO's initial program h determines the input variables that can be used to generate alternative AFS while imposing a prior on the programs the LLM will generate at successive steps. For these reasons it is important for guid- ing the search process effectively. We consider an initial acquisition function that takes the functional form of the EI and has inputs the union of the inputs given EI, UCB and Poft (see code in Fig. 6). The AF returns an integer representing the index of the point in vector of potential locations that should be selected for the next function evalu- ation. All programs generated by the LLM share the same inputs and output, but vary in their implementation, which
'We explored using random selection of initial points as an alternative to Fl. However, this approach did not yield good results as using a random selection was incentivizing the generation of functions with a stochastic output, for which convergence results are not reproducible.
"When 7 1. only the initial program will be available in DB thus the prompt in Fig. will be simplified by removing acquisit iion funct ion_vl v1 and replacing v_2 with v v_1.
In this work we consider the programs with a score in the top 20th percentile.
4

---

FunBO: Discovering Acquisition Functions for Bayesian Optimization with FunScarch
"""Iprrae Bayesian Optimization by discovering a new
1.0
EI
Poft
FunBO
acquisition function."".
0.8
Mean
Random
R,
UCB
0.6
def couuussttion_fucttiinnn winnetiion.W(prddicttiie mean, predictive incumbent, beta=1.0);
0,4
"""Returns the index of the point to collect
0.2
(Full docstring in Fig. 8)"""
0.0
Code for lonest-scoring sampled AF return
10
20
30
40
50
Trials
def wtt.issttoooffuuntiiooo (prrediictive (ereeeettooo.o.reeiiiiieeeeeeee mean, predictive incumbent, beta=1.0);
def
mean,
predictive_var incunbent. beta=1.0):
"." Improved version of "acquisition. fun """ Cade for scoring sampled AF.
"""Returns the index of the point to collect
(Full docstring in Fig. 8). = (incumbent predictive_nean
return
sprddprrecttvveeevar
def acquisition incumbent, beta=1.0):
nean, predictive
predictive_ste mp..qrt (predictive__ar)
vals = (incumbent predictive_
.." Improved version of the previous
norm.cdf(2) predictive norm.
iequisition._uuetiinn uee
return np. avggaa
Left: FunBO prompt includes two previously generated AFs which are sampled from DB and are sorted in ascending order based on the score achieved Gr- The generates third AF, acquisi fit_tfonntttttiiteeeeeeee wwwfffnnnnnvv representing an improved version of the highest scoring program. Right: OOD-Bench. Code for crraao (bottom) and average BO performance when using general purpose AFS and Oraano (top). Shaded area gives std. dev./2. The line gives R. i.. zero average regret.
Figure 2.
trials). Specifically, we use the score Sh (GTo) defined as
the evaluation process is presented Appendix B.
gy(m),,n)) y1 9 g
+ (1 Thg ] (1)
Programs database. Similar to FunSearch, scored AFS are added DB, which keeps population of correct programs following island model (Tanese, 1989). DB is initialized with number Nop of islands that evolve independently. Sampling of hę and hy from DB is done by first uniformly sampling an island and, within that island, sampling pro- grams by favouring those that are shorter and higher scoring. A new program generated when using hG and hy in the prompt added to the same island and, within that, a cluster of programs performing similarly on GTo. see Ap- pendix C for more details.
GT
where, for each Ij. 2/ the ground truth optimal value, a/=0 gives the optimum found at 1 08, 25, is the found optimum for gj with h" and Thy gives the number trials out 7 that h" selected before reaching Vž (if the optimum for function gy was not found, then Thy T indicate that all available trials have been used). The first term in the square brackets of Eq. (1) quantifies the discrepancy between the function values at the returned optimum and the true optimum. This term becomes zero when "j,,- equals 2/ indicating a failure to explore the search space. Con- versely, if he successfully identifies the true optimum, such that gj (a) )..-)= ur. this term reaches its maximum value of one. The second term in Eq. (1) captures how quickly h" identifies vf. When Th3 = T. indicating the algorithm has not converged, this term becomes zero, and the score is solely determined by the discrepancy between the discov- ered and true optimum. If, instead, the algorithm reaches the global optimum, this term represents the proportion of trials, out of the total budget T, needed to do so. Code for
4. Experiments
Our experiments explore FunBO's ability to generate novel and efficient AFS across a wide variety of settings. par- ticular, we demonstrate its potential to generate AFS that generalize well to the optimization of functions both in distribution (ID, i.c. within function classes) and out of dis- tribution (OOD, i.e. across function classes) by running three different types of experiments:
"This is the input value corresponding to the optimal value found among the of initial observations D. We assume this to he different from the ground truth optimum.
1. OOD-Bench tests generalization across function classes by rumning FunBO with & containing different standard global optimization benchmarks and testing on a set F
5

---

FunBO: Discovering Acquisition Functions for Bayesian Optimization with FunScarch
Styblinskytang function
FunBO
might include functions with different scales, we normalize the regret values to be in [0, 1] before averaging them. Note that the mean and standard deviations shown in all regret plots represent the performance variation across the set of test functions, not across multiple independent runs of a BO algorithm on a single function instance. For instance, for OOD-Bench, the mean and standard deviation Fig. 2 (right, top) are over the 9 distinct test functions.
UCB
150
100
100
50
50
-4
2
150
0
To isolate the effects of different AFS, we employ the same experimental setting across all methods in terms of: (i) the number of trials T; (ii) the hyperparameters of the GP sur- rogate models (tuned offline); (iii) the evaluation grid for the AF, which is set to be a Sobol grid (Sobol', 1967) on the input space; and (iv) the initial design, which includes the input point yielding the maximum function value on the grid. Note that use GP model with zero mean function and RBF kernel across experiments. Therefore, the discov- ered AFs are conditioned on this choice of surrogate model. We acknowledge that while this setup ensures a consistent comparison across all AFS by removing confounding effects (e.g., from the quality of hyperparameter optimization rou- tines), it might deviate from fully realistic deployment scenario. For this reason, in Section D.6, we included comparison of the AFS found by FunBO for OoD-Bench when evaluated using the standard Bo pipeline available in Bo' BoTTorc, which employs default settings for aspects such as AF optimization, GP hyperparameter optimization, and the random selection of initial points.
100
50
Weierstrass function
-0.4 -0.2
0.0
0.2
0.4
Figure 3. OOD-Bench. Different AFS trading-off exploration and exploitation for two mne-iimeniional objective functions (green lines). Blue and gray trajectories track the points queried by Crrunnn. Et and UCB over 150 trials (right y-axis).
that similarly comprises diverse functions in terms of smoothness, input ranges and dimensionality and output magnitudes. We do not scale the output values nor nor- malize the input domains to facilitate learning. but rather use the objective functions as available in standard BO packages out-of oottoof-he-beee tet-the-bos In this case G and F do not share any particular structure, thus the generated AFS are closer to general-purpose. AFS.
2. ID-Bench, HPO-ID and GPs-ID test FunBO-generated AFS within function classes for standard global optimization benchmarks, HPO tasks, and general function classes, re- spectively. As this setting is closer to the one considered by meta- weea-learnnng approaches introduced in Section 2, we compare FunBO against MetaBO (Volpp et al., 2020),' 9 the state-of-the-art transfer AF,
All experiments are conducted using FunSearch with de- fault hyperparameters! unless otherwise stated. We employ Codey, an LLM fine-tuned on large code corpus and based on the PaLM model family Paggel-PaLM---Teee 2023), to generate AFs.! 12
OOD-Bench. We test FunBO's capabilities to generate AFS that perform well across function classes by including the one-dimensional functions Ackley, Levy, and Sehwefel in Gr, and using the one-dimensional Rosenbrock function for Gv. We test the resulting O'runna on nine very differ- ent objective functions: Sphere (d = 1). Styblinski Tang 1), Weierstrass (d = 1), Beale (d = 2), Branin (d = 2), Michalewicz (d = 2), Goldstein-Prrn
3. FEW-SHOT demonstrates how FunBO can be used in the context of few-shot fast adaptation of an AF, In this case, the AF is learned using a general function class as 9 and is then tuned, using a very small (5) number of examples, to optimize a specific synthetic function. We compare our approach Hsieh et al. (2021), 10 the most relevant few-shot learning method.
(d = 2) and Hart- mann with both d 3 and d = 6. We do not compare against MetaBO as (i) was developed for settings in which the functions in 9 and F belong to the same class and, (ii) the neural AF is trained with evaluation points of given dimension, thus it cannot be deployed for the optimization of functions across different d. For completeness, we re-
We report all results in terms of normalized average simple regret on a test set, R2 as a function of the trial t. For an objective function f. this is defined as RS = =f - y" where y" is the ground truth optimal value and 2; is the best selected point within the data collected up to t. As F
"See code at hethubboommgooggeeeeemm////uussu "Codey is publicly accessible via its API (Vertex AI, 2023). For AF sampling, we used 5 Codey instances running on tensor processing units on computing cluster. For scoring, we used 100 CPU evaluators per LLM instance.
implementation at 10We used the author-provided implementation at githuh.com/inghssef/fAA..
'W used the author-provided fethub.com/hoschheccrrhhe/eeeeee
6

---

FunBO: Discovering Acquisition Functions for Bayesian Optimization with FunScarch
Branin
Goldstein Price
Hartmann
0.8
0.8
0.8
MetaBO
UCB Polt
FunBO A 0
MetaSO
UCB Pon
FuniB fl 0
MenaBO
UCB
hunBO
0.7
0.7
0.7
Pon
f
0.6
Meon
Random
0.6
Mean
Random
0.6
Meon
Random
0.5
0.5
0.5
x 0.4
/ 0.4
2 0.4
0.3
0.3
0.3
0.2
0.2
0.2
0.1
0.1
0.1
0.0
0.0
0.0
2
8
10
20
25
10
15 Trials
25
30
Trials
Trials
Figure 4. ID-Bench. Average BO performance when using general- purpose AFS (gray lines), the AF learned by MctaBO (black dashed line) and Arrano (blue line) on 100 function instances. Shaded area gives + std. dev./2. The red line represents Rt 0, i.e. zero average regret.
port comparison with dimensiionllit--gnnoti version of MetaBO in Appendix D.I (Fig. 9) together with all exper- imental details, e.g., input ranges and hyperparameters.
in the literature (Perrone al., 2019; Li et al., 2018).
ID-Bench. Next we evaluate FunBO capabilities to gen- erate AFs that perform well within function classes using Branin, Goldstein-Priet and Hartmann (d =0 3). For each of these three functions, we train both FunBO and MetaBO with |g] = 25 instances of the original function obtained by scaling and translating it with values in [0.9, 1.1] and [-0.1, 0.1]4 respectively. 13 For FunBO we randomly assign 5 functions in G to Gy and keep the rest in GT.- We test the performance of the learned AFS on another 100 instances of the same function, with randomly sampled values of scale and translation from the same ranges. We additionally compare against BO algorithm that uses EI, UCB, Poft, MEAN or a random selection of points. All tlyee-paramette settings for this experiment are provided in Appendix D.2. Across all objective functions, CyymBo leads to convergence performance that outperforms general-purpose AFs (Fig. 4), More importantly, despite using the same inputs of El or UCB, FunBO is able to reach performances that are compa- rable or superior to those of APS that are parameterized by neural networks and use additional inputs (Fig. 4). In terms of interpretabilityy notice how the AF for Goldstein-Price thus
AF interpretation: In this experiment, Orunso (Fig. 2, right bottom plot) represents a combination of EI and UCB which, due to the beta xpredi ve_std term, is more ex- ploratory than El but, considering the incumbent value, still factors in the expected magnitude of the improvement and reduces to EI when beta=0. This determines the way O'unnao trades-off exploration and exploitation which can be visualized by looking at the "exploration path", i.e., the sequence of x values selected over f, as shown in the right plots of Fig. (l measured on the secondary y-axis). For objective functions that are smooth, for example Styblinski- Tang, the exploration path of Orunnao matches those of and overlapping). In this
UCB (top plot, note that trajectories scenario, all AFS exhibit similar behavior, converging to x (red vertical line) with fewer than 25 trials. When instead the objective function has a lot of local optima (bottom plot) as in Weierstrass, both EI and UCB get stuck after a few trials while FunBO continues to explore the search space eventually converging to x". Notice how in this plot the convergence paths of all AFS differ and only the blue line aligns with the red line, i.e., converges to x", after few trials.
(Fig. 12) can be written as o² ( Pp P giving a modified Poft where the probability observing an improvement over the incumbent is multiplied by the predictive variance.
Using OFunBo to optimize the nine functions in F leads to a fast and accurate convergence to the global optima (Fig. 2, right top plot). This is confirmed when extending the test set include 50 scaled and translated instances of the functions in F (Fig. 9, right). Finally, note how the top right plot in Fig. 2 shows surprisingly good performance of random search. This is due to random search perform- ing competitively on functions with numerous local optima, which are generally harder to optimize. Aggregating per- formance across all functions in F highlights that no single known general-pun purposse AF consistently outperforms the oth- ers. This aligns with the well-established understanding that the effectiveness of AFS can vary significantly across black-box functions and is consistent with findings reported
The AFs found in this experiment (code in Figs. 11-13) are "customized" to given function class thus being closer, in spirit, to the transfer AF. However, in order to further validate the generalizability of O'uunno found in OOD-Bench, we tested this AF across instances of Bramin, Goldstein and Hartmann (Fig. 10, green line). We found it to perform well against general-purposs AFS thus confirming the strong results observed in OOD-Bench while being, as expected, slower than AFS customized to a specific objective.
HPO-ID, We test FunBO on two HPO tasks where the goal
17 Throughout the paper we adopt MetaBO's translation and scal- ing ranges.
7

---

FunBO: Discovering Acquisition Functions for Bayesian Optimization with FunScarch
1.0
BooSt
GPs.
d
MetaBO
UCB Ron
FunBO -
0.4
MetaBD
UCB
Funso
1.0
PSAP
UCB Randon
Pun30
Pot
Mean
0.3
0.8
0.6
0.6
0.2
0.4
0.4
0.2
0.1
0.2
0.0
0.0
0.0
5
10
15 Trials
20
25
30
0
10
20
30
40 Trials
50
60
70
0
10
20
30
40
50
Trials
Figure 5. Average BO performance when using known pereral-purpose AFS (gray lines), the AF learned by MetaBO (black dashed line) and Ctennso (blue line). Shaded area gives + std, dev./2. The red line represents Rs = 0, i.e. zero average regret. Left: HPO-ID. Middle: GPs-ID with d 1. Right: FEW-SHOT.
is to minimize the loss (d- = 2) of an RBF-based SVM and an AdaBoost algorithm.. 14 As in ID-Bench, we test the abil- ity to generate AFS that generalize well within function classes. Therefore, we train FunBO and MetaBO with losses computed on a random selection of 35 of the 50 available datasets and test on losses computed on the remaining 15 datasets. For FunBO we randomly assign 5 datasets to Gv and keep the rest in GT- FunBO identifies AFS (code in Fig. 15-16) that outperform all other AFS in AdaBoost (Fig. 5, left) while performing similarly to general-purpose or meta- learned AFS for SVM (Fig. 14). Across the two tasks, O'unno found in OOD-Bench still outperforms general-purpose AFS while yielding slightly worse performance compared to MetaBO and PunBO-customized AFS (Fig. 14, green lines).
those with low or moderate EI values. This increases the "peakiness" of the AF, leading to stronger exploitation of the most promising point(s). The term (1 + (z (z/beta) ) **2 sqrt (var) * **2 penalizes points more heavily if they have a very high EI (2 is large and positive) and/or high uncertainty (sqrt (var) is large). This might act as a regularizer against over-optimism or excessive jumps into highly uncertain areas, even if they look very promising according to EL. It's novel way of balancing exploration and exploitation discovered by FunBO.
FEW-SHOT. We conclude our experimental analysis by demonstrating how FunBO can be used in the context of few-shot adaptation. In this setting, we aim to learn an AF for a specific function class by "adapting" an initial AF with small number of instances from the target class.
GPS-ID. Similar results are obtained for general fune- tion classes whose members do not exhibit any partic- ular shared structure. We let GT, include 25 functions sampled from a GP prior with d = 3. RBF kernel and length-scale drawn uniformly from [0.05, 0.5]. We test the found AF on 100 other GP samples defined both for = and d = and length-scall scale values sampled sim- ilarly. As done by Volpp al. (2020), we consider ammensioonality-ggostii version of MetaBO that allows de- ploying the function learned from d = 3 functions on d = 4 objectives. We found arunDo to outperform all other AFS (code in Fig. 18) in d = 4 (Fig. 5, right) while match- ing EI and outperforming MetaBO in d = 3 (Fig. 17, left). In terms of interpretability, note how the AF found in this (1 + (z/beta) + sqrt (var)) ) ++2. This function calculates the standard EI, squares it, and then divides it by a penalty term that increases with both the standard- ized improvement z= (incumbent the uncertainty aqrt (var). The squaring of the EI non- linearly amplifies regions with high EI values relative to
We consider Ackley (d = 2) as the objective function and compare against FSAF (Hsich et al., 2021), the closest few- shot adaptation method for BO. FSAF trains the initial AF with a set of GPS, adapts it using 5 instances of scaled and translated Ackley functions, then tests the adapted AF on 100 additional Ackley instances, generated in the same manner. Note that FSAF uses wide variety of GP functions with different kernels and various hyperparameters for training the initial AF. On the contrary, FunBO few-shot adaptation is performed by setting the initial / function to the one found in GPS-ID (Fig. 5, right plot, green line) using 25 GPS with RBF kernel, and including the instances of Ackley used by FSAF in GT. Despite the limited training set, FunBO adapts very quickly to the new function instances, identifying an AF (code in Fig. 19) that outperforms both general purpose AFS and FSAF (Fig. 5, right).
case can be simplified be written (EI**2)
mean) /std and
5. Conclusions and Discussion
We tackled the problem of discovering novel, well- performing AFS for BO through FunBO, a FunSearch-based algorithm that explores the space of AFS by letting L.M.M it- eratively modify the AF expression in native computer code to improve the efficiency of the corresponding bo algorithm.
14 We use precomputed loss values across 50 datasets given as part of the HyLAP project. For SVM, the two hyperparameters are the RBF kernel parameter and the penalty parameter while for AdaBoost they correspond the number of product terms and the number of iterations.
8

---

FunBO: Discovering Acquisition Functions for Bayesian Optimization with FunScarch
We have shown across variety of settings that FunBO learns AFS that generalize well within and across function classes while being easily adaptable to specific objective functions of interest with only a few training examples.
adaptations of this problem, e-g., constrained optimization, noisy evaluations, multi-fidelity settings alternative sur- rogate models. In addition, FunBO can be used to search in the space of functions with different inputs thus potentially discovering, e.g., non-myopic AFS. Our method is inher- ently flexible and can accommodate such extensions, which we view as natural follow-up work. Additionally, FunBO demonstrates the potential to harness the power of LLMS while maintaining the interpretability of AFS expressed in code. This opens an exciting avenue for exploring how and what assumptions can be encoded within AFS, based on the desired program characteristics and prior knowledge about the objective function. Finally, the discovered AFS might have intrinsic value, independently of how they were discov- ered. Future work could focus on more extensively testing their properties and identify those that can be added the standard suite of AFS available in BO packages.
Limitations. FunBO inherits the strengths of FunSearch along with some of its inherent constraints. While FunSearch allows finding programs that are concise and interpretable, it works best for programs that can be quickly evaluated and for which the score provides an accurate quan- tification of the improvement achieved. Therefore, po- tential limitation of FunBO is the computational overhead associated with running a full BO loop for each candidate AF on every function in the set G. This limits the scalability of FunBO for larger sets 9 and hinders its application to more complex optimization problems, such as those with multiple objectives. However, several practical scenarios exist where the functions in 9 are relatively inexpensive to evaluate, thus lowering the evaluation cost incurred during FunBO's discovery phase. For instance, 9 could consist of: (i) faster, lower-fidelity simulators or simplified analytical models related to the expensive target problem f:: (ii) cheap- to-evaluate surrogate models learned from previous related tasks; or (iii) hyperparameter optimization losses on smaller datasets where training and evaluation are faster than the final, large target dataset. It is important to note that the computational cost incurred during this one-time search phase for a novel AF can be offset by its subsequent per- formance during online deployment, where the discovered AF may significantly reduce the number of costly function evaluations needed on the actual target problems.
Impact Statement
This paper presents work whose goal is to advance the field of Machine Learning. There are many potential societal consequences of our work, none which we feel must be specifically highlighted here.
References
Benjamins, C., Raponi, E.,, Jankovic, A., Doem, C., and Lin- dauer, M. Self-adjusting weighted expected improvement for Bayesian optimization. In International Conference on Automated Machine Learning, 2023.
Furthermore, the simple metric considered in this work in Eq. (I), only captures the distance from the true optimum and the number of trials needed to identify it. More research needs to be done to understand if a metric that better char- acterizes the convergence path for a given AF can improve FunBO's performance. In addition, each FunBO experiment shown in this work required obtaining a large number of LLM samples. This means that the overall cost of experi- ments, which depends on the L.M used as well as the algo- rithm's 's implementation (e.g. single-threaded or distributed, as originally proposed by FunSearch), can be high. Finally, as reported by Pareees et al. (2023), the variance in the quality of the AF found by FunBO is high. This is due the randomness in both the LLM sampling and the evolutionary procedure. While were able to reproduce the results shown for ID-Bench, HPO-ID and GPS-ID with different FunBO experiments, finding APS that perform well across function classes required multiple FunBO runs.
Bergstra, J., Bardenet, R., Bengio, Y., and Kégl, B. Algo- rithms for hyper-parameter optimization. In Advances in Neural Information Processing Systems, 2011.
Calandra, R., Seyfarth, A., Peters, J., and Deisenroth, M. P. Bayesian optimization for learning gaits under uncer- tainty: An experimental comparison on dynamic bipedal walker. Annals of Mathematics and Artificial Intelligence, 76:5-23, 2016.
Chen, A., Dohan, D., and So, D. Evoprompting: Language models for code-level neural architecture search. In Ad- vances in Neural Information Processing Systems, 2024a.
Chen, L., Chen, J., Goldstein, T., Huang, H., and Zhou, T. InstructZero: Efficient instruction optimization for black- box large language models. In International Conference on Machine Learning, pp. 6503-6518, 2024b.
Future work. This work opens up several promising av- enues for future research. While our focus here was on the simplest single-output BO algorithm with GP surrogate model, FunBO can be extended to learn AFS for various
Chen, Y., Hoffman, M. W., Colmenarejo, S. G., Denil, M., Lillicrap, T. P., Botvinick, M., and Freitas, N. Learning to learn without gradient descent by gradient descent. In International Conference on Machine Learning, pp. 748-756, 2017.

---

FunBO: Discovering Acquisition Functions for Bayesian Optimization with FunScarch
Chen, Y., Song, X., Lee, C., Wang, Z., Zhang, R., Dohan, D., Kawakami, K., Kochanski, G., Doucet, A., Ranzato, M., Perel, S., and de Freitas, N. Towards learning uni- versal hyperparameter optimizers with transformers. In Advances in Neural Information Processing Systems, pp- 32053-32068, 2022.
Hoffman, M., Brochu, E., De Freitas, N., et al. Portfolio allocation for Bayesian optimization. In Conference on
Uncertainty in Artificial Intelligence, pp. 327-336, 2011.
Hsieh, B.-J., Hsieh, P.-.., and Liu, X. Reinforced few-shot acquisition function learning for Bayesian optimization. In Advances in Neural Information Processing Systems, pp. 7718-7731, 2021.
Cheng, J., Liu, X., Zheng, K., Ke, P., Wang, H., Dong, Y., Tang, J., and Huang, M. Black-box prompt optimization: Aligning large language models without model training. In Annual Meeting of the Association for Computational Linguistics, 2024.
Jiang, A. Q. Li, W., Tworkowski, S., Czechowski, K., Odrzygóřdż, T., Miloć, P., Wu, Y., and Jamnik, M. Thor: Wielding hammers integrate language models and au- tomated theorem provers. In Advances in Neural Infor- mation Processing Systems, pp. 8360-8375 2022.
Cho, H., Kim, Y., Lee, E., Choi, D., Lee, Y., and Rhee, W. Basic enhancement strategies when using Bayesian optimization for hyperparameter tuning of deep neural networks. IEEE Access, 8:52588-52608 2020.
Jones, D. R., Schonlau, M., and Welch, W. J. Efficient global optimization of expensive black-box functions. Journal of Global Optimization, 13:455-492, 1998.
Coffiman, E. G., Garey, M. R., and Johnson, D. S. Approxi- mation algorithms for bin-packing - an updated survey. In Ausiello, G., Lucertini, M., and Serafini, P. (eds.), A/- gorithm Designfor Computer System Design, pp. 406. Springer, 1984.
Korovina, K., Xu, S., Kandasamy, K.. Neiswanger, W., Poe- 2os, B., Schneider, J., and Xing, F. Chembo: Bayesian op- timization of small organic molecules with synthesizable recommendations. In International Conference on Artifi- cial Intelligence and Statistics, pp. 3393-3403, 2020.
Ding, Q., Kang, Y., Liu, Y.W.., Lee, T. C. M., Hsieh, C.-J., and Sharpnack, J. Syndicated bandits: A framework for auto tuning hyper-parameters in contextual bandit algo- rithms. In Advances in Neurul Information Processing Systems, pp. 1170-1181, 2022.
Koza, J. R. Genetic programming as a means for program- ming computers by natural selection. Statistics and com- puting, 4:87-112, 1994.
Kristiadi, A., Strieth-Kalthoff, F., Skreta, M., Poupart, P., Aspuru-Guzik, A., and Pleiss, G. A sober look at LLMs for material discovery: Are they actually good for Bayesian optimization over molecules? In Inter- national Conference Machine Learning, pp. 25603- 25622, 2024.
Fernando, C., Banarse, D. S., Michalewski. H., Osindero, S., and Rocktäschel, T. Promptbreeder: Self-referential self- improvement via prompt evolution. In International Con- ference on Machine Learning, pp. 1 13481-13544, 2024.
Feurer, M. Letham, B., and Bakshy, E. Scalable meta-learning for Bayesian optimization using ranking- weighted Gaussian process ensembles. In AutoML Work- shop at International Conference on Machine Learning, 2018.
Kushner, H. J. A versatile stochastic model of a function of unknown and time varying form. Journal of Mathemati- cal Analysis and Applications, 5(1):150-167, 1962.
Kushner, H. J. A new method of locating the maximum point of an arbitrary multipeak curve in the presence of noise. Journal Basic Engineering. 86(1):97-106, 1964.
Frazier, P. I., Powell, W. B., and Dayanik, S. A knowledge- gradient policy for sequential information collection. SIAM Journal on Control and Optimizaaiion, 24150 2439, 2008
Lai, T. L. and Robbins, H. Asymptotically efficient adaptive allocation rules. Advances in Applied Mathematics, 6(1): 4-22, 1985.
Garnett, R. Bayesian Optimization. Cambridge University Press, 2023.
Lehman, J., Gordon, J., Jain, S., Ndousse, K., Yeh, C., and Stanley, K. O. Evolution through large models. In Banzhaf, W., Machado, P., and Zhang, M. (eds.), Hand- book of evolutionary machine learning, pp. 331-366. Springer, 2023.
Google-PaLMM PaLM-2-Team PaLM Technical Report. arXiv preprint ar arXiv: 10403, 2023.
Guo, Q., Wang, R., Guo, J., Li, B., Song, K., Tan, X., Liu, G. Bian, J., and Yang, Y. EvoPrompt: Connecting LLMs with evolutionary algorithms yields powerful prompt op- timizers. In International Conference on Learning Repre- sentations, 2024.
Li, L., Jamieson, K., DeSalvo, G., Rostamizadeh, A., and Talwalkar, A. Hyperband: A novel bandit-based approach to hyperparameter optimization. Journal of Machine Learning Research, 18(1 (11185):1-52 2018
10

---

FunBO: Discovering Acquisition Functions for Bayesian Optimization with FunScarch
Liu, F., Xialiang, T., Yuan, M., Lin, X., Luo, F., Wang, Z., Lu, Z.,, and Zhang, Q. Evolution of heuristics: Towards efficient automatic algorithm design using large language model. In International Conference on Machine Learning, pp. 32201-32223, 2024a.
Ranković, B. and Schwaller, P. Bochemian: Large language model embeddings for Bayesian optimization of chemical reactions. In Adaptive Experimental Design and Active Learning in the Real World Workshop at Advances in Neural Information Processing Systems, 2023.
Liu, T., Astorga, N., Seedat, N., and van der Schaar, M. Large language models to enhance Bayesian optimization. In International Conference on Learning Representations, 2024b.
Rasmussen, C. E. and Williams, C. K. Gaussian Processes for Machine Learning. MIT Press Cambridge, MA, 2006.
Komera-Pareder B., Barekatain, M., Novikov, A., Balog, M., Kumar, M. P., Dupont, E., Ruiz, F. J.,, Ellenberg, J. S., Wang, P., Fawzi, O., et al. Mathematical discoveries from program search with large language models. Nature, pp. 1-3, 2023.
Lyu, W., Yang, F., Yan, C., Zhou, D., and Zeng, X. Batch Bayesian optimization via objeetvoeeective acquisition en- semble for automated analog circuit design. In Interna- tional Conference on Machine Learning, pp. 5306-3314, 2018.
Saltanis, R. P. One method of multiextremum optimiza- tion. Avtomatika i Vychislitel naya Tekhnika (Automatic Control and Computer Sciences), 5(3):33-38, 1971.
Maraval, A., Zimmer, M., Grosnit, A., and Bou Ammar, H. End-to-end meta- Bayesian optimisation with transformer neural processes. In Advances in Neural Information Processing Systems, 2024.
Schulman, J., Wolski, F., Dhariwal, P., Radford, A., and Klimov, O. Proximal policy optimization algorithms. 2017.
arXiv preprint arXivv:107..000
Meyerson, E., Nelson, M. J., Bradley, H., Gaier, A., Moradi, A., Hoover, A. K., and Lehman, J. Language model crossover: Variation through few-shot prompting. ACM Transactions on Evolutionary Learning and Optimization, 4(4), 2024.
Snoek, J., Larochelle, H., and Adams, R. P. Practical Bayesian optimization of machine learning algorithms. Advances in Neural Information Processing Systems, 2012.
Sobol', M. On the distribution of points in a cube and the approximate evaluation of integrals. Zhurnal Vychis- litel noi Matematiki i Matematicheskoi Fiziki, 7, 1967.
Mockus, J. On Bayesian methods for seeking the extremum. Proceedings of the IFIP Technical Conference, pp. 400- 404, 1974.
Song, L.,, Gao, C., Xue, K., Wu, C., Li, D., Hao, J., Zhang, Z., and Qian, C. Reinforced in-context black-box opti- mization. arXiv preprint arXiv:2402.1 /7423, 2024a.
Müller, S., Feurer, M., Hollmann, N., and Hutter, F. Pfns4bo: In-context learning for bayesian optimization. In Inter- national Conference on Machine Learning, pp. 25444- 25470, 2023.
Song, X., Tian, Y., Lange, R. T., Lee, C., Tang, Y., and Chen, Y. Position: leverage foundational models for black-box optimization. In International Conference on Machine Learning, 2024b.
Nasir, M. U., Earle, S., Togelius, J., James, S., and Cleghorn, C. LImatic: Neural architecture search via large language models and quality diversity optimization. In Genetic and Evolutionary Computation Conference, pp. 1110-1118, 2024.
Sun, T., Shao, Y., Qian, H., Huang, X., and Qiu, X. Black- box tuning for language- -ow-e-ooo-d---eeeeee----e In Inter- national Conference on Machine Learning, pp. 20841- 20855, 2022.
Perrone, V., Shen, H., Seeger, M. W., Archambeau, C., and Jenatton, R. Learning search spaces for Bayesian optimization: Another view of hyperparameter transfer learning. In Advances in neural information processing systems, 2019.
Swersky, K., Snoek, J., and Adams, Multi-task Bayesian optimization. In Advances in Neural Infor- mation Processing Systems, 2013.
Polu, S. and Sutskever, I. Generative language model- ing for automated theorem proving. arXiv preprint arXiv:2009.03393, 2020.
Tanese, R. Distributed Genetic Algoritiums for Function Optimization. PhD) thesis, University of Michigan, 1989.
Tribes, C., Benarroch-Lclong S., Lu, P., and Kobyzev, I. Hyperparameter optimization for large language model instruction -unntnttttnnnnnn In AAAI Conference on Artificial In- telligence, 2024.
Ramos, M. C., Michtavy, S. S., Porosoff, M. D., and White, A. D. Bayesian optimization of catalysts with in-context learning. arXiv preprint arXiv:2304.0m 05341, 2023.

---

FunBO: Discovering Acquisition Functions for Bayesian Optimization with FunScarch
Vertex AI, G. C. Code models overview.
2023. URL https://cloud.goon/ com/
vertex vertex-ai/docee aiddccs//////////o generative-ai/cote code-models-overviln
Volpp, M., Fröhlich, L. P., Fischer, K., Doerr, A., Falkner, S., Hutter, F., and Daniel, C. Meta-learning acquisition functions for transfer learning in Bayesian optimization. In International Conference on Learning Representations, 2020.
Wang. C., Liu, X., and Awadallah, A. H. Cost-effective iyperparaameter optimization for large language model generation inference. In International Conference on Automated Machine Learning, pp. 21-1, 2023.
Wang, Z. and Jegelka, Max-value entropy search for effi- cient Bayesian optimization. In International Conference on Machine Learning, pp. 3627-3635, 2017.
Wistuba, M. and Grabocka, J. Few-shot Bayesian opti- mization with deep kernel surrogates. In International Conference Learning Representations, 2021.
Wistuba, M., Schilling, N., and Schmidt-Thieme. L. Scal- able Gaussian process-based transfer surrogates for hy- perparameter optimization. Machine Learning, 107(1): 43 43-78, 2018.
Yang. C., Wang. X., Lu, Y., Liu, H., Le, Q. V., Zhou, D., and Chen, X. Large language models as optimizers. In International Conference on Learning Representations, 2024.
Yao, Y., Liu, F., Cheng, J., and Zhang. Q. Evolve cost- aware acquisition functions using large language models. In Affenzeller, M., Winkler, S. M., Kononova, A. V., Trautmann, H., Tušar, T., Machado, P., and Bäck, T. (eds.), Parallel Problem Solving from Nature PPSN XVIII, pp. 374-390. Springer Nature Switzerland, 2024.
Yogatama, D. and Mann, G. Efficient transfer learning method for automatic hyperparameter tuning. In Interna- tional Conference Artificial Intelligence and Statistics, pp. 1077-1085, 2014.
Zhang, M. R., Desai, N., Bac, J., Lorraine, J., and Ba, Using large language models for hyperparameter opti- mization. In Foundation Models for Decision Making Workshop at Advances in Neural Information Processing Systems, 2023.
Zheng, M., Su, X., You, S., Wang, F., Qian, C., Xu, C., and Albanie, S. Can GPT-4 perform neural architecture search? arXiv preprint 2304 10970, 2023.
12

---

FunBO: Discovering Acquisition Functions for Bayesian Optimization with FunScarch
A. Related work
LLMS as mutation operators. FunBO expands FunSearch (Romera- Pareeree et al., 2023), an evolutionary algorithm pairing an LLM with an evaluator to solve open problems in mathematics and algorithm design. The idea of using LLMS as mutation operators paired with scoring mechanism has been explored to create optimize code for robotic simulations, to optimize prompts (Guo et al., 2024), to evolve stable diffusion images with simple genetic algorithms (Meyerson et al., 2024). Other works explore the use of LLMS to search over neural network architectures described with Python code (Nasir et al., 2024; Zheng et al., 2023; Chen et al., 2024a). to find formal proofs for automatic theorem proving (Polu & Sutskever, 2020; Jiang et al., 2022) or to automatically design heuristics (Liu et al., 2024a). Symbolic regression, similarly to FunBO, also searches for mathematical expressions, which are often represented as trees. However, while FunBO employs an L.M for both generation and mutation, symbolic regression typically evolves these expressions using genetic operators (Koza, 1994). In addition, FunBO operates directly on code representations and uses full bо performance its fitness metric. Therefore, while both methodologies aim to find functional forms, their search space representations and primary evolutionary operators are different.
-mprovemeen loop (Lehman et al., 2023), to
Meta-learning for BO. Our work is also related to the literature on meta-learning for BO. In this realm, several studies have focused on meta-learning an accurate surrogate model for the objective function by exploiting observations from related functions, for instance by using standard multi-task GPS (Swersky et al., 2013; Yogatama & Mann. 2014), ensembles of GP models (Feurer et al., 2018; Wistuba et al., 2018: 8; Wistuba & Grabocka, 2021) or neural processes that are trained to approximate the posterior predictive distribution through in-context learning on any prior distribution that can be efficiently sampled from (Müller et al., 2023). Others have focused on meta-learinng neural networks with access to gradient information (Chen et al., 2017) transformers (Chen et al., 2022; Song et al., 2024a). Note that, while meta-learned surrogate models explicitly learn structure from past functions by observing data-points for each of them, methods that meta-learn AFs via G implicitly learn similarities between these objectives by observing the optimization pattern achieved by previously sampled AFS for each objective function in G. Interestingly, the most significant performance gains observed for the approach proposed by Chen et al. (2022) stem from using standard AF (EI) on top of the transformer architecture for output predictions. This confirms the continued importance of AFS as crucial components in BO, even when combined with wansformer-basee approaches, and highlights the importance of method such FunBO that can be seamlessly integrated with these newer architectures, potentially leading to further improvements in performance. More relevant to our work are studies focusing on transferring information from related tasks by learning novel AFs that more efficiently solve the classic exploration -eploittation trade-off in BO algorithms (Volpp et al., 2020; Hsich et al., 2021; Maraval et al., 2024). In contrast to prior works in this literature, FunBO produces AFS that are more interpretable, simpler and cheaper to deploy than neural network-based AFS and generalize not only within specific function classes but also across different classes.
purpoose optimizers by using recurrent
LLMS and black-box optimization. Several works have investigated the of LLMS to solve black-box optimization problems. Song et al. (2024b) discussed different ways in which foundational language models can revolutionize optimization, from harnessing the vast wealth of information encapsulated in free-form text to enriching task comprehension. Both Liu et al. (2024b) and Yang et al. (2024) framed optimization problems in natural language and asked LLMS to iteratively propose promising solutions and/or evaluate them. Similarly, Ramos et al. (2023) replaced surrogate modeling with LLMS within a BO algorithm targeted at catalyst or molecule optimization. Kristiadi et al. (2024) also investigated whether L.LSS can accelerate BO in the molecular space while Ranković & Schwaller (2023) explored the integration of LLMS with BO in the domain of chemical reaction optimization. Other works have focused exploiting black-box methods for prompt optimization (Sun et al., 2022; Chen et al., 2024b; Cheng al., 2024; Fernando et al., 2024), solving HPO tasks with LLMS (Zhang et al., 2023) or identifying optimal LLM hyperparameter settings via black-box optimization approaches (Wang et al., 2023; Tribes et al., 2024). Concurrent to our work, Yao et al. (2024) propose using an LLM coupled with an evolutionary procedure to find cost-aware AFS.
AFS representations Works proposing new meta-learned or general-purpose AFS can also be classified based on the representation used for the AFS. Unlike general-purpose AFS, for which an analytical representation is available, recent works have explored representing AFS via neural networks code. Among the works using neural networks, Volpp et al. (2020) proposed neural AF that is a MLP with ReLU activations while Chen et al. (2022) and Maraval et al. (2024) jointly trained surrogate models and AFS via transformers or neural processes. Instead, the recent work by Yao et al. (2024) represents AFS for settings with limited experimentation budgets in code.
13

---

FunBO: Discovering Acquisition Functions for Bayesian Optimization with FunScarch
inport numpy as up scapy stats import norm
def Aequisittion_fuutioon (ieeiiicennnnn.rrriciieeeennn predictive_var, incumbent, beta=1.8):
"*"Reuurns the index of the point collect in vector of eval points.
Given the posterior mean and posterior variance of a GP model for the objective function, this function computes an heuristic and finds its optimun. The next function evaluation will be placed at the point corresponding to the selected index in vector of possible eval points.
Args:
predictive_neatt
of shape [nuntpoints, dim] containing the predicted mean values for the GP model on the objective function for 'nun_points' points of dimensionality 'dim'.
predictive var: an array of shape [num points, din] containing the predicted variance values for the GP model on the objective function for 'nun_points' points of dimensionality 'din'.
incumbent: current optimum value of objective function observed. possible hyperparaneter construct heuristic.
Returas:
An integer representing the index of the point in the array of shape [num_points, din] that needs to be selected for function evaluation.
z (incunbent predictive mean)) / sert sp. (predictáve_ predictive_ste ap-sqrt (preddicccti vals (incumbent predictive mean) norn. cdE() predictive.std (2)
return ap. argaax (vals)
Figure 6. Python code for FunBO's initial h function with full docstring.
B. Code for FunBO components
Fig. 6 gives the Python code for the initial acquisition function used by FunBO, including the full docstring. The doestring describes the inputs of the function and the way in which the function itself is used within the evaluate function e. Evaluation of the functions generated by FunBO is done by first running a full BO loop (see Fig. 7 for Python code) and then, based on its output (the initial optimal input value, the true optimum, the found optimum and the percentage of steps taken before finding the latter), computing the score in the Python code of Fig. 8. Note how the latter captures how accurately and quickly Bo algorithm using the proposed AF finds the true optimum.
C. Programs Database
The DB structure matches the one proposed by FunSearch (Romera-Paredes et al., 2023). We discuss it here for completeness. multiple-deme model (Tanese, 1989) is employed to preserve and encourage diversity in the generated programs. Specifically, the program population in DB is divided into Nos islands, each initialized with the given initial and evolved independently. Within each island, programs are clustered based on their scores on the functions Gry, with AFS having the same scores grouped together. Sampling from DB involves first uniformly selecting an island and then sampling two AFS from it. Within the chosen island, cluster is sampled, favoring those with higher scores, followed by sampling program within that cluster, favoring shorter ones. The newly generated AF is added to the same island associated with the instances in the prompt, but to a cluster reflecting its scores on GTy Every 4 hours, all programs from the Nop islands with the looestg best AF are discarded. These islands are then reseeded with single program from the surviving islands. This procedure climinates underperforming AFS, creating space for more promising programs. See the Methods section in Romera- Paaaa-a al. (2023) for further details.
14

---

FunBO: Discovering Acquisition Functions for Bayesian Optimization with FunScarch
"Evaluate an AF with full BO loop for the objective f."" inport GPy
inport numpy up inport utils
def (f,_ objective function to minimize cnauissitionffunctiii h given by LLM
nun eval points 1000, nun trials 30):
"*"Ron 30 loop return the minimum value found the percentage trials required to reach it.
Get evaluation points for AP. potieval_points( Sobol grid on the f's input space
number of points
eval_points utils. peals..gle.eallloootttte num_eval_points)
Get the initial paint with get_initial_designn This is set to be the point giving the maximum (worst) function evaluation among eval_points
initial initialy utils. (f)
Initialize GP paraeeers..,.s.. pre-compute the RBF kernel param-easss
* for each given f. These are returned by get.hrperparameters().. hp = geils.get hyperparaneters((5)
Initialize kernel and model.
kernel (wn..aaiacccuuniiiiiiiniaaaaa
variance=hp( variance'
( ARD=hp('ard'))
model oraallsssGRegressiniiiiitttaaa initial_y, kernel)
Get initial predictive mean and var.
predictive_mean, predictive_var model, predict
initial optimum value.
found_nin initial.inn. float (np. (model.)
Get true optimun value. true_min (g..ia((((eval_pointt))
Optinization loop. for in (nngennu.triials)) selected_ idx new_input val__points(selected_l
predictive_var, Cound_min)
point using
# Get point.
new_output f(new f(new_input) # Evaluate new point.
Append to dataset.
medel.set (ap. concatenate new input,, axis=0), concatenate (aaccate new output), axis=0)) Get updated nean and var predictive_nean predictive_war model. (reddict(evlll (ap.mdn(nodeel..)
optinun
Get percentage trials (note that & to identify the optimun.
the number of points in the initial design) needed percentage steps before converging (ap. andel Y) - initial (happepp)) / nun_trials) found_min == true_min else 1.0
return (found_min, min, tine initial min.y. percentage_steps.bll wwt.:booooreeccvveeriieee
Figure 7. Python cade for the first part of used in FunBO. This function full loop with given number of trials and points Sobol grid to assess how efficiently a given AF allows optimizing f.
15

---

FunBO: Discovering Acquisition Functions for Bayesian Optimization with FunScarch
"""Score an AF given the output of run_bo().
inport numpy up
def score score(found min, true min, initial min.y, percentage steps before, www.re.ccoweereeeeeeeeeeeeeeeeee converging): Compute score based on the output of ()) ""*
Get score based on how close the found optimun is to the true one (first term in Eq. (1)).
score_ming reree_ee 1.0 ap. (fouud true_min) / (initial min_y true_min)
Get score based how the percentage of trials needed identify the true optimun (second term in Eq. (1)).
score steps needed 1.0 percentage steps steps_before before. converging
return score. ccore_nin_reachee
score_steps.neede
Figure 8. Python code for the second part of used in FunBO. Based the output of boO, this function computes capturing how accurately and quickly an AF allows identifying the true optimum.
D. Experimental details
In this section, we provide the experimental details for all our experiments. We run FunBO with 48hrs, B 12 and Nos 10. To isolate the effect of using different AFS and eliminate confounding factors related to AF maximization surrogate models, we maximized all AFS on fixed Sobol grid (of size Nso) over each function's input space. We also ensured the same initial design across all methods (including the point with the highest/worst function value on the Sobol grid) and used consistent GP hyperparameters which were tuned offline and fixed. In particular, we use a GP model with zero mean function and RBF kernel defined as Kg (XX') = ofexp(-1112
x'112 with f = (f, of) where and of are the length-scale and kernel variance respectively. The Gaussian likelihood noise o² is set to le 5 unless otherwise stated. We set 7' = 30 for all experiments apart from IIPO-ID and GPs-ID for which we use T 20 to ensure faster evaluations of generated AFS. We used the MetaBO implementation provided by the authors at retaining default parameters except for removing the local maximization of AFS and ensuring consistency the initial design. We followed the same procedure for FSAF, using code available at github.com/pingtsse//SSS/. We ran UCB with Experiment-specific settings detailed below.
D.1. OOD-Bench
The parameter configurations adopted for each objective function used in this experiment, either in in F. are given in an kernel. Scaled and translated functions are obtained with
Table 1. Notice that for Hartmann with d 3 translations sampled uniformly in 0-1.1, 0.1]d and scalings sampled uniformly in [0.9, 1.1]. Fig. 9 gives the results achieved version of MetaBO that does not take the possible evaluation points
by O'Fnnoo (blue line) and dimensionality as input to the neural AF. This allows the neural AF to be trained on one-dimensional functions and be used to optimize functions across input dimensions.
D.2. ID-Bench
The parameter configurations for Branin, Goldstein-Prin and Hartmann are given in Table 2. For this experiment, we adopt the parameters used by Volpp et al. (2020) thus optimize the functions in the unit-hypercube and use ARD RBF kernels. Fig. 10 gives the results achieved by arunno (blue line) and the AF found by FunBO for OOD- RoD-Bench (green). The Python code for the found AFS is given in Figs. 11-13.
16

---

FunBO: Discovering Acquisition Functions for Bayesian Optimization with FunScarch
1.0
MetaBO-DA
UCB
FunBO
1.0
MetaBO-Da DA
UCB
FunBO
EI
Pofl
A,
EI
Pofi
R:
0.8
Mean
Random
0.8
Mean
Random
0.6
0.6
0.4
0.4
0.2
0.2
0.0
0.0
10
20
30
40
50
10
20
30
40
50
Trials
Trials
Figure 9. B0D-Bench. Average Bо performance when using known general-purpose purpose AFS (gray lines with different patterns), the AF learned by dimensionalliy-agooiii version of MetaBO (MetaBO-DA, black dashed line) and Cremao (blue line). Shaded area gives 1: std. d2v./2. The red line represents R2 = 0, i.e. zero average regret. Left: JF includes nine different synthetic functions. Right: Extended test including, for each function F, 50 randomly scaled and translated instances.
Branin
Goldstein Price
Hartmann
0.8
MeraBC
MetsO
Random
Meta50
Random
FunBo
FunB0
0.7
RunBO
Mean
FunbO. 000 Sench
Mean
FunBO, OCD-Bench
TuntO, Ben-h
0.6
0.6
0.6
UCB
Port
Pet
0.5
0.5
0.5
0.4
0.4
0.4
0.3
0.2
0.1
0.1
0.1
0.0
0.0
0.0
0
2
6
8
10
0
5
10
15
20
25
30
10
15 Trials
20
25
30
Trials
Figure 10. ID-Bench. Average BO performance when using known general-purpose AFS (gray lines with different patterns), Opunso found in OOD-Bench (green line), the AF learned by MetaBO (black dashed line) and Orumao (blue line) on instances Branin, Goldstein- Price and Hartmann. Shaded gives + std. dev./2. 22. The red line represents R. 0, i.e. average regret.
17

---

FunBO: Discovering Acquisition Functions for Bayesian Optimization with FunScarch
import numpy as from scipy.state inport norm
def htw.iiiioo_uuuctiionnneeeiiii mean, predictive_var incumbent, beta=1.0).
""Returns index of point collect
y-pred predictive_mean di f.ff.coureent.bsst..pp.e. incumbent y-pred
predictive_var
standarrd standard deviation np ap.aaiinummmp sqrt (predictive var). le-15)
z liff_current_best.y.rrdd / hound_standar__deiittt
vals
npp.sgr((predicciveeea)) (orm.cdf(()
8.
(dfom.cdd(()) norm. cdf(2 0.5)) predictive.
incunbent)
alpha diff_ccuremt_besttypppee if incumbent 0.8 else inf
alpha = ma..aaimmm 0.) (aalpha 0.5 # a) y-pred
y_vals np. absolute
np. abs * (a >= 0.)
for y_val in y_vals:
idx np. ww..agaaxx (y_val y-pred) hound_standarrddddviatii
vals[idx]
np argmax (vals)
Figure 11. ID-Bench. Python code for Oyunso for Branin. The BO performance corresponding to this AF is given in Fig. 4 (left).
inport
from scipy. stats inport norm
(preeiittepeddiiiiiee.e.. mean, predictive. var, incumbent, beta=1. : .** """Returss the index of the point to collect ... shape, predictive segeiiiveeeea.hhappp best_score 0.0
g_i 0.8
predectie..vvr((sape-10//// 11h((/)) 8= dim predictive_wart- up. isfiniit (predictive_var)] ip..ssfiiiiteppreeittieeeeee
1.8
for range(predictive shape
(incumbent predietive_mean(i)
new_score predictive_var(i] * norm. cdurrer_z.. 0.5)
if new_score best_score:
best_score new_score
g = return g_i
Figure 12. ID-Bench. Python code for Greano for Goldstein-Price. The performance corresponding this AF is given in Fig. (middle).
18

---

FunBO: Discovering Acquisition Functions for Bayesian Optimization with FunScarch
Table J. Parameters used for OOD-Bench.
X
Ns
i
of
d²
Ackley
[-4, 4, 4]
1000
0.21
28.19
le 5
Levy
| 10, 10]
1000
1.05
83.32
le
Schwefel
500,
1000
18.46
76868.65
lc
Rosenbrock
50
1000
1.20
87328.20
1e 5
Sphere
[-5,
1000
18.46
924202.43
1c
Styblinski-Tang
1-5, 5]
1000
7.34
119522207.86
Weierstrass
0.,5, 0.5]
1000
0.01
0.39
le
Beale
2
[]4
10000
0.46
546837.32
le 5
Branin
2
5, 10] [0, 15)
10000
4.65
155233.52
le
Michalewicz
10,
10000
0.22
0.10
1c
Goldstein- Price
2
2
10000
0.27
117903.96
le 5
Hartmann-3
3
00, 1|3
1728
[0.716, 0.298, 0.186]
0.83
1.688c 11
Hartmann-6
[0, 1]
729
1.0
1.0
Table Parameters used for D-Bench.
d
X'
Ns
of
Branin
2
[0, 1]
961
0.235,, 0.578
2.0
8.9e 16
Coldstein-Price
[0, 1
961
[0.130, 0.07]
0.616
le
6
Hartmann-3
[0, 1],
1728
[0.716, 0.298, 0.186]
0.83
1.688e 11
D.3. HPO-ID
For this experiment, we adopt the GP hyperparameters used by Volpp et al. (2020). From the training datasets used in MetaBO, we assign "bands", "wine", "coil2000", "winequality-red"" and "titanic" for Adaboost, and "bands", "breast-cancerr, "banana", "yeast" and "vehicle" for SVM to Gv. We keep the rest in Gr. Fig. 14 gives the results achieved by O'uunoo (blue lines) and the found by FunBO for oOn-Bench (green lines). The Python code for the found AFS is given in Figs. 15-16.
D.4. GPS-ID
The functions included in G and F are sampled from a GP prior with RBF kernel and length-scale values drawn uniformly from [0.05, 0.5). The functions optimized in the input space [0, 1|3 with Nsn 1728 points. In terms of GP hyperparameters. set o = 1.0, x² le- 20 and use the length-scale value used to sample each function as f. Fig. 17 gives the results achieved by ArFunoo and the AF found by FunBO for OOD-Bench. The Python code for d'Fumao is given in Fig. 18.
D.5. FEW-SHOT
For this experiment, the Ackley functions used to "adapt" the initial AF obtained by scaling and translating the output and inputs values with translations and scalings uniformly sampled in 1-0.1, 0.1] and [0.9, 1.1] respectively. The test set includes 100 instances of Ackley similarly obtained with scale and translation values in [0.7, 1.3] and [-0.3, 0.3jd respectively. Furthermore, we consider [o, 1] as input space and use Nsa 1000. The GP hyperparameters are set to = [0.07, 0.018] (ARD kernel), of = 1.0 and o² 8.9e 16. Python code for Ctynnoo given Fig. 19.
D.6. OOD-Bench with BoTorch eval pipeline
To ensure a consistent comparison across all AFS, in the manuscript, we have presented results obtained by testing AFS using a fixed Sobol grid, fixed hyperparameters (tuned offline), and a fixed initial design that includes the input point yielding the maximum function value on the grid. As this setup might deviate from a fully realistic deployment scenario, this section,
19

---

FunBO: Discovering Acquisition Functions for Bayesian Optimization with FunScarch
inport numpy as up from scipy, inport from scipy import
def acquisition acquisittio_ffucciion(nnrree (preeiiioppreeiiiiieeeeemaaa predictive_var, incumbent, beta=1.8). """Retunss the index of the point to collect ...
diff cif_.currett_bee best_mean incumbent predictive
standard deandind sprt (preddiiitttiie..... var)
diff_current.bestttte
standard_deviattan
vals diff_current.best.meaa
(x)
norn. cdf (2) 82))## norn.edf(e
(2)
index = mp..paaaaae argnar (stats. www.cnrrr.cdeee...o cdf(vals, a=-0.1, b=
return index
Figure ID-Bench, Python code for Cinunno Hartmann. The Bo performance corresponding to this is given Fig. 4 (right).
AdaBoost
SVM
1.0
1.0
EI
FunBO
FunBO
0.8
Mean
FunBO, 00D-Bench
0.8
Mean
FunBO, GOD-Bench
R
-
0.6
0.6
0.4
0.4
0.2
0.2
0.0
0.0
0
5
10
15 Trials
20
25
30
5
10
15 Trials
20
25
30
Figure 14. HPO-ID. Average Bо performance when using known general-purpose purpose AFS (gray lines with different patterns), a meta-learned AF by MetaBO (black dashed line), Ovanno found in OOD OOD-Bench (green lines) and Cryano (blue lines). Shaded area gives + std. dev. The red line represents N. = 0, ie.. zero average regret.
inport numpy
from scipy. stats import norm
def crquisitiin_functioon (prnttioorrediictivee.eam predictive_war, incumbent, beta=1.0):
"""Returns the index of the point collect ... ex-exx(-beta)
2.0 beta ep.exb(bbeta)
alpha (2.) & beta tp..srt(predictiveeee
(incumbent - predictive mean) alpha
vals -ebs(ee * exp
power (z, 2)) - 1.4 + cl incumbent
2.0 beta np power 2)
vals - np. log (np. power (alpha, 2))
vals [ml.mni.argnincc [vv]in np. (vas)
Figure 15. HPO-ID. Python code for Ormmno for AdaBoost. The BO performance corresponding to this AF is given in Fig. 5 (left).
20

---

FunBO: Discovering Acquisition Functions for Bayesian Optimization with FunScarch
inport попру as from scipy.stats inport пorn
fut.ttioooouuuuttiooooooreee ". ""Returns index of point collect ... -..
predictive_var, incumbent, beta=1.8):
(incunbent predictive_nean) np sert (predietive_var)
vals (incumbent predictive_mean) * norn. cdf(z
np.sqrt (prodictive__ar) norn.pdf(2)
t0_val norm normcloc=nncuubent
wtw.rrdditiiee..aa.dd.dddceoooeo (incumbeet))))
tl_val = norm. (d)()
vals ((vals tl_val t0_val) (1 tl_val)
vals/(1 = tl_val+tti.val 2)/beta)
return arw.aaggaaax... (vals)
Figure 16. HPO-ID. Python code for Ormmno for SVM. The BO performance corresponding to this AF is given in Fig. 14 (right).
GPs, d= =
GPs, = 4
0.4
MetaBO
Random
0.4
MetaBO
Pofl
EI
Fun80
FunBO
Mean
FunBO, 00D-Bench
Mean
FunBO, O0D-Bench
UCB
Ag = 0
A;
0.3
Pofl
0.3
0.2
c 0.2
0.1
0.1
0.0
0.0
10
20
30
40 Trials
50
60
70
0
10
20
30
40 Trials
50
60
70
Figure 17. Average Bo performance when using known general-purpose APS (gray lines with different patterns), the AF learned by MetaBO (black dashed line), Osumno found OOD-Bench (green lines) and Orumno (blue lines). Shaded area gives + std. dev./2. The line represents RG = 0, i.c. zero average regret. Left: GPs-ID. JF includes functions with d = 3. Right: FF includes functions with d = 4.
inport numpy
from scipy. stats inport
def
(prnittvenn predictive_var, incumbent, beta 1.0):
.. ""Returns the index of the point collect ... ...
(incunbent predictive_nean) / np. sert sert (predictive__var) ((incumbent predictive mean) norm.cdf(2 ap.edic(prriiitvvee
(or)
vals vals / (1 (z beta)##2 np (pre (pp.sarr(prediiciivevaa)) return пр awgmax (vals)
Figure GPs-ID. Python code for Oremaoo The BO performance corresponding to this AF is given in Fig. 5 (right).
21

---

FunBO: Discovering Acquisition Functions for Bayesian Optimization with FunScarch
Inport numpy as up
scipy.stats stats import
def htw.:aiciooouuccioooonccecccccoo
predictive_var, incumbent, beta=1.8):
"Returns the index of the point collect nun_points, predictive_neanshhan
a 14
2 (predictive nean mean vals 1 ((1 / beta) np. predictive. 0.80061) **2)
0.00001 incumbent) / app.sar((reedcctive..u var)
beta_sqrt_p_z sqes
vals ** (z both))/)))rrddccciiu...rr
(1+ speta_srrt.p... / sart(preddicciiee.aa)))) predictive.var) #
1- пp (=sssr((prddicctiee.aa))+)))))))
vals += beta sqrt_____ p.2 / wp.sert(preeicctiee..aa)))
# predictive_var
+ (beta_sqrt_p_z / np. tpp.sar((prddicivcc.aa))))
predictive_var)ez
wals (1 (z / beta)**2) vals- (1 (z / beta)* mp.exp(- 1)
vals ap. sa predictive var) 0.80081)
np.sqrt(
predictive
vals 8= (np. & predictive_var) predictive_var)
vals #s 2redictive_varte2
vals[:num tointts // 2]
return np. argmax (vals)
Figure 19, FEW-SHOT, Python code for Osunto- The performance corresponding to this AF given Fig. (right).
we repeat the evaluation of the AF found for OOD-Bench using the standard BoTchch evaluation pipeline, as demonstrated in Colab notebooks closed-loop tutorial and custom acquisition tutorial.
We adhere to the settings specified in these Colab notebooks, thus we optimize the numerically at each trial, fit the hyperparameters of the GP model every trial using L-BFGS-B B, and randomly select initial points. For each test function, we ran the algorithm with 10 different random initial designs and plot the results by averaging over the runs for each function in DOD-Bench. Note that we use the exact plot formatting from BoTorch shown in the linked Colabs (which differs from the other plots in the manuscript). Fig. 20 shows how, across all functions, FunBO performs either comparably or better than El and UCB.
22

---

FunBO: Discovering Acquisition Functions for Bayesian Optimization with FunScarch
624
0.0030 Devond pere
nember of cosemnation Bepein inci points
Sphere
(h) Styblinski-Tamg
(c) Weierstrass
3.3
-350
-311
-1.5
(d) Beale
(e) Branin
(I) Michalewicz
-20
(g) Goldstein-Price
(h) Hartmann d
Hartmann
Figure 20. Performance comparison of FunBO (green) against Bl (orange) and UCB (red) on the test functions for OOD-Bench when using the BoTorch evaluation pipeline where AFS are optimized numerically each trial, hyperparameters are refitted each trial using L-BFGS-B, B, and initial points are selected randomly. Results are averaged over 10 runs with different random initial designs for cach function. Highlighted areas give standard deviation over the random initial designs.
23


Reviews:
[{"review_id": "TAHS1w7Lxm", "reviewer": "Reviewer_zkmL", "summary": "The paper proposes FunBO, a method to learn novel acquisition functions to increase efficiency of the Bayesian optimisation. In particular, acquisition function is represented as a Python program and FunSearch is adopted to search over programs' space. Authors evaluate their approach on both in-distribution settings, i.e., the target function is within the same class as given training functions, and out-of-distribution settings. It is shown that FunBO outperforms the considered baselines.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for your feedback and for highlighting the strengths of our work.  We hope the responses below clarify the motivation and practical considerations for FunBO.\n\n**Benefits of Representing AFs as Programs** Representing AFs as programs offers the following advantages:\n- *Interpretability*: As highlighted in our paper, code-based AFs are inherently interpretable. It is possible to directly inspect, analyze, and understand the logic behind the optimization strategy employed by the discovered AF. This contrasts with neural network based AFs, for which it is difficult to understand why they perform well or how they balance exploration/exploitation. \n- *Deployability and Simplicity*: AFs discovered by FunBO are outputted as concise code snippets (e.g., Python functions). As mentioned in Section 1, these can be readily integrated into existing BO libraries and workflows with minimal overhead.\n- *Leveraging Foundational Knowledge*: LLMs trained on vast code and text have knowledge about code but also about existing BO strategies. FunBO aims at leveraging this embedded knowledge to efficiently explore the program space and construct effective heuristics.\n\nWe have further clarified the benefits of the code representation by adding text elaborating on these advantages (Interpretability, Deployability, and Leveraging Foundational Knowledge) around line 92 in Section 1.\n\n**Practical Use Cases**  In the limitation section we mentioned the fact that FunBO's evaluation cost might be high during the discovery phase as it involves running a full BO loop for each candidate AF on functions in G. However, there are scenarios where the evaluation cost for the functions in G might relatively cheap. For instance:\n- *Simulations/Models at Lower Fidelity*: G could consist of faster, lower-fidelity simulators or simplified analytical models related to the expensive target problem f.\n- *Surrogate Models*: G could include cheap surrogate models learned from previous related tasks. \n- *ML Hyperparameter Optimization*. G can represent the performance of a ML  model on different (potentially smaller) datasets, where training/evaluation might be faster than on the final, large target dataset.\n\nNote that the computational cost incurred during this one-time search phase is compensated during the online deployment phase where the resulting discovered AF reduces the number of costly function evaluations needed.\n\nWe have expanded the discussion on the computational cost within the Limitations paragraph in Section 5 and gave the examples listed above as cases where the auxiliary functions used during the search phase might be relatively inexpensive.", "claims_and_evidence": "The experiments convincingly show the advantages of the proposed FunBO method.", "methods_and_evaluation_criteria": "FunBO is evaluated on diverse range of functions, covering both in-distribution and out-of-distribution settings.", "theoretical_claims": "N/A", "experimental_designs_or_analyses": "Seems valid.", "relation_to_broader_scientific_literature": "I think the problem of learning problem-specific acquisition functions is important problem. In general, Bayesian optimisation arises in many applications, thus many fields could benefit from FunBO that improves the efficiency of the optimisation.", "essential_references_not_discussed": "N/A", "other_strengths_and_weaknesses": "N/A", "other_comments_or_suggestions": "N/A", "questions_for_authors": "1. I am curious about the intrinsic motivation of representing AFs with programs? What are the real benefits behind it? Representing them as programs leads to the requirement of doing hard discrete search, thus making the overall procedure even more expensive. Overall, I understand that it comes from the fact that modern LLMs became good priors for code generation and can generate \"executable\" programs, thus giving good start for such search, but besides that, what are the benefits compared to representing AF via neural network?\n\n2. As you also mention in the limitations, FunBO might be really strong in the situations where running full BO loop is very cheap/fast to provide the feedback to an LLM, then FunBO can iterate quickly. What are the interesting practical use-cases for such setting?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "Qa2Dq2uaph", "reviewer": "Reviewer_X7CF", "summary": "&nbsp;\n\nThe authors present FunBO, an algorithm that leverages an LLM to propose novel acquisition functions (AFs) for Bayesian optimization. The authors evaluate FunBO under three settings, in-distribution (ID), out-of-distribution (OOD), and few-shot. FunBO relies on the availability of a large set of related tasks (ID) or unrelated tasks (OOD) with which to evaluate new AF candidates. Overall, although the method is interesting, I have some concerns about the empirical evaluation of FunBO. First, the experiments are run using fixed hyperparameters for the GP surrogate and without acquisition function maximization which doesn't reflect the real-world application of Bayesian optimization. Second, it would be nice to see FunBO evaluated on more real-world black-box optimization problems such as molecule generation [13] or on a standard hyperparameter optimization benchmark [11]. Third, the fact that random search outperforms other AFs on standard optimization benchmarks in the authors' experiments  seems to contradict the results of previous studies and raises some questions about the empirical evaluation which are not directly answerable because the authors have not provided their source code. Nonetheless, I will be willing to revise my score if the authors can address the concerns raised.\n\n&nbsp;\n\n## **Post-Rebuttal Update**\n\n&nbsp;\n\nUnfortunately, the authors did not have sufficient time to reply to my final comment and supply either the code or the full details of their additional experiments such that they can be reproduced. My primary concern hence remains that the empirical results are somewhat suspicious given that e.g. the performance of random search against standard AFs is at odds with prior literature. During the AC-reviewer discussion phase I will try my best to independently verify the results in BoTorch. If the authors could supply the mathematical form of the FunBO acquisition function they evaluated as a confidential comment to the AC to be disseminated to me that would be great.\n\n&nbsp;", "questions": "", "limitations": "", "rating": 2, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for your in-depth review of our work and valuable feedback. \n\n**Transformer Cost** We agree that re-training large transformer architectures is expensive. Our statement was intended to mean that a pre-trained transformer surrogate model could benefit from using a cheap-to-evaluate, FunBO-discovered AF instead of standard AFs (like EI, as used in Chen et al. 2022), rather than implying FunBO would be involved in the transformer training itself. \n\n**AF/HP optimization** We removed the local maximization step from the MetaBO baseline to ensure a consistent comparison framework where all AFs are optimized over the same Sobol grid. This was done to isolate the performance differences attributable purely to the AF's selection logic, removing confounding effects from the quality of optimization routines. For the same reason, we used fixed GP hyperparameters. We aimed for a controlled comparison of the AFs' inherent selection strategies and believe this approach leads to an evaluation that is sound and appropriate as recognized by Reviewer FFHS (“The experimental design carefully controls for confounding factors by maintaining consistent GP hyperparameters …”). We will clarify that this might deviate from a fully realistic deployment scenario.\n\n**G cheaper than f** FunBO can run even if functions in $G_{\\text{Tr}}$ are expensive, but the offline discovery phase becomes proportionally costly. The practicality of FunBO relies on the idea that functions in $G_{\\text{Tr}}$ are indeed cheaper (e.g., lower fidelity simulations, see response to Reviewer zkmL) and the large offline cost is amortized by finding an AF that saves numerous evaluations of the truly expensive target function f during online deployment.\n\n**Questions**:\n1) This refers to the AF signature where `predictive_mean` and `predictive_var` are NumPy arrays, as opposed to simpler functions found by FunSearch that might only take scalar or tuple inputs.\n2) This occurs if $|G|$ is small and/or if the user decides to use all available functions for training to maximize the training signal e.g. in few shot settings.\n3) Using a completely random initial AF encourages the LLM to generate AF code that itself included stochastic elements (e.g., various calls to `np.random`). This makes the resulting BO loop non-deterministic, making scoring inconsistent and reproducibility difficult. \n4) The mean and standard deviation shown in the regret plots represent the performance variation across the set of test functions (e.g 9 distinct test functions for OOD-Bench), not across multiple independent runs of the BO algorithm on a single function instance.  We used std/2 for visualization purpose.\n5) Thanks for pointing us to these references. Notice that the figures you mention are averaging the convergence results over multiple random initializations. In our case we are averaging the convergence results over multiple, completely different, functions. Plotting the performance on the single functions reveals that known AFs outperform random search in most cases, performing better for lower dimensional functions and being competitive on functions with numerous local optima. \n6) This refers to the limited number of functions available in G. We will rephrase this to be “by leveraging access to a number of evaluations for a *limited* set of objective functions.”\n7) We omitted them to avoid clattering the figure but we are happy to add them back to the manuscript. \n\n\nOther comments:\n1) For Fig 4 (right panel, HM3), MetaBO is indeed outperforming FunBO. However, in the statement you report we refer to “general purpose AF” (e.g. EI) that are indeed underperforming compared to FunBO.\n2) This is a typo. $T_{h_{j}^{\\tau}}$​ represents the number of BO trials required by the algorithm using AF $h_j^{\\tau}$ to identify the true minimum value $y_j^*$​. This is recorded for each AF $h_j^{\\tau}$ sampled at step $\\tau$. Note that also $x^*_{j,h^\\tau} $ should be $x^*_{j,h_j^\\tau}$. \n3) The sampling scheme details are given in Appendix C and they are kept fixed with respect to FunSearch. The code for the latter is available on [GitHub](https://github.com/google-deepmind/funsearch/blob/main/implementation/programs_database.py) making the algorithm reproducible. \n4) As specified on line 312, all experiments are conducted using FunSearch with default hyperparameters, including the number of examples in the prompt.\n5) Thank you for providing this helpful list of suggested references.  We will incorporate them in the revised manuscript and add a related discussion.  Regarding symbolic regressions, these methods also searches for mathematical expressions, often represented as trees and evolved using genetic operators. FunBO differs by operating directly on code representations, leveraging an LLM for generation/mutation, and using full BO performance as the fitness metric. While related in the goal of finding functional forms, the search space representation and operators are different.", "claims_and_evidence": "&nbsp;\n\nIn terms of the claims made in the paper the authors state:\n\n&nbsp;\n\n>\"Across all objective functions, αFunBO leads to a convergence performance that outperform general purpose AFs (Fig. 4).\"\n\n&nbsp;\n\nIt seems as though MetaBO is stronger than FunBO on Hartmann?\n\n&nbsp;\n\nIn the experiment section the authors claim in relation to the performance of random search that, \n\n&nbsp;\n\n>\"This is due to random search performing competitively on functions with numerous local optima, which are generally harder to optimize.\"\n\n&nbsp;\n\nThis does not seem to be borne out in the literature. I discuss this in more detail below. In the Related Work section, the authors state, \n\n&nbsp;\n\n>\"This confirms the continued importance of AFs as crucial components in BO, even when combined with transformer-based approaches, and highlights the importance of a method such as FunBO that can be seamlessly integrated with these newer architectures, potentially leading to further improvements in performance.\"\n\n&nbsp;\n\nRetraining a transformer architecture is likely to be very expensive. Will such an integration not fall foul of the limitations highlighted in the authors' conclusion regarding the expense of running FunBO?\n\n&nbsp;", "methods_and_evaluation_criteria": "&nbsp;\n\n1. It looks as though Equation 1 is underspecified. From the authors' description in the paragraph below, it is implied that the number of steps tao required to identify the optimum is recorded (otherwise at a fixed budget T, the second term would always be 1) yet this isn't indicated in Equation 1.\n\n2. The authors state that, they removed the local maximization of AFs from the MetaBO baseline. Why was this done? Presumably because the authors' omit AF maximization in FunBO?\n\n3. In their experiments the authors state,\n\n&nbsp;\n\n>\"To isolate the effect of using different AFs and eliminate confounding factors related to AF maximization or surrogate models, we maximized all AFs on a fixed Sobol grid (of size NSG) over each function’s input space.\"\n\n&nbsp;\n\nSurely the relevant comparison is whether FunBO can yield better acquisition functions in real-world problems where acquisition function maximization and hyperparameter tuning of the surrogate model would take place? It is colloquially accepted that the quality of the GP surrogate is a more important factor than the choice of acquisition function [10, 12] in real-world BO performance and so it would make sense to run an empirical evaluation where GP hyperparameter optimization and AF maximization were present. One option would be to take a SOTA BO algorithm such as HEBO [10] and evaluate whether FunBO could improve its performance on the Bayesmark benchmark [11].\n\n&nbsp;", "theoretical_claims": "&nbsp;\n\nNot applicable.\n\n&nbsp;", "experimental_designs_or_analyses": "&nbsp;\n\n1. The method hinges on the assumption that the related functions G are much cheaper to evaluate relative to f?\n\n2. Why did the authors choose an RBF kernel in place of a Matern 3/2 kernel as is commonly adopted for standard optimization problems.\n\n3. OOD-Bench seems to be the most practically relevant benchmark. For this experiment it would be nice (but expensive) to see the contents of G_tr and G_v rotated with optimization of GP hyperparameters. \n\n4. Although MetaBO uses fixed GP hypers and the authors may have been concerned with a direct comparison, I reiterate that I believe a set of experiments that would support the real-world utility of FunBO would feature optimization of the GP hypers and maximization of the AF.\n\n5. The authors use two example AFs in the prompt. How was this number chosen? Was there a sensitivity analysis on the number of AFs to include? Do the authors think the number of examples affects performance?\n\n&nbsp;", "relation_to_broader_scientific_literature": "&nbsp;\n\n1. It may be worth briefly discussing connections between FunSearch and OPRO [7]. While FunSearch maintains scores for each heuristic, OPRO maintains scores for each prompt.\n\n2. How does the authors' approach differ from symbolic regression approaches uses genetic programming? Did the authors consider this?\n\n&nbsp;", "essential_references_not_discussed": "&nbsp;\n\n1. When introducing Bayesian optimization, it may be worth citing the originating papers for the method [2, 3] as discussed in [4].\n\n2. Reference [5] should be cited when introducing Expected Improvement (EI) as discussed in [4].\n\n3. For probability of improvement, the originating work is [2] and for knowledge gradient the originating work is [5] as discussed in [4].\n\n4. In the Related Work, it would be worth the authors mentioning the references [14, 15] which both leverage LLM embeddings for black-box optimization.\n\n5. InstructZero was accepted at ICML 2024.\n\n&nbsp;\n\n**__REFERENCES__**\n\n&nbsp;\n\n[1] Benjamins, C., Raponi, E., Jankovic, A., Doerr, C. and Lindauer, M., 2023, December. [Self-Adjusting Weighted Expected Improvement for Bayesian Optimization.](https://proceedings.mlr.press/v224/benjamins23a) In International Conference on Automated Machine Learning (pp. 6-1). PMLR.\n\n[2] Kushner, HJ., [A Versatile Stochastic Model of a Function of Unknown and Time\nVarying Form](https://www.sciencedirect.com/science/article/pii/0022247X62900112). Journal of Mathematical Analysis and Applications 5(1):150–167. 1962.\n\n[3] Kushner HJ., [A New Method of Locating the Maximum Point of an Arbitrary Multipeak Curve in the Presence of Noise](https://asmedigitalcollection.asme.org/fluidsengineering/article-abstract/86/1/97/392213/A-New-Method-of-Locating-the-Maximum-Point-of-an?redirectedFrom=fulltext). Journal of Basic Engineering 86(1):97–106. 1964.\n\n[4] Garnett, R., [Bayesian optimization](https://bayesoptbook.com/). Cambridge University Press. 2023.\n\n[5] Saltines, VR., One Method of Multiextremum Optimization. Avtomatika i Vychislitel’naya Tekhnika (Automatic Control and Computer Sciences) 5(3):33–38. 1971.\n\n[6] Lyu, W., Yang, F., Yan, C., Zhou, D. and Zeng, X., 2018, July. [Batch Bayesian optimization via multi-objective acquisition ensemble for automated analog circuit design.](https://proceedings.mlr.press/v80/lyu18a.html?ref=https://githubhelp.com) In International conference on machine learning (pp. 3306-3314). PMLR.\n\n[7] Guo, Q., Wang, R., Guo, J., Li, B., Song, K., Tan, X., Liu, G., Bian, J. and Yang, Y., [Connecting Large Language Models with Evolutionary Algorithms Yields Powerful Prompt Optimizers.](https://openreview.net/forum?id=ZG3RaNIsO8) In The Twelfth International Conference on Learning Representations.\n\n[8] Daulton, S., Ament, S., Eriksson, D., Balandat, M. and Bakshy, E., 2023, December. [Unexpected improvements to expected improvement for Bayesian optimization.](https://proceedings.neurips.cc/paper/2023/hash/419f72cbd568ad62183f8132a3605a2a-Abstract-Conference.html) In Proceedings of the 37th International Conference on Neural Information Processing Systems (pp. 20577-20612).\n\n[9] Balandat, M., Karrer, B., Jiang, D., Daulton, S., Letham, B., Wilson, A.G. and Bakshy, E., 2020. [BoTorch: A framework for efficient Monte-Carlo Bayesian optimization.](https://proceedings.neurips.cc/paper/2020/hash/f5b1b89d98b7286673128a5fb112cb9a-Abstract.html) Advances in neural information processing systems, 33, pp.21524-21538.\n\n[10] Cowen-Rivers, A.I., Lyu, W., Tutunov, R., Wang, Z., Grosnit, A., Griffiths, R.R., Maraval, A.M., Jianye, H., Wang, J., Peters, J. and Bou-Ammar, H., 2022. [HEBO: Pushing the limits of sample-efficient hyper-parameter optimisation.](https://www.jair.org/index.php/jair/article/view/13643) Journal of Artificial Intelligence Research, 74, pp.1269-1349.\n\n[11] Turner, R., Eriksson, D., McCourt, M., Kiili, J., Laaksonen, E., Xu, Z. and Guyon, I., 2021, August. [Bayesian optimization is superior to random search for machine learning hyperparameter tuning: Analysis of the black-box optimization challenge 2020.](https://proceedings.mlr.press/v133/turner21a.html) In NeurIPS 2020 Competition and Demonstration Track (pp. 3-26). PMLR.\n\n[12] Shahriari, B., Swersky, K., Wang, Z., Adams, R.P. and De Freitas, N., 2015. [Taking the human out of the loop: A review of Bayesian optimization.](https://ieeexplore.ieee.org/abstract/document/7352306) Proceedings of the IEEE, 104(1), pp.148-175.\n\n[13] Gao, W., Fu, T., Sun, J. and Coley, C., 2022. [Sample efficiency matters: a benchmark for practical molecular optimization.](https://proceedings.neurips.cc/paper_files/paper/2022/hash/8644353f7d307baaf29bc1e56fe8e0ec-Abstract-Datasets_and_Benchmarks.html) Advances in Neural Information Processing Systems, 35, pp.21342-21357.\n\n[14] Kristiadi, A., Strieth-Kalthoff, F., Skreta, M., Poupart, P., Aspuru-Guzik, A. and Pleiss, G., 2024, July. [A Sober Look at LLMs for Material Discovery: Are They Actually Good for Bayesian Optimization Over Molecules?.](https://proceedings.mlr.press/v235/kristiadi24a.html) In International Conference on Machine Learning (pp. 25603-25622). PMLR.\n\n[15] Rankovic, B., Schwaller, P., BoChemian: [Large language model embeddings for Bayesian optimization of chemical reactions.](https://openreview.net/forum?id=A1RVn1m3J3) In: NeurIPS 2023 Workshop on Adaptive Experimental Design and Active Learning in the Real World. 2023.\n\n&nbsp;", "other_strengths_and_weaknesses": "&nbsp;\n\n1. To my mind the experiments performed by the authors although extensive do not contain enough real-world examples. A hyperparameter optimization problem with two hyperparameters e.g. the SVM experiments is not particularly compelling. It would be more interesting to see the method applied to a setting such as molecule generation [13] for example or the Bayesmark benchmark [11]. If these problems are prohibitive from a cost perspective it would be nice to see this limitation explicitly mentioned in relation to these applications (instead of abstractly as is currently the case in the conclusion).\n\n2. It would be great if the code could be released. If there is a delay on industrial approval for code release this will hopefully have been addressed in the time between submission and the rebuttal phase.\n\n&nbsp;", "other_comments_or_suggestions": "&nbsp;\n\n1. There are some missing capitalizations in the references e.g. \"Bayesian\".\n\n2. There are some missing journal/conference indications for arXiv papers in the references e.g. [1] was published at the AutoML conference.\n\n3. Typo in the introduction, \"In other to avoid\".\n\n4. When listing approaches to mitigate reliance on a single acquisition function in the introduction it would also be worth mentioning ensemble approaches such as MACE [6] where the Pareto front over different acquisition functions is computed.\n\n5. In the preliminaries section, it would be worth articulating the purpose of the auxiliary objective functions g_j. It would also be worth articulating any assumptions about the relation between these auxiliary functions g and the objective to be optimized f. The authors later state that, \"The learned AFs are trained on the set G, whose functions are assumed to be drawn from the same distribution or function class associated to f.\" For clarity it would be worth including this assumption when the set G is first introduced.\n\n6. PI would probably be a less cumbersome acronym than PofI?\n\n7. Typo in Figure 6 docstring, \"finds its optimum\"\n\n8. Typo in Table 1 and Table 2, the likelihood noise parameter sigma (1e-5) should not have a subscript f.\n\n9. Typo section 3, \"an algorithm discovery problem\".\n\n10. The clarity of the description of the terms in Equation 1 could be improved e.g. $y^{*}\\_j$ is the known optimal value of the objective $g_j$ rather than the optimum which would be the corresponding $x_j$. The authors describe $x_{j}^{t=0}$ as the optimal input value at $t=0$. What does this mean? The optimal value according to the surrogate model at $t=0$? The authors also describe the \"found optimal input value\". I think this would be better described as the model's estimate of the optimal input value or similar.\n\n11. In Figure 2, it would help if scores were indicated for each acquisition function to highlight the ascending order in which they appear in the prompt.\n\n12. In the definition of regret on page 6 it is somewhat confusing to use $y*$ as the notation for the true optimum. Typically $f(x)$ refers to the noiseless value of the objective and $y$ refers to a noisy value of the objective.\n\n13. On page 6, the authors state, \"with less than 25 trials\". This should be \"with fewer than 25 trials\". \"Fewer\" is typically used with countable nouns such as trials, oranges, ICML submissions, whereas \"less\" is used with uncountable nouns such as time, money, and water.\n\n14. In Figure 8, the variable **percentage_steps_needed** is not defined.\n\n15. Line 746, Figure 7, is there a typo? **found_min = initial_min_y = float(np.min(model.Y))**.\n\n16. Line 762, Figure 7, comment is broken.\n\n17. Line 764, Figure 7, **if found_min == true_min**. Should there not be some tolerance factor here if this is real Python code?\n\n18. Line 793, using $\\mathbf{X}$ is a slightly strange way of defining the RBF kernel if $\\mathbf{X}$ is the design matrix. Why not use the standard $\\mathbf{x}$ notation?\n\n19. Can the authors explain line 948 in Figure 12, \"predictive_var [(shape−10)//2] *= dim\".\n\n&nbsp;", "questions_for_authors": "&nbsp;\n\n1. The authors state, \"FunBO explores a significantly more complex function space where programs take as inputs multiple arrays.\". Can the authors give an example of what they mean by \"multiple arrays\"?\n\n2. The authors state that, \"When no validation functions are used ($G = G_{Tr}$), the AF with the highest average performance on $G_{Tr}$ is returned.\". When would the case arise where no validation functions are available?\n\n3. The authors state in footnote 6, \"We explored using a random selection of initial points as an alternative to EI. However, this approach did not yield good results as using a random selection was incentivizing the generation of functions with a stochastic output, for which convergence results are not reproducible.\". What do the authors mean by this? Can the authors present these negative results?\n\n4. For the plot in Figure 2, how many random seeds were taken? Why is std/2 used in place of the standard error?\n\n5. In Figure 9, the legend labels are very difficult to read. It seems as though random search is the next best performing acquisition function relative to FunBO? I see the authors later mention the strong performance of random search but the explanation isn't particularly convincing given results reported for the same optimization benchmarks in other papers e.g. in [8] the authors find that random search is the worst-performing AF on the Michaliewicz function in Figure 3. Additionally, in [9] Figure 6, the authors find that random search is the worst performing AF on the Hartmann 6D function (in the parallel BO setting with a batch size q=4). Even if the experimental settings are slightly different the overall trend is still very surprising. Additionally, how many random seeds were taken. Why is the half standard deviation used for errorbars in place of the standard error?\n\n6. In the abstract the authors state, \"we propose FunBO, an LLM-based method that can be used to learn new AFs written in computer code by leveraging access to a limited number of evaluations for a set of objective functions.\". What is meant by this? The number of evaluations seems quite large when aggregated across the set of related functions G required to evaluate the efficacy of the proposed AF.\n\n7. In Figure 5 why isn't random search shown in the first two panels?\n\n&nbsp;", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {"ethical_review_concerns": "&nbsp;\n\nNot applicable.\n\n&nbsp;"}}, {"review_id": "gZSggHkgKd", "reviewer": "Reviewer_aMJb", "summary": "The authors propose FunBO, a novel method for designing novel acquisition functions (AFs) for BO using LLMs. FunBO aims to design new AFs that perform well across a variety of objective functions. FunBO takes a target set of objective functions and some initial standard AF. Then, FunBO iteratively modifies the initial AF to improve the resultant BO performance on the set of objective functions. The result is a novel AF designed specifically to perform well across the given set of objective functions. The novel AF is output directly in code format, making the designed AF both easy to use and interpret. The authors provide extensive empirical analysis of FunBO, comparing BO performance with AFs designed by FunBO, to performance with both standard general purpose AFs, and with AFs customized specifically for the given objective function type (function-specific AFs). Results show that AFs designed by FunBO outperform general purpose AFs, and perform comparably with the function-specific AFs. Additionally, results show that AFs designed by FunBO perform well even on objective functions that were outside of the FunBO training distribution, demonstrating that the AFs designed by FunBO generalize well to other objective functions. Empirical results include both standard benchmark objective functions and hyperparameter optimization tasks.\n\n## update after rebuttal\nAs I stated in my comment below, the authors sufficiently answered all of my questions in their rebuttal, and I maintain that this work should be accepted for all of the reasons stated in my initial review.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for your feedback and for highlighting the strengths of our work. We appreciate the constructive comments and recommendation for acceptance. Regarding the specific questions:\n\n**Application to Other Real-World Problems** This is an excellent point. Our primary goal in this initial work was to establish the FunBO methodology itself, demonstrating its feasibility for discovering novel AFs from code and rigorously evaluating their performance and generalization capabilities using well-understood application domain (standard benchmarks and simple HPO problems). Exploring diverse real-world applications would require careful consideration of the specific problem characteristics and potentially curating relevant sets G representative of the target domain. We agree this is a significant and exciting direction, and we view it as important future work, building upon the foundation laid in this paper.\n\n**Incorporating Problem-Specific Information** Currently, FunBO provides context to the LLM through the structure of the initial AF, the function docstrings and the prompt structure including prior examples. Building on this, there are indeed several ways the prompt context could be further enriched:\n- *Adding Performance Data*: We could go beyond the implicit ranking in the prompt and explicitly include the numerical scores attained by the included AFs. This might provide the LLM with a clearer signal about the magnitude of improvement to aim for.\n- *Adding Self-Critique*: Another avenue, inspired by self-improvement techniques, could involve incorporating automatically generated or human-written critique/analysis of why the provided example AFs perform as they do directly into the prompt. This could potentially help the LLM focus on refining successful strategies or avoiding pitfalls.\n- *Adding Problem Descriptions*: Incorporating explicit textual descriptions of the problem domain (e.g., \"tuning hyperparameters for a binary classification model\" or \"optimizing a robotic grasp\"), as proposed. \n\nFrom an implementation perspective these can be easily incorporated in the prompt structure used by FunBO. It is plausible that providing richer information through any of these additions could allow the LLM to better leverage its internal knowledge base, potentially leading to further improvements in the discovered AFs. The exploration of the impact of richer prompt contexts (including scores, critique, or domain descriptions) is an interesting direction for future investigation, as mentioned in our Future Work section (Section 5).", "claims_and_evidence": "Yes.", "methods_and_evaluation_criteria": "Yes.", "theoretical_claims": "There are no theoretical claims or proofs in this paper as far as I am aware.", "experimental_designs_or_analyses": "Yes, I checked all experimental design setups and experimental results provided in the paper. All are valid as far as I am aware.", "relation_to_broader_scientific_literature": "Designing AFs for BO is a relevant problem because choosing or designing a good acquisition for a specific objective function can have a significant impact on optimization performance. Additionally, different AFs can be better or worse choices for different objective functions. New methods for designing good AFs that perform well across a variety of objective functions will be of interest to the broader scientific community, and especially to the Bayesian optimization research community.", "essential_references_not_discussed": "There are no essential references missing as far as I am aware.", "other_strengths_and_weaknesses": "Strengths: \n\n1: Improving accessibility of BO methods for practitioners: Designing AFs to maximize BO performance is a difficult problem that often requires advanced knowledge about Bayesian optimization. FunBO presents a way for practitioners in other fields (i.e. the natural sciences), who may want to apply BO methods to their specific problems, to design AFs without this advanced knowledge. Additionally, the fact that FunBO outputs designed AFs directly in python code makes this extremely user friendly (it’s as simple as copying and pasting the designed AF into the user’s .py file). \n\n2: Compelling empirical results: The empirical results provided are compelling and clearly demonstrate the author’s claims that 1) AFs designed by FunBO outperform general purpose AFs, 2) AFs designed by FunBO perform comparably even when compared to function-specific AFs, and 3) AFs designed by FunBO generalize well to objective functions outside of the training distribution. \n\nWeaknesses: \n\nA minor weakness of this paper is that the novel methodological contribution is on the smaller side. This is because FunBO builds on ideas from existing work (FunSearch) and applies these ideas to the new application of designing AFs. That being said, this is a very minor weakness because significant work was done to apply these ideas to this new application setting of designing AFs, and strong empirical results demonstrate that FunBO makes an impactful contribution to the literature. I therefore still recommend that this paper be accepted.", "other_comments_or_suggestions": "Typos: \n\nLine 44: “In other to avoid” should be “In order to avoid”\n\nLine 319: “that performs well across function classes” should be “that perform well across function classes” \n\nLine 356: “FunBO is able reach” should be “FunBO is able to reach”", "questions_for_authors": "Question 1: \nHave the authors considered applying FunBO to design AFs for additional real optimization problems outside of hyperparameter optimization? While hyperparameter optimization is one of the most common applications of BO, there is much recent work applying BO to various interesting problems in robotics, materials science, database tuning, drug design, nuclear reactor stability, etc. It would be cool to see if FunBO  could be applied to design AFs to improve BO for some of these other kinds of practical applications. \n\nQuestion 2: \nI wonder if additional information specific to the particular problem setting could also be provided to FunBO to further improve the AF design. For example, adding problem descriptive information such as “this is a problem in hyperparameter tuning” or “this is a problem in robotics” could be optionally provided to the LLM to allow FunBO to incorporate this information in the design of the AF. Is this something the authors have considered? Do the authors think this would be straightforward to do, and do you think that it could further improve performance of FunBO?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "cJWTtx8xxf", "reviewer": "Reviewer_FFHS", "summary": "This paper introduces FunBO, Building on the FunSearch framework to discover novel acquisition functions for BO. FunBO iteratively modifies an initial acquisition function to improve the BO performance. The discovered AFs are provided as explicit code snippets, enhancing interpretability and facilitating direct deployment without additional overhead. The method evaluates candidate AFs based on how accurately and efficiently they identify the global optimum of a set of objective functions.\n\nThe authors show that FunBO discovers AFs that generalize well both within specific function classes and across different types of functions. These AFs outperform established alternatives like EI and UCB, while achieving competitive performance against specialized AFs designed for specific function types.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for your in-depth review of our work and valuable feedback. \n\n**Missing Baselines** Our work focuses (footnote 2) on AFs that can be evaluated in closed form given GP posterior parameters thus being fast-to-evaluate and easily deployable while avoiding complexities like Monte-Carlo sampling (e.g., for entropy/KG methods). This is a deliberate choice to avoid the complexities of approximations (e.g., MC sampling), which would make both the evaluation of candidate AFs within FunBO computationally intractable and the generation of corresponding code by the LLM significantly more difficult.\n\nAs we restricted FunBO's search space to focus on discovering AFs within the class of closed-form deterministic functions, we *ensured a fairer comparison* by contrasting the discovered AFs against appropriate baselines like EI/UCB/PoI.\nIn terms of end-to-end methods, we view FunBO as complementary. As noted in our Related Work section, Chen et al. (2022) found that even their transformer-based model benefited from using a standard AF (EI) for action selection. This suggests that AFs discovered by FunBO could potentially be integrated into such end-to-end frameworks in the future to further boost performance.\n\n**Computational Cost & Variance** We acknowledge that the computational cost of FunBO is significant. Note however that FunBO is primarily intended as an offline procedure to discover high-quality AFs. The computational expense is incurred during this one-time search phase. Once discovered, the resulting code-based AF is typically very cheap to evaluate during the actual online BO task. The idea is that FunBO invests computation upfront to find an AF that significantly improves sample efficiency later.  \nWhile variance in the quality of the discovered AFs might further increase the computational cost, it's crucial to distinguish between the variance of the search process from the value of its outcome. Once found, a successful AF discovered by FunBO has an intrinsic value that is independent of the stochastic process that generated it. The AF represents a novel optimization strategy whose performance can be evaluated and used independently of the original search. \n\n**Discrete AF Optimization** Within our experimental evaluation, the optimization of the AF was performed over a discrete Sobol grid to ensure fair comparison across all AFs. However, the discovered AFs are not fundamentally restricted to discrete optimization. As long as the surrogate model is differentiable (like a GP with RBF kernel) and the generated code includes differentiable functions, the AF can be optimized using continuous methods, similarly to EI or UCB.  For instance, we could rewrite the AF in Fig 2 to take the input location x, return the corresponding scalar acquisition value and optimize it via L-BFGS. \n\n**Robustness to Surrogate Model** Our current work focused on discovering AFs conditioned on a specific surrogate model, as stated in Sec 4, to isolate the AF's contribution. The performance of the discovered AFs might indeed depend on this choice. However, the FunBO methodology itself is flexible.  By simply modifying the `run_bo` implementation to use a different surrogate model, FunBO can be used to discover AFs tailored to the specific model's properties. \n\n**Interpretation of Discovered AFs** Note that we included an AF interpretation on line 287 and commented on the Goldstein-Price AF on line 359.  As an additional example, the AF found for GPs-ID can be simplified to be written as `(EI ** 2) / (1 + (z / beta)**2 * sqrt(var))**2`.  This function calculates the standard EI, squares it, and then divides it by a penalty term that increases with both the standardized improvement `z = (incumbent - mean) / std` and the uncertainty `sqrt(var)`.  The squaring of the EI non-linearly amplifies regions with high EI values relative to those with low or moderate EI. It increases the \"peakiness\" of the acquisition function, leading to stronger exploitation of the most promising point(s). The term `(1 + (z/beta)**2 * sqrt(var))**2` penalizes points more heavily if they have a very high expected improvement (z is large and positive) and/or high uncertainty (`sqrt(var)` is large). This might act as a regularizer against over-optimism or excessive jumps into highly uncertain areas, even if they look very promising according to EI. It's a novel way of balancing exploration and exploitation discovered by FunBO. \n\n**Number of Repetitions** The mean and standard deviation shown in the regret plots represent the performance variation across the set of test functions, not across multiple independent runs of the BO algorithm on a single function instance. For instance, for OOD-Bench, the average/std dev is over the 9 distinct test functions. We will clarify this point in the captions.", "claims_and_evidence": "The paper's primary claim is that FunBO can discover explicit, code-based acquisition functions that outperform traditional options like EI and UCB. The experimental results do support this claim across various settings - the discovered AFs generally perform better than standard acquisition functions in most of the tests.\n\nHowever, the evidence to me is not strong enough. While the authors show performance improvements over traditional AFs, the margins of improvement vary considerably across different test functions, and in some cases appear modest.\n\nThe paper makes a strong case for interpretability - providing the actual code for all discovered AFs is compelling evidence for this aspect. But regarding performance claims, a more comprehensive comparison against state-of-the-art methods would have strengthened the paper's position. The authors acknowledge high variance in the quality of discovered AFs, which raises questions about the consistency and reliability of the approach without multiple runs.", "methods_and_evaluation_criteria": "The evaluation method is generally sound and appropriate in BO:\n\n* The authors use standard benchmark functions and synthetic GP functions that are common in the BO literature.\n* The use of normalized average simple regret as the primary metric is standard.\n* The experimental design carefully controls for confounding factors by maintaining consistent GP hyperparameters, evaluation grids, and initial designs across all methods.\n\nHowever, I think the major limitation is the selection of baselines. The authors only compare against classical AFs (EI, UCB, PofI) and some recent meta-learning approaches (MetaBO, FSAF), they omit comparisons with several important recent advances:\n\n* More advanced AFs such as entropy-based methods (e.g., Max-Value Entropy Search), which have shown superior performance in many settings.\n* End-to-end methods that jointly learn surrogate models and acquisition strategies.\n\nThis gap makes it difficult to assess how FunBO compares to the current state-of-the-art beyond traditional methods. While the focus on comparing against interpretable baselines is understandable given FunBO's emphasis on interpretability, including at least some of these more advanced methods would have provided a more complete picture of FunBO's competitive standing in the broader landscape of modern BO techniques.", "theoretical_claims": "The paper does not make formal theoretical claims requiring proof verification.", "experimental_designs_or_analyses": "The experimental design appears methodologically sound.", "relation_to_broader_scientific_literature": "FunBO potentially addresses challenges in scientific optimization domains where traditional AFs struggle. This approach could particularly benefit high-dimensional problems, multi-objective optimization scenarios, or constrained optimization settings that are prevalent in scientific applications like drug discovery, materials science, and engineering design.", "essential_references_not_discussed": "The literature review appears comprehensive, covering classical AFs, meta-learning approaches, and LLM-augmented BO methods. The authors also acknowledge concurrent work by Yao et al. (2024) on representing AFs in code. \n\nI would only suggest authors include several other meta-learning BO papers [1-3], and one LLM BO paper [4].\n\n[1] Song et al. \"Reinforced in-context black-box optimization.\"\n\n[2] Müller et al. \"Pfns4bo: In-context learning for bayesian optimization.\"\n\n[3] Yang et al. \"MONGOOSE: Path-wise smooth Bayesian optimisation via meta-learning.\"\n\n[4] Song et al. \"Position: Leverage foundational models for black-box optimization.\"", "other_strengths_and_weaknesses": "* The computational cost of FunBO is prohibitively high (48 hours of training), which severely limits its practical utility. This investment only makes sense when repeatedly optimizing similar functions, but in most real applications, trying several standard AFs would be much faster than running FunBO.\n* While the authors emphasize interpretability as a key advantage, they don't actually interpret any of their discovered AFs. I would have expected at least one example where they explain why a particular code structure performs better than traditional AFs on a specific task. Without such analysis, the claimed interpretability benefit remains largely theoretical.\n* The current implementation seems limited to discrete evaluation points (Sobol grid). It's unclear how the approach would extend to true continuous optimization of acquisition functions, which might be important for some applications.\n* As the authors acknowledge, there's high variance in the quality of discovered AFs, requiring multiple FunBO runs to find good solutions. This inconsistency further increases the already substantial computational demands.", "other_comments_or_suggestions": "* There appears to be a duplication in the related work section - the paragraph on \"LLMs and black-box optimization\" is repeated almost verbatim in the page 12.\n* Figure 3's top plot doesn't effectively demonstrate FunBO's advantages, as the exploration trajectories for all methods overlap significantly. A more visually distinct example would better highlight the differences in exploration strategies.\n* It would be helpful to see an ablation study on the impact of different components of FunBO (e.g., evaluation metric design, island model parameters).", "questions_for_authors": "* How robust are the discovered AFs to changes in the surrogate model (e.g., different kernel functions or non-GP models)?\n* Can you provide any interpretation why the discovered AFs that you showed in the appendix perform better than traditional AFs on the corresponding task?\n\nI think this work is super interesting and I really like the idea, I am willing to increase my score if some of my concerns/questions are properly addressed.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: This paper uses LLMs to directly propose routines that can be used as acquisition functions for Bayesian optimization. Overall, I think this is a pretty creative use of an LLM's ability to generate small amounts of path critical code, and the acquisition functions designed by FunBO seem to be competitive with or outperform both general purpose and purpose built ones in some fairly interesting empirical results. I am generally not concerned about the authors' promise to address the last few remaining concerns that weren't addressed already in the author feedback (e.g., specific acquisition functions discovered), and I believe these are relatively minor additions to the manuscript.
Decision: Accept (poster)