Scaling Multi-Task Bayesian Optimization with Large Language Models
Authors: Yimeng Zeng
Natalie Maus
Haydn Thomas Jones
Jeffrey Tao
Fangping Wan
Marcelo Der Torossian Torres
Cesar de la Fuente-Nunez
Ryan Marcus
Osbert Bastani
Jacob R. Gardner
Venue: ICLR 2026 Poster (2026)
Keywords: Bayesian optimization
large language models
protein design
meta learning
scientific discovery
Primary area: probabilistic methods (Bayesian methods, variational inference, sampling, UQ, etc.)
TL;DR: Using large language models to enhance very large scale multi-task Bayesian optimization.

Abstract: In multi-task Bayesian optimization, the goal is to leverage experience from optimizing existing tasks to improve the efficiency of optimizing new ones. While approaches using multi-task Gaussian processes or deep kernel transfer exist, the performance improvement is marginal when scaling beyond a moderate number of tasks. We introduce **BOLT**, an initialization-only transfer strategy that distills prior BO runs into an LLM which proposes candidates for new tasks, while the surrogate at test time remains single-task. The LLM is periodically fine-tuned on top solutions from completed runs, creating a closed loop where better BO outputs yield better initializations over time. This decoupled design scales to roughly 1500 tasks without the saturation observed for shared-surrogate MTBO and adds only a small, amortized overhead relative to the BO inner loops. We evaluate on two domains: database query optimization and antimicrobial peptide design. We demonstrate that LLM-generated initializations steadily improve and accelerate BO, and with sufficient fine-tuning, a few LLM samples often match or surpass full ''from-scratch'' BO with far fewer oracle calls.

Published as a conference paper at ICLR 2026
SCALING MULTI-TASK BAYESIAN WITH LARGE LANGUAGE MODELS
OPTIMIZATION
Yimeng Zeng', Natalie Maus', Haydn Thomas Jones', Jeffrey Tao', Fangping Wan". Marcelo Der Torossian Torres". Cesar de Fuente-Nunez', Ryan Mareus', Osbert Bastani', Jacob Gardner! 'Computer and Information Science, University of Pennsylvania PPerelman School of Medicine, University of Pennsylvania yimengz @seas. upenn. edu
ABSTRACT
In multi-task Bayesian optimization, the goal is to leverage experience from op- timizing existing tasks improve the efficiency of optimizing new ones. While approaches using multi-task Gaussian processes deep kernel transfer exist, the performance improvement marginal when scaling beyond moderate number of tasks. We introduce BOLT, an initialization-only transfer strategy that distills prior BO runs into an LLM which proposes candidates for new tasks, while the surrogate at test time remains single-task. The LLM is periodically fine-tuned on top solutions from completed runs, creating a closed loop where better BO out- puts yield better initializations over time. This decoupled design scales to roughly 1500 tasks without the saturation observed for shared-surrogate MTBO and adds only small, amortized overhead relative to the BO inner loops. We evaluate two domains: database query optimization and antimicrobial peptide design. We demonstrate that L.L--geeerteed initializations steadily improve and accelerate BO, and with sufficient fine-tuning, few LLM samples often match or surpass full "from scratch" BO with far fewer oracle calls.
INTRODUCTION
Multi-task optimization seeks to use related, previously observed tasks to accelerate the optimization of new ones. Multi-task optimization appears naturally in variety of domains where similar prob- lems are encountered repeatedly, such as hyperparameter optimization, material science, database query optimization, and drug design. Formally, suppose have tasks (1.2,.... 7), each associ- ated with its own objective function fx).. For each task E (1,2,.... T), we seek to find some xj such that
x arg me(()))....... xEX
We focus on the setting where, for each task, we have collected dataset D, of observations, and wish to leverage this data when optimizing unseen test tasks.
Multi-task Bayesian optimization (BO) has traditionally learned across tasks by building a shared
surrogate, typically via multi-output GPs and/or shared-weight neural feature extractors (Swersky
et al. 2013 Perrone et al. 2018 Feurer 2018 Patacchiola et al. 2020 Hakhamaneshi et al. 2022). standard approach involves placing a multi-output GP over the input-task space, decomposing the
kernel as an input kernel k(x, x') and task kernel ket, ('). Despite their effectiveness, many of these methods - with the notable exception of recent work such as Wang et al.) (2024b) - tend
to saturate in performance after tens of training tasks and do not extract additional performance improvement on new tasks when given hundreds or thousands of related tasks.
We propose Bayesian Optimization with LLM Transfer (BOLT), straightforward approach to multi- task BO that departs from the framework building related task information into the BO surrogate model. Instead, BO completes optimization for training tasks, fine-tune large language model (LLM) to, given a task description or context C' generate solutions for that optimization problem that
strong initialization for BO.

---

Published as a conference paper at ICLR 2026
This approach creates a self-reinforcing feedback loop: BO generates high-quality solutions that can leverage to fine-tune the LLM; the fine-tuned LLM, in turn, produces better initializations that improve BO performance. Over time, the LLM learns to directly generate solutions that are highly competitive, enabling top-k-samples from the LLM (requiring just a few oracle calls) to outperform full "from scratch" BO runs (requiring large number of oracle calls). This iterative improvement enables BOLT to scale and still extract value from thousands of tasks. We validate BOLT on two diverse and challenging domains where many related tasks are available.
We transfer knowledge across tasks by decoupling it from the test-time surrogate and using it only for initialization. Rather than maintain a multi-task surrogate, BOLT distills prior experience into an LLM that proposes candidate solutions from a task description C'ft], after which standard single-task BO run refines them. This removes surro--u design choices, allows usage of any BO method unchanged, and improves with scale: as more tasks are solved, initialization quality rises rather than saturates. After sufficient fine-tuning on BO-discovered solutions, the LLM becomes a strong few-shot optimizer, and running BO on top of its samples yields further gains (see $
This design contrasts with recent LLM-based multi-task BO (MTBO) systems. Optformer seeks to predict entire optimization trajectories (Chen et al. 2022), and LLAMBO uses in-context surrogates with acquisition (Liu et al., 2024). BOLT instead uses the LLM strictly for initialization within closed loop: BÔ finds high-quality solutions; we fine-tune them; the LLM returns stronger starts. Ablations show that simple alternatives (e.g.. sampling in trust regions around previous so- lutions) and untuned LLM (BOLT-0) underperform, highlighting the benefit of the closed-loop, initialization-only approach (
Contributions
1. We propose BOLT, a scalable and simple alternative to traditional multi-task BO, leveraging LLMs to generate strong initial solutions for new tasks. BOLT leverages a combination of high- quality optimized solutions produced by BO and self-auggmentation for fine-tuning.
We validate BOLT on two challenging, high-throughput domains-databan query optimization and antimicrobial peptide design and show that initialization quality improves with scale, avoid- ing the saturation of common shared-GP GP methods and outperforming recent LLM-based MTBO.
3. We show that, after sufficient fine-tuning, the LLM becomes a strong few-shot optimizer, often matching or surpassing full "from scratch" BO runs with far fewer oracle calls; running BO on top of those samples improves further.
4. We provide detailed compute analysis and ablations demonstrating that BCLT's fine- uming/sell-auggmentttonn adds as little as ~ 1% overhead relative to single-task BO runs, adding minimal computational costs for extra performance.
BACKGROUND
Bayesian optimization (BO). Bayesian Optimization (BO) (Kushner 1962
1964 Močkus 1975
Snoek et al. (012)) is an iterative approach to optimize black-box functions in a sample-effccient manner. On each step of the optimization, a supervised probabilistic surrogate model (usually a Gaussian Process (GP) (Rasmussen) (003)) is conditioned on all data collected so far. Then, the surrogate model's predictive posterior distribution p(y x, D) is used to decide what data point(s) should be evaluated next, typically by maximizing some acquisition function, defined with respect trade off. Finally, selected points are evaluated on the black-box function and added to the dataset. This iterative process continues until the evaluation budget is reached.
to p(y | x, D). which guides the opploration-cxplottaaee
Structured optimization via latent space BO. BO has recently been applied to optimizing struc- tured search spaces, such as molecular and amino acid sequences, by leveraging latent space Bayesian optimization. This approach incorporates a variational autoencoder (VAE) to map struc- tured inputs into a continuous latent space, where BO is performed (Kingma and Welling 2014 Eissman et al. 2018 Tripp et al. 2020 Grosnit et al. 2021 Siivola et 2021 | Stanton et al, 2022
Maus et al. 2022). Structured inputs (e.g., amino acid sequences) are mapped to continuous la- tent representations z by the VAE encoder & o(x). This creates a transformed continuous (latent) representation of the structured search space where BO can be directly applied Hianez-Bombarelli jet al. 2018 Griffiths and Hernández- candidate points then decoded by the VAE decoder, T(z), to reconstruct structured outputs for
2020 Kusner et al. 2017). The corresponding latent
2

---

Published as a conference paper at ICLR 2026
evaluation. For large combinatorial structured search spaces, such as the space of organic molecules or the space of all peptide amino acid sequences, the latent space of the VAE is typically high- dimensional (on the order of several hundred dimensions) in order to represent the large structured space effectively (Chu et al. 2024 Lee et al. 2025).
Optimizing antimicrobial peptides. In antimicrobial peptide design, we seek peptides (se- quences of amino acids) that minimize the MIC (minimum inhibitory concentration, measured in 1 mol ! for some target bacterial pathogen. MIC is measure of the concentration of the peptide required to inhibit growth of the target bacterial pathogen (Kowalska- Kroochaa and Dudek- Wicher, 2021]. A key challenge in antimicrobial peptide design is that many modern bacterial pathogens have developed resistance to modern antibiotics. To solve this challenge, Wan et al. (2024) propose designing new peptides with high sequence similarity to template peptides mined from extinct or- ganisms. The template peptides themselves do not typically achieve sufficiently low MIC for target bacterial pathogens. However, since these template peptides have not been encountered in nature for thousands of years, modern antimicrobial resistant bacteria have not evolved resistance to them. follows that new peptides are more likely to evade antibiotic resistance if they are designed to be similar to the extinct template sequences. We employ this strategy, optimizing antimicrobial peptides with minimum threshold sequence similarity to the extinct template peptides from Wan ett al. (2024). We also employ latent space BO to optimize over the structured space of amino acid sequences.
Optimizing database query plans. Query optimization in data management systems involves translating a declarative SQL query into an execution plan that efficiently retrieves the correct re- sults (Gracfe and McKennal 1993). This problem has been extensively investigated in the field of data management (Leis et al. 2017). as the difference in execution time between an optimal and a poorly chosen query plan can be several orders of magnitude (Leis et al. 2015). Since individual query plans are composed of discrete characteristics (e.g. join order trees), the search space of pos- sible query plans is structured and combinatorial. We therefore employ latent space BO. We use the string representation for query plans proposed by Tao et al.] (2025) to pre-train VAE model that maps the structured space of query plans to a continuous latent space where BO can be applied.
Database query plan optimization with right-censored observations. In database query opti- mization, our black-box objective function measures the execution latency of the query plan. "Good" and "bad" query plans can have latencies differing by multiple orders of magnitude (Leis et al., 2015). This can lead to the majority of optimization runtime being taken up by evaluating a small number of poorly performing plans. A natural solution to this problem is to time out objective function evaluations after they have reached some threshold latency т, resulting in right-censored observations. A right-censored observation is an observation at data point x where we observe only that y 2 r for some chosen timeout threshold т, rather than observing the typical noisy ob- jective value y. Prior work has been done to extend Bayesian optimization methods to the setting of right-censored observations. Hutter et al. (2013); Eggensperger et al. (2020) extended Bayesian optimization methods to the setting of right-censored observations by introducing an EM-like algo- rithm to impute the values of censored observations. Eggensperger et al. (2020) expanded on this, defining single surrogate model capable of being conditioned on the combination of censored and uncensored data gathered. (Tao et al.) (2025) extend this to the setting of approximate GP surrogate models. Since we focus on tasks that involve large function evaluation budgets, we employ Tao al. 2025)ss 's proposed method of modeling censored data with approximate GPs.
3 BAYESIAN OPTIMIZATION WITH LLM TRANSFER (BOLT)
We propose Bayesian Optimization with LLM Transfer (BOLT), an iterative framework for using LLMSs to improve Bayesian optimization (BO) performance across a family of related tasks. We are given a set of T training tasks defined by objective functions fi (x). assume that, for each objective function have context or task description C' that can be a natural language or other input description that differentiates ft from any other task in the ap- plication domain. For example, this might be the text of a SQL query we are trying to optimize. Consequently, BOLT is specifically designed for domains with a shared input space and informative textual descriptions.
We additionally
For each training task, we assume we have optimized the objective with some BO procedure, result- ing in the optimization trajectories (D; ) with each Dj containing the top-K observations from
3

---

Published as a conference paper at ICLR 2026
the trajectory for the tth task. Our goal is leverage this training data to learn an LLM-based "ini- tialization policy" π that, when presented with new related tasks high- set of candidate solutions for BO to further refine.
([fr+ proposes
These two procedures - (1) using BO to collect high-quality data for training tasks, and (2) using the LLM to initialize BO for new tasks can be used as "outer-loop" "omerrloopp"innne-looooo"" approach to solving large number of related tasks sequentially, where the LLM is periodically updated as more optimization runs complete.
Because the LLM and BO only interact through generating initialization and generating fine-tuning data respectively, our approach here is relatively agnostic to the specific underlying implementation of BO used to optimize each task. This enables the straightforward use of the full range of recent BO advances on high-dimensionss constrained, and other optimization settings.
Inner Loop: LLM-Initialized Bayesian Optimization
Algorithm 1:
Algorithm 2: Outer Loop: LLM Fine-
Tuning via BO Trajectories
Require Task L. context C' LLM m...
Require Dataset Do = {(C),f)), x;,
budget B, batch
LLM To. iterations T
Ensure : +Optimized solutions X²
Ensure #Fine-tuned L.M Initialize D < Do. π <- To for iteration k: to ' do foreach task 1 in batch do
Xini +- (C(()
// LLM proposes
candidates
Evaluate Minit 4- [Xiii
D <- (Xinis Mini))
X7 <- INNERLOOP(L Ta, B. b) // Run BO D + DU ((C[))
Initialize (P(Xini Mimit)
for step i 1 to [B/b] do
X;)
X arg max, a<<;;; GP)
// Augment with top
// Acquisition
solutions
Unest f- nena))
++ (X U Xsen, U yrea)
Fine-tune TTs on augmented datasct D Update model paramoters via instruction prompting
Update GP with new observations
Retum X; f top--((X)
// Beat
solutions
Return Kт // Final fine-tuned LLM
Initializing BOLT. At initialization for workload of tasks, we have only an un-tuned L.M ROLT- that is generally useless for the task setting because it is unaware of even the specific format for candidate suggestions. For the first iteration, solve T optimization tasks with single-task BO routine where we initialize BO using some standard initialization procedure. We run optimization on each of the T initial tasks, and extract the optimization trajectories {D;) from each
LLM fine-tuning. The LLM fine-tuning process employs supervised learning using OpenAl's GPT-4O-MINT MINI-0718 model through their API. From the optimization trajectories {D; H- tract the top-K observations from each of the T runs completed so far. We use these observations along with the task contexts {C(fi) to construct fine-tuning dataset Da. Each training instance contains:
1. A system prompt shared across all tasks in the workload/problem domain, which specifies the objective (e.g., generating efficient join orderings).
A user prompt with the task-specific context Cifi] (e.g., the SQL query requiring optimization). response prompt containing the high-pe geghhperffrming solution x discovered through BO.
We fine-tune using OpenAI's standard fine-tuning API (OpenAletal, 2024 our data into the required JSONL format (i.e., prompt-solution pairs) and then upload it via the fine- tuning API to initiate training. The model is trained to minimize the negative log-likelihood of the solution tokens given the task context C::
Specifically, we format
Σ (r:
We note that our approach leverages full model fine-tuning rather than extensive and/or manual prompt engineering (Lester al. 2021 Li and Liang 2021). This allows the model learn the task
4

---

Published as a conference paper at ICLR 2026
requirements through the context-solution pairs in Dn, rather than explicit instructions. However, for scenarios requiring few-shot learning on untrained models, more careful prompt engineering may be beneficial. This fine-tuning process produces an updated model that encodes the knowledge from Dn. In our experiments, we will refer to an LLM trained on 7 tasks in this way as BOLT-T.
LLM fine-tuning frequency. The number of tasks T that we collect at initialization time and during each round of the BOLT "outer-loop" represents non-trivial trade-off due to the compu- tational cost of both running BO and the cost of fine-tuning the LLM. Fine-tuning the LLM more frequently results in both additional computational and monetary costs, but allows subsequent BO runs to complete more efficiently (with fewer black-box function evaluations). this paper, we erred on the side of lower monetary cost in exchange for additional cost in black-box function eval- uations. Specifically, we fine-tuned an LLM 4 times for the query plan optimization task and 7 times for the antimicrobial peptide design task shown in Figure 2
Using the LLM for multi-task BO. Once we have a fine-tuned model, BOLT-T, we can leverage the fine-tuned LLM's capabilities generate higher-quality initialization points for subsequent op- timization tasks. For set of new tasks {4)] we sample from BOLT-7 generate the same number of initialization points used by the baseline "from scratch" approach. The sampling prompt maintains the same structure the training prompt without the assistant response. The BOLT-T generated solutions are refined with a standard BO routine, and the lop-K performing solutions for each task t-along with their contexts Cifi]aae incorporated into the training set for the next round of fine-tuning.
Self-Augmentation. As the fine-tuned LLM en- hances few-shot generation with more optimiza- tion data, is worth exploring whether the costly sequential BO processes can be minimized. Thus, we explore "self-improvemmet" methods to re- fine the LLM policy without the expense of ad- ditional optimization runs (Algorithm 3). Specif- ically, once an LLM has been fine-tuned using some of the tasks we set aside for training, we prompt it to generate additional solutions for all available tasks in that problem setting. We then score these solutions using the problem's oracle and fine-tune the LLM again directly with this la- beled self-generated data, a manner similar to self-instruction in LLM training (Shypula et al.) 2024]. By filtering and fine-tuning on its own best outputs, the LLM can iteratively teach itself how to propose better solutions.
Algorithm 3: Self-augmentation for LLM
Finetuming
Require
Tasks T, L.LM no, iterations
criteria C
Ensure : FFine-tuned LLM T#aT
Sample D> +- Do. <- no
for iteration & = 1 to do
do foreach task
Xinit ~ +o+x(t) // Generate
samples
Xinit <- Select C) // Select best samples
D - DU x, Xtma)
// Augment dataset
Fine-tune To0+ on D
Return रr
// Final fine-tuned LLM
4 EXPERIMENTS
We evaluate BOLT on two distinct problem domains, each with a large number of related tasks. For both domains, problem definitions and solutions be represented as strings. This allows BOLT to operate both in sequence space, where the LLM learns from optimization trajectories, and in latent space, where BO makes additional progress using LLM-sampled initializations.
Implementation details. For the inner optimization loop, we implement a constrained version of the LOL-BO algorithm (Maus et al., 2022) using BoTorch and GPyTorch (Balandat et al., 2020) Gardner et al. 2018). For query optimization, we use an acquisition batch size of I with a budget of 4,000 oracle calls, while for peptide design, we employ a larger acquisition batch size of 50 with a budget of 20,000 oracle calls.
The outer loop BOLT-T models use instruction prompting (Mishra et al. 2021 Longpre et al. 2023) to guide the LLM in producing optimized sequences. Figure 4 shows the template used to prompt GPT-40-MING for efficient query plans. After each optimization iteration, we augment the training set with the highest-scoring sequences from the optimization trajectory and fine-tune GPT-4O-MIN MINI on this expanded dataset. When fine-tuning the LLM for the query plan optimization task, we use OpenAl's automatic batch size selection option. For the peptide design task, found that using the
5

---

Published as a conference paper at ICLR 2026
automatic batch size option did not provide similar boost in performance, and we a constant batch size of 10. For both tasks, we fine-tune tine-tuned the LLM for 2 epochs and used the default OpenAI LR multiplier hyperparameter of 1.8. To ensure the solutions always have the correct syntax, out characters that do not correspond to strings of integers or valid amino acids for the respective tasks. Figure (5 provides additional details on the fine-tuning process and prompts used for the peptide task.
filter
Database query plan optimization. Database query plan optimization focuses on finding query plans (including join orderings and their operators) with low execution time for a given query. We take a subset of 2933 queries from the Cardinality Estimation Benchmark introduced by Negi ict al. (202I), keeping 99 queries for validation. Following Tao et al.] (2025), we perform BO over query plans by encoding join orders and operators as integer lists, which are then mapped to 64- dimensional continuous latent space using the pre-trained query plan VAE from Tao et al. (2025). randomly generated 1, 169, 890 query plans based on the database schema, sep-
For pretraining, arated into 80/10/10 splits. For the initial "from scratch" runs with no LLM, we initialize with the set of 50 query plans used by BAO (Marcus et al. ) an MI-powered query optimizer we use as a baseline, that produces reasonable but non-optimal plans. Subsequent runs use 50 M-saplleedd query plans per query as initialization points. All points are sampled using a temperature parameter of 0.7 unless otherwise specified. The "task description context" C' used to fine-tune the LLM for this task is the full SQL query string; Appendix ( shows that initialization performance is robust across nearby temperatures and that invalid generations remain rare in this regime.
are given library of 1000 Antimicrobial peptide design. For the peptide design application, extinct, weakly antimicrobial seed peptides S = {s1, S... A task in this setting is to take a particular seed peptide s; and make modifications to it to minimize the minimum inhibitory con- centration (MIC) against A. Baumannii ATCC 19606, measured in /moo// L. We created a library of L 1000 extinct peptides and held out the last 100 as validation. We ensure edited peptides maintain a minimum 75% similarity to the seed peptide, defined by d(s.s")
len(S) where d is the Lev- enshtein distance between them. All of the validation peptides are at least 25% different from any other peptide in the library. Although the seed peptides don't achieve low MICs, the hope is that bacteria are less likely to have developed resistance to their variations they come from extinct species (Wan et al. 2024). We assess MICs with the APEX model and utilize a VAE trained on 4.5 million amino acid sequences map peptides into 256-dimensional latent space (Wan al. [0244] Torres et al. 2024). Initial optimization uses 1000 randomly mutated sequences with a similarity constraint of 75% the seed. Subsequent runs utilize 1000 LLM sampled peptides. All points are sampled using a temperature parameter of 1.0 unless otherwise specified. We use the seed amino acid sequence as the "task description context" C for LLM fine-tuning.
Baselines. We compare BOLT against a range of baseline approaches. First, we compare to "from scratch," single task LO which we will refer to as STBO, which operates without prior task knowledge, Second, we compare to a common strategy for multi-task BO, e.g.. Patacchiola et al.) (2020); Hakhamaneshi et al. (2022); Perrone et al. 2018), where shared GP is trained all tasks through a neural network feature extractor using the optimization trajectories from training tasks. This shared GP is then checkpointed and used on test tasks. Several papers have found success with variations of this approach. ABLR (Perrone et al. 2018) uses independent Bayesian linear regression heads per task on the shared feature extractor, while FSBO (Wistuba and Grabocka 2021) uses an adaptation of DKT (Patacchiola et al. 2020). In this paper, for the final supervised model, we use the same PPGPR model (Jankowiak et al. (020) as the BO inner-loop in our method as we require scalability but find this results in better performance than Bayesian linear regression or random Fourier feature models (Rahimi and Recht 2007).
We also compare against methods that utilize an ensemble of Gaussian process experts, POGPE and SGPE (Schilling et al. 2016). POGPE utilizes an ensemble approach with one expert from each previous task, while SGPE extends this by adding an additional expert trained on data for the current task only with higher weighting. We evaluate several configurations of these methods (with 5, 10, and 20 experts) to identify optimal performance. Additionally, we evaluate two transformer-based methods: Optformer (Chen et al. 2022) and LLAMBO (Liu et al. 2024). Optformer employs a fine- tuned transformer model (in implementation, GPT-4O-MINI) for hyperparameter optimization, while LLAMBO uses out-of-the-box LLMs (also GPT-4O-MIXG in our case, compared to GPT-3.5 in the original paper) for both surrogate modeling and acquisition function optimization. Due to

---

Published as a conference paper at ICLR 2026
Optimitotion Start
Optinizaiios Start
Summod query runtime (ncemaizze)
pretlcsod (oormal
303-0
3000
1000
Figure I: Bayesian optimization performance on (Left) query plan optimization and (Right) antimi- crobial peptide design. The z-axis represents the number of oracle calls per held-out test task. In both settings, BOLT outperforms or matches baselines with just initialization data before optimiza- tion begins. ROLT-7 indicates that the LLM was fine-tuned the top solutions from T training tasks. Markers are plotted every 50 oracle calls, and lines represent the sum across test tasks, with shaded bands indicating +1 standard error. Fine-tuning rounds for the LLM are not depicted on the rraxiis the LLM is updated only between batches of training tasks and remains fixed during eval- uation on held-out tasks. At test time, each run starts from identically sized initialization sets (DB: 50 plans; peptides: 1,000 sequences), with BOLT replacing the baseline initializations with samples from the fine-tuned model. Consequently, the advantage observed the "optimization start" reflects improved initial candidates, after which a standard STBO loop proceeds unchanged.
context length limitations with these transformer-basee methods, we maintain a sliding window of the last 100 oracle calls for both approaches. For LLAMBO, we impose a budget limit of 10 million input tokens per experiment to manage computational costs. Additional details on all baselines can be found in Section C
4.1 OPTIMIZATION RESULTS
In Figure I we demonstrate that initializing BO with BOLT significantly improves optimization efficiency across both domains. On the query optimization task (left), while DKT/FSBO makes improvements over STBO, the gains appear to plateau after only 20 tasks. In contrast, BOLT suc- cessfully scales to over 1400 tasks and converges to higher quality solutions faster. On the peptide design task (right), BOLT shows similarly strong performance, while DKT/FSBO struggles to take advantage of the data collected for separate templates. Notably, BOLTTgenerated initializations al- ready outperform the respective baselines the optimization start (i.e., before the first BO step on each test task), and this gap increases over the evaluation budget. We emphasize that the a-axis in Figure T records oracle calls made on the test task only; the LLM is not updated during these evaluations. All LLM fine-tuning occurs offline on previously solved training tasks, and in our runs, a small number times per domain (four rounds for DB; seven rounds for peptides).
The GP expert methods show mixed results. For the database task, POGPE-5 and SGPE-5 demon- strate better performance than their variations with more experts, while for the peptide task, POGPE- 10 and SGPE-10 yield the best results compared 5/20 experts. Consistent with findings from Schilling et al. 20166). POGPE generally outperforms SGPE across both domains. However, both ensemble approaches are consistently outperformed by BOLT, and even fall behind STBO and DK- TYFSBO in several cases. We did not run POGPE or SGPE with a larger numbers of experts both methods scale poorly, requiring updating of a number of GPs proportional to the number of tasks.
The transformer-basee methods demonstrate notable limitations in Figures I and 2 Optformer achieves performance slightly worse than STBO on the database task while showing significantly poorer results on the peptide task. LLAMBO performs substantially worse across both domains, showing minimal progress during optimization. Due its computational dequris-requiring L.M inference for both surrogate modeling and acquisition-LLL.. mization steps within our token budget constraints.
completed fewer than 100 opti-
We find that on both tasks once BOLT reaches sufficient scale, it begins to few-shot generate initialization data for BO that significantly better performing than the final results found by all baseline methods, including STBO, DKT/FSBO, the GP expert methods, and waasformer-basee approaches.
7

---

Published as a conference paper at ICLR 2026
Summed query runtime
25
PodgeSgi BUILis
BOLTA00
Average prodieted MIC
20.0
BOLT:10
BOLT
BOLT-400
BOLT-30
BOLT-500
20
15.0
BOLT
12.5
10.0
Number of few shot samples
Figure 2: Evaluating BOLT in the few-shot setting and comparing to full optimization runs in both problem settings (Left:: query plan optimization; Right: peptide design). In each plot, we show objective values accumulated across all validation tasks for various methods. Scatter points illustrate the few-shot performance of BOLT using different number of tasks, and relevant domain baselines (e.g.. PostgreSQL, BAO for query optimization). Horizontal dashed lines indicate the performance of various full BO runs and other optimizers, shown for comparison. These results demonstrate that BOLT's few-shot performance is often comparable to or surpasses that of full BO runs.
and few-shot optimizer. In Figure 2 BOLT demonstrates strong few-shot general- ization capabilities, even achieving single-shot performance competitive with traditional approaches. In query optimization, all BOLT variants outperform the top BAO solution within 5 samples. No- lably, BOLT-1138 and BOLT-1426 surpass PostgreSQL in a single sample, indicating their potential for rapid deployment low latency scenarios. The performance of BOLT consistently improves with more iterations across both tasks, except at 50 samples on the query optimization task, where BOLT- 138 slightly outperforms BOLT-1 1426. This may due to variances L.M sample genera- tion or training. Overall, results confirm BOLT's robustness when scaling to thousands of tasks. We further compare our few-shot performance (Figure 2 before x-axis break) to full BO runs (Figure P after x-axis break). In both tasks, BOLT achieves few-shot results comparable to the full BO runs.
BOLT
Compute overhead and wall-clock. Across both domains, the dominant cost is the inner-loop BO, not the LLM components. On the database tasks, an STBO run requires roughly 15-20 GPU- hours per task (avg. 18)8), amounting to ~ 25k GPU-hours over the full workload. By comparison, BOLT's outer-loop fine-tuning and sampling add only small, amortized overhead. At the 50-task scale, the total compute with BOLT ~7% above STBO, and by the full workload 1,426 tasks, is ~1% above STBO (Table 3 Fine-tuning consumed ~-0M tokens in total (OpenAT API; $1800), which we also report as a conservative local-equivalent of 4000 GPU-hours; generating 50 BOLT initializations per task took 1 GPU-minute per task (~244 GPU-hours across all DB tasks). Detailed per-method runtimes and token/cost accounting are summarized in Section
For completeness, we also quantify the cost of the self-augmentation step used in the outer loop: end-to-end, this adds on the order of tens of GPU-hours for the full workload (negligible relative to the BO inner loops); see Appendix for the precise accounting and setup.
4.2 ABLATION STUDIES
LLM self-augmentation. We investigate whether self-augmentation as outlined in Algorithm can improve LLM performance while avoiding the computational expense of the inner-loop BO on the query plan optimization task. We apply the celf-ntgmenttooon process to the four fine-tuned LLMs in Table [ generating 10 samples from each across all 2, 933 training tasks, keeping only queries that outperform the best query plan from BAO's solutions. We then use these datasets as additional fine-tuning to create self-augmented versions of the L.LMs.
Table I shows that this self- augmentation yields substantial improvements even without additional tasks optimized by BO. Both self-augmented models converge to a similar performance level, achieving a summed runtime of about 62 seconds across the 99 validation queries. This conver- gence suggests a natural performance plateau after training on either 1,500 tasks (BOLT-1426) or 1,100 tasks plus self-gen -eferaeed samples (BOLT-11 111-+1111))) The consistency of this plateau across different training approaches further demonstrates BOLT's robustness when scaling to large task sets. This self-augmentation experiment indicates that once the LLM has been fine-tuned to suf-
8

---

Published as a conference paper at ICLR 2026
12
Method
@est
LLM (BOLT-50) no SA
87.84
LLM (BOLT-893) no SA
82.31
LLM (BOLT-1138) no SA
78.16
LLM (BOLT-1426) no SA
63.68
LLM (BOLT-50)
82.25
LLM (BOLT-893)
63.05
LLM (BCL.-1113)
61.46
LLM (BOLT-1426)
61.54
Figure 3: One-shot comparison of BOLT- 1426 BAO (best-of-50). For each query, we sample one plan from the fine-tune fin-ttned LLM and compare with BAO's best-of-50 initialization (blue BOLT better; orange = BAO better). Top: sam- pling with T 0.7. Bottom: greedy with T = 0.0. In the one-shot setting, purely greedy sampling is better.
Table Ablation study for self seff-augmentation (SA) conducted on the query optimization task. For each the LLMs with different training task sizes, we perform SA and generate 50 query plans from the LLM. We measure the best summed query execution time across the valida- tion tasks from among these 50 samples.
ficient performance, can generate additional fine-tuning data, reducing the number of runs required. Additionally, our framework scales to more training tasks without performance loss.
One-shot capability. Figure 3 isolates the one-shot behavior of the final, fine-tuned model: for each held-out SQL query, we draw single plan from BOLT-1426 and compare to BAO's best-of-50 initial plans. No BO steps are run in this analysis. particular, improvements to BO appear already with tens of training tasks (Figure I while the strong one-shot behavior in Figure reflects the capability that emerges after scaling to many tasks.
Impact of data quality on training. We perform an ablation to assess the importance of using "better" versus "more" training data for fine-tuning LLMs through iterations of BOL T. Starting with the BOLT-1138 model, we collect top solutions from a new BO round and train two variants: 1) BOLT-1426, which adds all new solutions to the original BOLT-1133 1-188-138 set. 2) BOLT-1138" which instead replaces an equal number of old solutions to maintain the same training set size. As shown in Table both benefit from higher-quality data, suggesting "better" data boosts performance. How- ever, ROLT-1 138* underperforms BOLT-1422 which incorporates more and better data, confirming that both factors enhance model performance.
Best@
BOLT-1138 no SA
BOLT-1138 no SA
BOLT-1426 no SA
@est@
78.16
64.03
63.68
Best@21 20
82.59
70.52
66.23
@est@ 10
90.19
74.40
70.21
@
102.99
85.21
76.28
@e
127.97
129.26
102.29
@
202.04
193.64
160.22
Table 2:
Comparing LLMs fine-tuned with (Left) data from 1 1138 tasks, (Right) data from 1426
tasks, and (Middle) data from 1138 tasks, but including the extra data from BOLT-1426. and remov- ing data from older tasks. This is done the DB task.
RELATED WORK
Language models as optimizers. Large language models (LLMs) have recently gained attention as sequence optimizers capable of tackling diverse black-bor box tasks where direct gradient information unavailable difficult to compute. LLM- optimizers leverage the flexibility of natural language prompts to encode candidate solutions, constraints, and relevant task information. Methods like OPRO illustrate how iterative prompting can refine solutions (Yang et al. 2024 Zelikman al.

---

Published as a conference paper at ICLR 2026
2024). while other approaches integrate self self-improving strategies that reuse tigh-performing LLM outputs for further fine-tuning (Shypula et al., 2024). This set of techniques has been applied to biophysical domains such molecular design and protein engineering, where the LLM proposes mutations to enhance certain properties, as well as to program optimization tasks where the LLM
speeds up code execution time (Shypula et al. 2024 Wang al. 2024a Madani al. 2023).
Database optimization. Recent work has applied Bayesian optimization (BO) to improve overall
database performance (Zhang et al. 2022; Nardi et al. 2019 Cereda al. 2021 by tuning the parameters of the database configuration. As far as we are aware, Tao et al. (2025 were the first to
apply BO to the specific setting of database query plan optimization considered in this paper. Other
work has applied reinforcement learning (RL) to query plan optimization (Marcus et al. [201 Yang
jet al., 2022 Zhu et al.) 2023). RI query optimizers learn from mistakes and improve performance over time. Unlike BO, however, RL requires large supervised datasets for pre-training and typically aims to minimize cumulative query latency rather than achieving the lowest possible latency.
DISCUSSION AND LIMITATIONS
We first highlight a few limitations. First, our approach requires that all tasks problem setting have the same input domain (a problem that has been explored c.g. by Fan et al. (2022)). We further require the existence of a task description context C[ that can be used an LLM prompt to define the task. This excludes common MTBO settings where tasks are primarily distinguished by data (e.g.. hyperparameter optimization across datasets) rather than by concise textual descriptions: for such settings, approaches such as Wang et al. (2024b) are likely more appropriate. Finally, we note that the cost of LLM fine-tuning is significantly higher than simple gradient updates of shared feature extractor, even though our experiments indicate that this overhead is small when amortized over thousands of tasks ($4) Appendix ()).
Despite these limitations, in two real-world applications where BOLT was applicable it yielded strong results. Few-shot generation matched "from scratch" BO runs, and initializing BO from the LLM samples often improved performance further. Moreover, the interplay between the LLM and Bayesian optimization is noteworthy. Despite interest in using LLMs for optimization ( (Yang et al.)
2024 Zelikman et al., 2024 (Shypula al. 2024 Wang et al., 2024a] Madani al., 2023). finding initial strong solutions to fine-tune them is challenging in some domains. Bayesian optimization, by offering in-depth search, is an excellent candidate for this.
REPRODUCIBILITY STATEMENT
We have relcased code at (attps://gil hub. www..////mmmnge (innYiennng-eeg//0111) We document task definitions, budgets (DB 4k calls; peptides batched at 50), and compute usage and hardware details. We report LLM fine-tuning hyperparameters and provide an open-source LLM recipe mirroring GPT-40-MIN 40MMIIIIII runs (see ş Appendix D.
ETHICS STATEMENT
This research introduces BOLT, method leveraging large language models to enhance multi-task Bayesian optimization, with demonstrated applications in antimicrobial peptide design and database query optimization. The potential to accelerate the discovery of novel peptides could significantly benefit public health, particularly in combating antimicrobial resistance. Similarly, improving database query efficiency can lead substantial computational and energy savings across many industries.
However, acknowledge the potential for AI misuse in biological design. In applying these power- ful methods, expert oversight, rigorous validation, and adherence to established safety and regulatory frameworks must be highlighted. Additionally, the use of large-scale LLMs raises considerations garding computational accessibility and responsible AI development.
10

---

Published as a conference paper at ICLR 2026
ACKNOWLEDGEMENTS
N. Maus was supported by the National Science Foundation Graduate Research Fellowship; J. R. Gardner was supported by NSF awards (IS-2145644 and DBI-2400135: C. de Fuente-Nunez was supported by NIH grant R35GM138201, and by Defense Threat Reduction Agency grants HD- TRA11810041, HDTRAA-21-1--0 and HDTRAI-23-1-0001.
REFERENCES
Maximilian Balandat, Brian Karrer, Daniel R. Jiang, Samuel Daulton, Benjamin Letham, An- drew Gordon Wilson, and Eytan Bakshy. BoTorrch A framework for efficient Monte-Carlo Bayesian optimization. Advances in Neural Information Processing Systems, volume 33, pages Curran Associates, Inc., 2020.
21524-21538.
Stefano Cereda, Stefano Valladares, Paolo Cremonesi, and Stefano Doni. CGPTuner: a contex- tual gaussian process bandit approach for the automatic tuning of IT configurations under vary- ing workload conditions. Proceedings of the VLDB Endowment, 14(8): 1401-1440111113 April 2021. ISSN 2150 doi: 10. 00.14778/3453390.44544444 14778/345788 3457404
Utthttss:://// acm. /oo//do//C.
Yutian Chen, Xingyou Song, Chansoo Lee, Zi Wang, Qiuyi Zhang, David Dohan, Kazuya Kawakami, Greg Kochanski, Arnaud Doucet, aurclaurelio Ranzato, Sagi Perel, and Nando de Freitas. Towards learning universal hyperparameter optimizers with transformers. In Proceed- ings of the 36th International Conference on Neural Information Processing Systems, NeurlPS "22, Red Hook, NY, USA, 2022. Curran Associates Inc. ISBN 9781713871088.
Jaewon Chu, Jinyoung Park, Seunghun Lee, and Hyunwoo J. Kim. Inversion-based la- tent bayesian optimization. In A. Globerson, L. Mackey, D. Belgrave. A. Fan, U. Pa- quet, J. Tomezak, and C. Zhang, editors, Advances in Neural Information Process- ing Systems, volume 37, pages 68258-68286 Curran Associates, Inc., 2024. URL ottps:://proceeeingg. www.:eiinnnnnnnuiueeeeeeee neurips wttppppppppppppppcttpptpppteep.t (e..91a922221111333eeeee10001100
påf
Katharina Eggensperger, Kai Haase, Philipp Müller, Marius Lindauer, and Frank Hutter. Neural model-based optimization with right-censored observations. arXiv preprint arXiv:2009.13822 2020.
Stephan Eissman, Daniel Levy, Rui Shu, Stefan Bartzsch, and Stefano Ermon. Bayesian optimiza- tion and attribute adjustment. In Proc. 34th Conference on Uncertainty in Artificial Intelligence, 2018.
Zhou Fan, Xinran Han, and Zi Wang. Hyperbo+: Pre-training a universal prior for bayesian opti- mization with hierarchical gaussian processes. arXiv preprint arXiv:2212.105.0 2022.
Matthias Feurer. Scalable meta-learning for bayesian optimization using ranking- gaus- sian process ensembles. In ICML 2018 AutoML Workshop, 2018. URL hhtpps:/appi.
Jacob Gardner, Geoff Pleiss, Kilian Q Weinberger, David Bindel, and Andrew G. Wilson. GPy- Torch: Blackbox matrix-matte Gaussian process inference with GPU acceleration. In Advances in Neural Information Processing Systems, volume 31, pages 7576-7586. Curran Associates, Inc., 2018.
Rafael Gömez-Bombarelli, Jennifer N. Wei, David Duvenaud, José Miguel Hernández-Lobato, Benjamín Sänchez-Lengeling Dennis Sheberla, Jorge Aguilera-Iparraguirre Timothy D. Hirzel, Ryan P. Adams, and Alán Aspuru-Guus -uzikk Automatic chemical design using a data-driven continu- representation of molecules. ACS Central Science, 4(2):268 276, Feb 2018. ISSN 2374-7943. doi: 10, ha..0e///ccccntsci....... 7b00572. 7.00007
URL / 1021:/1....11111110 /t..///oooo........eeee.c.......
7b00572

---

Published as a conference paper at ICLR 2026
Goetz Graefe and William J. McKenna. The volcano optimizer generator: Extensibility and effi- cient search. In Proceedings of the Ninth Interaational Conference on Data Engineering, page 209-218, USA, 1993. IEEE Computer Society. ISBN 0818635703.
Ryan-Rhys Griffiths and José Miguel Hernández-Lobato. Constrained bayesian optimization for automatic chemical design using variational autoencoders. Chem. Sci., 11(2):777-886, January 2020.
Antoine Grosnit, Rasul Tutunov, Alexandre Max Maraval, Ryan-Rhys Griffiths, Alexander Imani Cowen-Rivers, Lin Yang, Lin Zhu, Wenlong Lyu, Zhitang Chen, Jun Wang, Jan Peters, and Haitham Bou-Ammar. High-dimensional Bayesian optimisation with variational autoencoders and deep metric learning. CoRR, abs/2106. (000600369,... 2021.
Kourosh Hakhamaneshi, Pieter Abbeel, Vladimir Stojanovic, and Aditya Grover. JUMBO: Scalable Multi-task Bayesian Optimization using Offline Data, March 2022. URL http://arxiv.
org/abs/ /aabs//16.. 00942 MrXiv:2106.0094 [cs].
Frank Hutter, Holger Hoos, and Kevin Leyton-Brown Bayesian optimization with censored re- sponse data. CoRR, abs/1310.1947, 2013.
Carl Hvarfner, Erik Orm Hellsten, and Luigi Nardi. Vanilla Bayesian optimization performs great in high dimensions. In Proceedings of the 4/st International Conference on Machine Learning, volume 235 of PMLR, pages
JMLR, 2024. URL htttps://proceedingg
mlr. prwss/ v235 wtt.///aaaeerrrr// html
Martin Jankowiak, Geoff Pleiss, and Jacob Gardner. Parametric Gaussian process regressors. In Hal Daumé III and Aarti Singh, editors. Proceedings of the 37th International Confer- ence on Machine Learning, volume 119 of Proceedings of Machine Learning Research, pages 4702 4712. PMLR, 13-18 Jul 2020. URL httpp::/pprcceed htt.gg/.........//11/119 jankow wwwpiaak2aa..mm ntm.
mlr
Diederik P. Kingma and Max Welling, Auto-Encoding Variational Bayes. In 2nd International Conference Learning Representations, ICLR 2014, Bansf, AB, Canada, April 14-16, 2014, Conference Track Proceedings, 2014.
Beata Dowalska-Krochmen and Ruth Duchk-Wicher. The minimum inhibitory concentration of an- tibiotics: Methods, interpretation, clinical relevance. Pathogens, 10(2):165, 2021.
Harold J Kushner. A versatile stochastic model of function of unknown and time varying form.
Journal of Mathematical Analysis and Applications, 5(1):150-167, 1962. ISSN 0022-247X. doi: wttp:///w.....000000000000001000
URL https:://///. /tttps://www... sci.sseeiiiecdrrett
com/ http://ww..oooeeeee..eee//////22
Harold J Kushner. A new method of locating the maximum point of an arbitrary multi- peak curve in the presence of noise. Journal of Basic Engineering, 86(1):97-106. 1964.
doi: https:/do..oog//0.11111113531113
URL http:psss//smmmdlggiiaalloos
asme hrtp:://eecuueeeeeeee.eeeeaaaaaa
Matt J. Kusner, Brooks Paige, and José Miguel Hermández-LLbbroo. Grammar variational autoen- coder. In Doina Precup and Yee Whye Tch, editors. Proceedings of the 34th International Con- ference on Machine Learning, volume 70 of Proceedings of Machine Learning Research, pages 1945-1954. PMLR, 06-11 Aug 2017. URL https: pettps://proeeddingg mwt.press.vTo.
Kusneri7a.Hie
Seunghun Lee, Jinyoung Park, Jaewon Chu, Minseo Yoon, and Hyunwoo J. Kim. Latent bayesian optimization via autoregressive normalizing flows. In The Thirteenth International Confer-
on Learning Representations, 2025. URL https: http::::///peeevviiiee www.nee///ee fwr.mmirrmmm h
ECOOwRRaF1
Viktor Leis, Andrey Gubichev, Atanas Mirchev, Peter Boncz, Alfons Kemper, and Thomas Neu- mann. How good are query optimizers, really? Proceedings of the VLDB Endowment, (3):204-214) November 2015. ISSN 2050-8097.
doi: 10. 10.144778/220083.2555554 URL
http::/dde doi wtt.:oo.......o.ooooo 1ttp:/1...477828555555555 055.778//22555882285555
2850594
12

---

Published as a conference paper at ICLR 2026
Viktor Leis, Bernhard Radke, Andrey Gubichev, Atanas Mirchev, Peter Bonez, Alfons Kemper, and Thomas Neumann. Query optimization through the looking glass, and what we found running the Join Order Benchmark. The VLDB Journal. pages 1-26, 2017. ISSN 1066-8888, 0949-877X. doi: 10. 10.10077-00778.017-4400....
0007/800778-017-0400--
Brian Lester, Rami A1-Rfou, and Noah Constant. The power of scale for parameter-efficient prompt tuning. In Conference on Empirical Methods in Natural Language Processing, 2021. URL
httpps:/aat semanticscholart wrgg/orppuse 23:2299662888
Xiang Lisa Li and Percy Liang. Prefix-tuning: Optimizing continuous prompts for generation. Pro- ceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), pages 4582-4597, 2021.
URL atttps://ap. semanticscho org.aar.rrrr/oooo Sarg//CrrpssD:
230433941
Tennison Liu, Nicolás Astorga, Nabeel Scedat, and Mihacla van der Schaar. Large language models to enhance bayesian optimization. In The Twelfth International Conference on Learning Repre- sentations, 2024. URL httpps:///peerreeeew net ?
Shayne Longpre, Le Hou, Tu Vu, Albert Webson, Hyung Won Chung, Yi Tay, Denny Zhou, Quoc V Le, Barret Zoph, Jason Wei, et al. The flan collection: Designing data and methods for effective instruction tuning. arXiv preprint arXiv:2301. 300::2330111,88, 2023.
Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. International Confer- Learning Representations, 2019. URL https: bettps://openrrett http:////pppeeee..eee/eo
Bkg6RiCqY7
Ali Madani, Ben Krause, Eric Greene, Subu Subramanian, Benjamin Mohr, James M Holton, Jose Luis Olmos, Jr, Caiming Xiong, Zachary Z Sun, Richard Socher, James S Fraser, and Nikhil Naik. Large language models generate functional protein sequences across diverse families. Nat. August 2023.
Biotechnol., (8): 1ee1-19991100
Ryan Marcus, Parimarjan Negi, Hongzi Mao, Chi Zhang, Mohammad Alizadeh, Tim Kraska, Olga Papaemmanouil, and Nesime Tatbul. Neo: learned query optimizer. Proceedings of the VLDB Endowment, (2111):1705111770 July 2019. ISSN 2199. doi: 10.147778/334226333330000 URL http: doi org 10. 14778 33422633 3342644
Ryan Marcus, Parimarjan Negi, Hongzi Mao, Nesime Tatbul, Mohammad Alizadeh, and Tim Kraska. Bao: Making learned query optimization practical. In Proceedings of the 2021 In- ternational Conference on Management of Data, SIGMOD '21, page 1275-1288, New York, NY, USA, 2021. Association for Computing Machinery. ISBN 9781450383431. 1145 00055/4440111 0104448816.445255. 3452838
doi: 10.1145/
3448016.3452838. URLLtttps::/6d..rr///1
Natalie Maus, Haydn Jones, Juston Moore, Matt J Kusner, John Bradshaw, and Jacob Gardner. Local latent space bayesian optimization over structured inputs. In Advances in Neural Information
Processing Systems, volume 35, pages 34505-34518. Curran Associates, Inc., 2022. doi: 10.
48550WarXiv.2201.111
Swaroop Mishra, Daniel Khashabi, Chitta Baral, Yejin Choi, and Hannaneh Hajishirzi. Reframing Instructional Prommtp GPTk's Language| arXiv preprint arXh::2109.07830. 2021.
Jonas Močkus. On bayesian methods for seeking the extremum. In Optimization Techniques IFIP Technical Conference: Novosibirsk, July 1-7, 1974, pages 400 Springer, 1975.
Luigi Nardi, David Koeplinger, and Kunle Olukotun. Practical design space exploration. In 2019 IEEE 27th International Symposium on Modeling, Analysis, and Simulation of Computer and Telecommunication Systems (MASCOTS), pages 347-358. IEEE, 2019. URL https:
Parimarjan Negi, Ryan Marcus, Andreas Kipf, Hongzi Mao, Nesime Tatbul, Tim Kraska, and Mo- hammad Alizadeh. Flow-loss: Learning cardinality estimates that matter. Proc. VLDB En- dow., 14(11)200192208 jul 2021. ISSN 2150-8097. doi: 10.14778/34762493477699.. URL
htttps://doi.orge htt.:/o///...111/311111311 347622
13

---

Published as a conference paper at ICLR 2026
OpenAI, Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Floren- cia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, Red Avila, Igor Babuschkin, Suchir Balaji. Valerie Balcom, Paul Baltescu, Haiming Bao, Moham- mad Bavarian, Jeff Belgum, Irwan Bello, Jake Berdine, Gabriel Bernadert-Shapiro, Christopher Berner, Lenny Bogdonoff, Oleg Boiko, Madelaine Boyd, Anna-Luisa Brakman, Greg Brock- man, Tim Brooks, Miles Brundage, Kevin Button, Trevor Cai, Rosie Campbell, Andrew Cann, Brittany Carey, Chelsea Carlson, Rory Carmichael, Brooke Chan, Che Chang, Fotis Chantzis, Derek Chen, Sully Chen, Ruby Chen, Jason Chen, Mark Chen, Ben Chess, Chester Cho, Casey Chu, Hyung Won Chung, Dave Cummings, Jeremiah Currier, Yunxing Dai, Cory Decareaux, Thomas Degry, Noah Deutsch, Damien Deville, Arka Dhar, David Dohan, Steve Dowling, Sheila Dunning, Adrien Ecoffet, Atty Eleti, Tyna Eloundou, David Farhi, Liam Fedus, Niko Felix, Simón Posada Fishman, Juston Forte, Isabella Fulford, Leo Gao, Elie Georges, Christian Gib- son, Vik Goel, Tarun Gogineni, Gabriel Goh, Rapha GontijoLoopss, Jonathan Gordon, Morgan Grafstein, Scott Gray, Ryan Greene, Joshua Gross, Shixiang Shane Gu, Yufei Guo, Chris Hal- lacy, Jesse Han, Jeff Harris, Yuchen He, Mike Heaton, Johannes Heidecke, Chris Hesse, Alan Hickey, Wade Hickey, Peter Hoeschele, Brandon Houghton, Kenny Hsu, Shengli Hu, Xin Hu, Joost Huizinga, Shantanu Jain, Shawn Jain, Joanne Jang, Angela Jiang, Roger Jiang, Haozhun Jin, Denny Jin, Shino Jomoto, Billie Jonn, Heewoo Jun, Tomer Kaftan, Lukasz Kaiser, Ali Ka- mali, Ingmar Kanitscheider, Nitish Shirish Keskar, Tabarak Khan, Logan Kilpatrick, Jong Wook Kim, Christina Kim. Yongjik Kim, Jan Hendrik Kirchner, Jamie Kiros, Matt Knight, Daniel Kokotajlo, Lukasz Kondraciuk, Andrew Kondrich, Aris Konstantinidis. Kyle Kosic, Gretchen Krueger, Vishal Kuo, Michael Lampe, Ikai Lan, Teddy Lee, Jan Leike, Jade Leung, Daniel Levy. Chak Ming Li, Rachel Lim, Molly Lin, Stephanie Lin, Mateusz. Litwin, Theresa Lopez, Ryan Lowe, Patricia Lue, Anna Makanju, Kim Malfacini, Sam Manning. Todor Markov, Yaniv Markovski, Bianca Martin, Katie Mayer, Andrew Mayne, Bob McGrew, Scott Mayer McKinney, Christine McLeavey, Paul McMillan, Jake McNeil, David Medina, Aalok Mehta, Jacob Menick, Luke Metz, Andrey Mishchenko, Pamela Mishkin, Vinnie Monaco, Evan Morikawa, Daniel Mossing, Tong Mu, Mira Murati, Oleg Murk, David Mély, Ashvin Nair, Reiichiro Nakano, Ra- jeev Nayak, Arvind Neelakantan, Richard Ngo, Hyeonwoo Noh, Long Ouyang. Cullen O' OeKeefe, Jakub Pachocki, Alex Paino, Joe Palermo, Ashley Pantuliano, Giambattista Parascandolo, Joel Parish, Emy Parparita, Alex Passos, Mikhail Pavlov, Andrew Peng, Adam Perelman, Filipe de Avila Belbute Peres, Michael Petrov, Henrique Ponde de Oliveira Pinto, Michael, Pokorny, Michelle Pokrass, Vitchyr H. Pong, Tolly Powell, Alethea Power, Boris Power, Elizabeth Proehl, Raul Puri, Alec Radford, Jack Rae, Aditya Ramesh, Cameron Raymond, Francis Real, Kendra Rimbach, Carl Ross, Bob Rotsted, Henri Roussez, Nick Ryder. Mario Saltarelli, Ted Sanders, Shibani Santurkar, Girish Sastry, Heather Schmidt, David Schnurr, John Schulman, Daniel Sel- sam, Kyla Sheppard, Toki Sherbakov, Jessica Shieh, Sarah Shoker, Pranav Shyam, Szymon Sidor, Eric Sigler, Maddie Simens, Jordan Sitkin, Katarina Slama, Ian Sohl, Benjamin Sokolowsky, Yang Song. Natalie Staudacher. Felipe Petroski Such, Natalic Summers, Ilya Sutskever, Jie Tang, Nikolas Tezak, Madeleine B. Thompson, Phil Tillet, Amin Tootoonchian, Elizabeth Tseng, Pre- ston Tuggle, Nick Turley, Jerry Tworek, Juan Felipe Cerón Uribe, Andrea Vallone, Arun Vi- jayvergiya, Chelsea Voss, Carroll Wainwright, Justin Jay Wang, Alvin Wang, Ben Wang. Jonathan Ward, Jason Wei, CJ Weinmann, Akila Welihinda, Peter Welinder, Jiayi Weng. Lilian Weng, Matt Wiethoff, Dave Willner, Clemens Winter, Samuel Wolrich. Hannah Wong, Lauren Work- man, Sherwin Wu, Jeff Wu, Michael Wu, Kai Xiao, Tao Xu, Sarah Yoo, Kevin Yu, Qiming Yuan, Wojeiech Zaremba, Rowan Zellers, Chong Zhang, Marvin Zhang, Shengjia Zhao, Tianhao Zheng, Juntang Zhuang, William Zhuk, and Barret Zoph. Gpt-4 technical report, 2024. URL. https:// www.//og/////o/ooooooo/oeoe /2003 08774
Massimiliano Patacchiola, Jack Turner, Elliot J. Crowley, Michael O' Boyle, and Amos J Storkey. Bayesian meta-learning for the few-shot setting via deep kernels. In Ad- vances in Neural Information Processing Systems. Curran Associates, Inc., 2020. URI. http:///pppppeeeeeeeeeeeeeeeeeee files/pappxr222///Eii pdff
Valerio Perrone, Rodolphe Jenatton, Matthias W Seeger, and Cedric Archambeau. Scalable Hy- perparameter Transfer Learning. In Advances in Neural Information Processing Systems, vol- ume 31. Curran Associates, Inc., 2018. URL htt httpt:pss//ppccceeeiiii new.ips..crerrroooooo
html
14

---

Published as a conference paper at ICLR 2026
Ali Rahimi and Benjamin Recht. Random Features for Large-Scale Kernel Machines. In Advances in Neural Information Processing Systems, volume 20. Curran Associates, Inc., 2007. URL
013a006603bbcc332eee Feb8 wwtpppppppppp.....pppppppppappee f.les//pper//00////// bebbstract.htaa..
Carl Edward Rasmussen. Gaussian processes in machine learning. In Summer School on Machine Learning. pages 63-71. Springer, 2003.
Nicolas Schilling, Martin Wistuba, and Lars Schmidt-Thieme. Scalable hyperparameter optimiza- tion with products of gaussian process experts. In Paolo Frasconi, Niels Landwehr, Giuseppe Manco, and Jilles Vreeken, editors, Machine Learning and Knowledge Discovery in Databases, pages 33 48, Cham, 2016. Springer International Publishing. ISBN
Alexander Shypula, Aman Madaan, Yimeng Zeng, Uri Alon, Jacob Gardner, Milad Hashemi, Gra- ham Neubig, Parthasarathy Ranganathan, Osbert Bastani, and Amir Yazdanbakhsh. Learning performance-improvie code edits. In International Conference on Learning Representations, 2024. doi: arXiv:2302.07867.
Eero Siivola, Andrei Paleyes, Javier González, and Aki Vehtari. Good practices for Bayesian opti- mization of high dimensional structured spaces. Applied Al Letters, 2(2)= 2021.
Jasper Snoek, Hugo Larochelle, and Ryan P. Adams. Practical bayesian optimization of machine learning algorithms, 2012. URL stttps://ariis: www.//sg/aaaaaaaaooooo... /
wtt.://///ba..........2 2tt.bb//120..9922 2944
Samuel Stanton, Wesley Maddox, Nate Gruver, Phillip Maffettone, Emily Delaney, Peyton Green- side, and Andrew Gordon Wilson. Accelerating bayesian optimization for biological sequence design with denoising autoencoders, 2022. URL http://s./.arrt.ooooo//////////.
Kevin Swersky, Jasper Snoek, and Ryan Adams. Multi-Task Bayesian Optimization. In Advances in Neural Information Processing Systems, volume 26. Curran Associates, Inc., 2013. URL F33bal o httpa//pppppprreeseeeeeeeeeeeeee wwwesfiileaaaee /ileee//ppporr//11//aa.....
Jeffrey Tao, Natalie Maus, Haydn Jones, Yimeng Zeng, Jacob R. Gardner, and Ryan Mareus. Learned offline query planning via bayesian optimization, 2025. URL/ttts::/axxiv..o
abs/2502. 05256
Marcelo T. Torres, Yimeng Zeng, Fangping Wan, Natalie Maus, Jacob Gardner, and Cesar de Fuente-Nunez. A generative artificial intelligence approach for antibiotic optimization. bioRxiv, 2024. doi: 00.1100/2024.112776655777 early 0tLL///222//11//222///444
URL https://www wwwp:wwwwwwwww.cco/. wwtp:xixwiivgg/oteeeeeee
625757
Austin Tripp, Erik A. Daxberger, and José Miguel Hernández-Looato. Sample-efficient optimization in the latent space of deep generative models via weighted retraining. In Advances in Neural Information Processing Systems 33, 2020.
Fangping Wan, Marcelo
Torres, Jacqueline Peng. and Cesar de la Fuente-Nunez. Deep- learning-enabled antibiotic discovery through molecular de-extinction. Nature Biomedical En- gineering, 87708854-77111 Jul 2024. iSSN 2167-846X. doi: 10. 10.10 URI.
https://dol org 10. 1038 s45511 024 01201-11
Haorui Wang, Marta Skreta. Cher-Tian Ser, Wenhao Gao, Lingkai Kong, Felix Streith-Kalthor Chenru Duan, Yuchen Zhuang, Yue Yu, Yanqiao Zhu, Yuanqi Du, Alán Aspuru-Guzia Kirill Neklyudov, and Chao Zhang. Efficient evolutionary search over chemical space with large lan- guage models, 2024a.
16976
Zi Wang, George E. Dahl, Kevin Swersky, Chansoo Lee, Zachary Nado, Justin Gilmer, Jasper Snoek, and Zoubin Ghahramani. Pre-trained Gaussian Processes for Bayesian Optimization. Journal of Machine Learning Research, 25(212):1833, 2024b. ISSN 1533-7928. URL. http: // mm.ml..
Dxt/ppapers/s25/25.. 0269. html
Martin Wistuba and Josif Grabocka. Few-shot bayesian optimization with deep kernel surrogates.
arXiv preprint 10t:::211106677 2021.
15

---

Published as a conference paper at ICLR 2026
Chengrun Yang, Xuezhi Wang, Yifeng Lu, Hanxiao Liu, Quoc V. Le, Denny Zhou, and Xinyun Chen. Large language models as optimizers, 2024. hww.tttss//aaxii...tom www.xiii.oor/aaaoo /3300
03409
Zongheng Yang, Wei-Lin Chiang, Sifei Luan, Gautam Mittal, Michael Luo, and Ion Stoica. Balsa: Learning a query optimizer without expert demonstrations. In Proceedings of the 2022 Inter-
national Conference on Management of Data, SIGMOD/ODS '22. ACM, June 2022. doi:
10.1145/511221:3317855 CRRLAtt::/// http://xxxxoooooo/o.o/ooooo/ .ttp://t..oo..i/....oo..... 112p//00...4//31222..... 3517888
Eric Zelikman, Eliana Lorch, Lester Mackey, and Adam Tauman Kalai. op (stop): Recursively self-improving code generation, 2024. BRl/ittpeee/arrxiv.rr////a/////p
02304
Xinyi Zhang, Zhuo Chang, Yang Li, Hong Wu, Jian Tan, Feifei Li, and Bin Cui. Facilitating database tuning with hyper-parameter optimization: a comprehensive experimental evaluation. Proceed- ings of the VLDB Endowment, (5(9): 188-1821, May 2022. ISSN 2050-8097. doi: 10.14778/ 1338598.3538600 URL httLh:t:/:/// acm www. wwtp:/i//g/oo/oooooooooooooo/o/o http//////..11//1//11//1 14778 007777//553588.. 3511/335333888333844484
Rong Zhu, Wei Chen, Bolin Ding, Xingguang Chen, Andreas Pfadler, Ziniu Wu, and Jingren Zhou. Lero: A learning-to-rank query optimizer. Proceedings of the VLDB Endowment, 16
(6):1466-147) February 2023. ISSN 2150-8097.
doi: 10.14778//388110035883 URL
doi. 0tt....///...11117//333110000100 3583160
16

---

Published as a conference paper at ICLR 2026
A PROMPT DETAILS.
Iillustrate the prompt templates used for generating optimized query plans and peptide sequences, with GPT-40-MINI-071 18. Figure shows the template for database query optimization, where the system acts as assistant providing efficient orderings for given SQL query. Figure 5 displays the template for antimicrobial peptide design, where the system's role is to modify peptide sequences enhance antimicrobial activity.
and 5
System: You are a helpful assistant that provides efficient join orderings for given queries. User: (SQL query to be optimizea) Assistant: (Optimized zed query plan)
Figure 4: The prompt template used for prompting GPT-4O-MINI for generating optimized query plans.
System:
You are speciali zed assistant that modifies peptide sequences to enhance intimicrobial activity. Make up to 25 sequence modifications based on known antimicrobial peptide properties such as: positive charge, bydrophobicityy, and amphipathicity.
User: (Seed peptide to be modified) Assistant: fied peptide)
Figure 5: The prompt template used for prompting GPT-4O-MINI for generating optimized peptide sequences.
B COMPUTE DETAILS.
Hardware. Experiments ran on an internal cluster (18 GPUs across two servers): one server with 8x NVIDIA RTX A6000 (48 GB each; dual-socket CPU with 48 logical threads per socket) and one server with 10x NVIDIA RTX A5000 (24 GB each; dual-socket CPU with 24 logical threads per socket).
Scope. We report (i) per-task GPU-hours, (ii) aggregates 50 tasks and 1,426 tasks, and (iii) LLM fine-tuning and inference usage (tokens, USD, and local GPU-hour equivalents). Database (DB) query-plan used one GPU per task unless specified.
RUNTIME AND COST OVERVIEW
Single-task Bayesian Optimization (STBO). Each DB run required 5020 GPU-hours (avg. Rs 18). Across 1 1,426 tasks this is 25,000 GPU-hours, which is the dominant component of total compute.
Baseline MTBO methods.
SGPE/POGPE: > 100 GPU-hours per task; capped runs at 1 110 GPU-hours. These ap-
proaches did not reach 1,000 oracle calls within reasonable time, and their cost increases quickly with the number of tasks.
DKT/FSBO (10/20/50): 15-20 GPU-hours per task (similar to STBO). The extra feature- extractor cost minor relative GP training the same data.
17

---

Published as a conference paper at ICLR 2026
Method
GPU-hours / task
GPU-hours 50 tasks
GPU-hours/ 1,426 tasks
STBO
15-20
~90
~22.500-28,,72 Infeasible (quadratic scaling)
SGPE/POGPE DKT/FSBO
>10 (capped at 110)
55,000 (incomplete)
15 20
~00
~22,500 28,720 (cst.) ~ 22,750-28,997 (+1% vs. STBO)
BOLT + STBO
964 (+7% vs. STBO)
Table 3: Runtime comparison on DB query-plan tasks. BOLT's fine-tuning and inference add small, amortized overhead relative to the BO inner loop: ~7% at 50 tasks and 1% at 1,426 tasks.
BOLT overhead (fine-tuning + inference).
LLM fine-tuning (GPT-40-MINI -0718 via API): The largest used 26M tokens (~S78). Summed over BOLT-893/1 138/1426, fine-tuning consumed 60M tokens (~$1800, roughly ~400 GPU-hours if performed locally RTX A6000s (conservative equivalence).
Alternative local FT (Qwen 2.5-7B): 32 GPU-hours total Ax0000 for 8 hours).
Inference generate initializations: Sampling 50 candidates per task from locally fine-tuned model took ~~1 GPU-minute per task (about 24 GPU-hours across 1,426 tasks), negligible com- pared BO.
Takeaways. (i) The vast majority of compute is spent in the BO inner loop (STBO comparable inner loops in MTBO baselines). (ii) BOLT's
plus sampling a small batch of initial candidates-is small and amortizes quickly; ~7% at 50 tasks and only ~1% by 1,426 tasks (Table (iii) SGPE/POGPE were substantially slower per task and did not scale to our full regime. (iv) Even when counting fine-tuning using conservative local GPU-hour equivalent (~~400 GPU- hours) rather than low-cost API usage ($ 188 total), BOLT's added compute remains marginal relative to the 25k GPU-hours of BO.
Token/API usage. Across BOLT- 138/1426, fine-tuning used 60M tokens (~$180). Initial- ization inference across all DB tasks required ~44 GPU-hours in total.
C IMPLEMENTATION DETAILS.
C.1 VAE TRAINING AND UPDATE DETAILS.
For optimization structured input spaces, we utilize latent space Bayesian optimization via Variational Autoencoders (VAEs). The training and updating procedures for these VAEs differ by domain:
Database query plans. For the DB task, we use pre-trained query plan VAE (trained once on approximately 1.17M synthetic query plans generated from the schema), This VAE is not retrained or updated during our BO experiments.
Antimicrobial peptides. For the peptide task, the VAE maps amino-acid sequences to 256- dimensional latent space. Unlike the DB domain, this VAE jointly updated with the surrogate model every 10 optimization steps. None of the L = 1,000 extinct seed peptides (900 training 100 validation) used in our optimization tasks were included in the VAE's pre-training dataset.
C.2 DKT/FSBO IMPLEMENTATION DETAILS.
For the antimicrobial peptide design task, a PPGPR model was trained using the GPyTorch module. This model employed fully connected network with two hidden layers, each having dimension of 256 and SiLU activations. Training parameters included a batch size of 128, a learning rate of 0.01, and 1024 inducing points for all peptide design experiments.
18

---

Published as a conference paper at ICLR 2026
For the database query plan optimization task, the PPGPR model utilized fully connected network with two hidden layers, each with a dimension of 64 and SiLU activations. A batch size 16, a learning rate of 0.01, and 1024 inducing points were used for these experiments.
For both tasks, 50 STBO optimization trajectories were randomly selected. The DKT/FSBO- 10/20/50 models were trained using the first 10, 20, 50 of these trajectories, respectively. All models were trained for 20 epochs.
C.3 POGPE/SGPE IMPLEMENTATION DETAILS.
Similarly Section C.2 for the antimicrobial peptide design task, each expert model was PPGPR model implemented with GPyTorch, with a fully connected network with two hidden layers, each with dimension of 256 and SiLU activations. A batch size of 128, learning rate of 0.01, and 1024 inducing points were used.
For the database query plan optimization task, each expert model uses a fully connected network with two hidden layers, each with dimension of 64 and SiLU activations. The training used a batch size of 16, a learning rate of 0.01, and 1024 inducing points.
The same 50 STBO optimization trajectories from Section C.2 were used, and the first 5/ 5/10/20 tra- jectories were used to train the POGPE/SGPE expert models. POGPE, all experts were weighted equally. For SGPE experiments, the weighting scheme from (Schilinng et al. (2016) was adopted, where the independent GP for the target dataset carries the same weight as the entire set of experts.
C.4 OPTFORMER IMPLEMENTATION DETAILS.
For both the query plan optimization and antimicrobial peptide design tasks, GPT-4O-MINI-071 was fine-tuned on past optimization trajectories. To stay within context window limits, a maximum input context length of 100 trials and an output of 20 trials were used. The objective value ranges for both tasks were discretized into 1000 equidistant points. The training sets were constructed by randomly subsampling two trajectories of length 120 from the optimization trajectories. The query plan optimization task is trained on 27.4 million tokens and the antimicrobial peptide design task is trained on 4.8 million tokens. Both models were trained for epoch with batch size of 20 and an OpenAl learning rate multiplier 1.8.
Optimization was initialized using the same points single-task BO. During inference, a constant temperature of 0.7 was used. To manage inference token usage, a batch size of 20 was employed, where the model predicted the next 20 trials based the previous 100 trials. This was important as experiments ran for 4,000 (query plan) or 20,000 (peptide design) trials.
C.5 LLAMBO IMPLEMENTATION DETAILS.
The end-to-end LLAMBO method was utilized, leveraging GPT-4O-MINI-0711 for several com- ponents: generating candidate solutions, serving as surrogate model for the objective function (via in-context learning), and acting as conditional sampler to generate candidates for specific tar- get values. Similar to Optformer, a maximum input context window of 100 trials was enforced to prevent exceeding context limits.
The hyperparameters from the original LLAMBO paper were adopted, including an exploration hyperparameter a 0.1, and M 20. For the surrogate model, we sample K 10 MC predictions to compute the empirical estimates. Consistent with the LLAMBO paper, we use the same sampling parameters with temperature of 0.7 and top-p of 0.95. A limit of 10 million maximum input tokens per experiment was used to manage computational costs.
C.6 LLM SELF-AUGMENTATION DETAILS.
For the antimicrobial peptide design task, 200 samples were generated for each of the initial 800 training peptides during each self-augmentation round. Any peptides with a predicted MIC below 8 (indicating significant antimicrobial activity) were added to the training set for the subsequent round of BOLTT
19

---

Published as a conference paper at ICLR 2026
For the database query plan optimization task, 10 samples generated for each of the 2,933 train- ing queries. Query plans with a runtime lower than the best plan generated by the BAO optimizer were added to the training set for the next round of BOLI.
D ADDITIONAL ABLATIONS.
D.1 OPEN SOURCE LLMS.
To explore the viability of open-source models for BOLT, QWEN-2.5-7B and LLLAAA331144 1-8B were fine-tuned using the identical dataset that created BOLT-1426 from GPT-40-MIN-0771 for the database query optimization task. For evaluation, 50 query plans were generated from each LLM using a sampling temperature of 0.7. The best summed query execution time across the validation tasks from these 50 samples was compared.
Both models were fine-tuned on 4 NVIDIA RTX A6000 GPUs using per-device batch size of 4, learning rate of le-5 with the AdamW optimizer (I (Loshchilov and Hutter (019), and training epochs. The results, shown Table indicate that QWEN-2.5-7B performed slightly worse than fine-tuned OPT-40-MINI-071 07188, while LLAMA-3, 1-8B showed significantly lower performance. Due to the extensive number of inference calls and multiple fine-tuning rounds required by BOLT, the primary experiments were conducted using the OpenAI API due to hardware resource limita- tions.
Model
Summed runtime
GPT-4O-MINI-0718 QWEN-2.5-7B
61.54
62.04
LLAMA-3.1-8B
155.55
Table 4: Comparing open source LLMs fine-tuned with data used fine-tune BOLT-1426 against
OpenAl models fine-tuned on the same data.
D.2 RANDOM PERTURBATIONS AROUND PRIOR SOLUTIONS.
We test whether small random perturbations around prior best solutions provide stronger initial- ization. In the latent space, sample 50 candidates per validation task within axis-aligned trust regions (TR) centered at each prior best solution, with side-lengths {0.5", 0.56 ,..., 0.5"). We then decode and evaluate these candidates.
Table DB (first 10 validation queries). Summed runtime (seconds; lower is better) when initial- izing from random perturbations of prior best solutions within latent-space trust regions. "Previous solutions" repeats Table 7 for reference. A large trust region can degrade the validity or quality of decoded candidates (marked with t).
Method
Prev. sol.
TR (0.5²)
(0.5%)
(0.5%)
(0.5*)
(0.5%)
(0.5²)
(0.5")²
(0.5%)
Summed runtime (s)
9.18
9.18
8.94
8.94
8.92
8.97
9.07
38.48
8.62
BOLT- (init only)
Observation. Local perturbations around prior best solutions offer modest gains over using the unperturbed pool (Table 5), but remain weaker than BOLT initializations. This indicates that task- conditioned sampling provides benefits beyond local neighborhood search around previous best so- lutions.
20

---

Published as a conference paper at ICLR 2026
D.3 COMPATIBILITY WITH GLOBAL BAYESIAN OPTIMIZATION
In our main experiments, we utilize local, trust-region-basee based Bayesian optimization algorithms, as they are well-suited for high-dimensional latent spaces. Too demonstrate that BOLT algorithm- agnostic and functions as a plug-and-play initialization module, we evaluate its compatibility with a standard global BO approach.
We implemented a recent global BO algorithm tailored for high dimensions (Hvarfner et al. 2024 within the same VAE latent space on the database query optimization task. We compared global BO initialized with the standard BAO baseline against global BO initialized with BOLT-1420 samples. As reference, we also include our default local (trust-region) BO initialized with BOLT-1426 We evaluate performance up to 2,000 oracle calls rather than the full 4,000-call budget due to limited compute budget.
Table reports the summed runtime on the first 10 validation queries. Before optimization begins, the baseline BAO initialization yields a summed runtime of 12.10s, while the BOLT-1426 initializa- tion starts at much stronger 7.43
Table 6: DB (first 10 validation queries). Summed runtime (seconds; lower is better) comparing global and local BO architectures. BOLT initializations substantially improve optimization trajecto- ries regardless of the underlying BO algorithm.
Method
1,000 oracle calls
2,000 oracle calls
Global BO + BAO init
9.44
9.03
Global BO + BOLT-1426 init
6.94
6.92
Local (trust-region) BO BOLT-1426 init
6.72
6.59
Observation. BOLT initializations substantially improve the performance of global BO (e.g., 9.03 - 6.92 at 2,000 calls). While local (trust-region) BO maintains slight overall edge over global BO this specific environment, the relative gain provided by BOLT remains consistent. This confirms that BOLT effectively identifies high-quality basins for new tasks, accelerating both global and local acquisition strategies.
D.4 INITIALIZING WITH PRIOR BEST SOLUTIONS.
We test whether reusing the best solution from each previously optimized training task is compet- itive generic initializer for new tasks. We collect the best-performing solution from every training task completed by BOLT-1426 and form pool of "previous solutions." " For evaluation, we consider the first 10 validation queries (DB domain), treat this pool as the initialization set (same size as other initializers), and measure the summed query runtime (lower is better). We compare against: (i) single-task BO (STBO) initialized with the standard baseline set; (ii) BOLT initializations from fine-tuned models with different training sizes; and (iii) a full BO run initialized by BCLT-1426. No further LLM fine-tuning occurs during this evaluation.
Method
Summed runtime (s)
BOLT- 1426 + BO BOLT-1426
6.43
7.43
BOLT-1138
8.65
BOLT-893
9.08
STBO Previous solutions
8.23
9.18
Table 7: DB (first 10 validation queries). Summed runtime (seconds; lower is better) under dif- ferent initialization strategies. 1B0LT-1426+B0V runs full BO after initializing with BOLT-1426 samples; other rows report initialization-on/ performance the optimization start.
Simply reusing prior best solutions less effective than model geder-generated initializations, and falls behind both STBO and all BOLT variants considered (Table 7). This suggests cross-task misalign-
21

---

Published as a conference paper at ICLR 2026
ment: best solutions for earlier tasks do not align well with tasks, while BOLT samples tailored to the provided task description.
D.5 SAMPLING ERROR RATES AND TEMPERATURE ABLATIONS
Generation validity. We include lightweight filter in our pipeline remove characters that do not correspond to strings of integers (for DB plans) or valid amino acids (for peptides). This is primarily to ensure pipeline robustness, as non-valid samples are very rare at the sampling tem- peratures we use. To quantify this, we generated 50 samples for 100 DB tasks (5,000 total) across various temperatures T. shown in Table at our default DB temperature of 0.7, we generate 100% valid query plans. Only when pushing the temperature above 1.2 do observe noticeable increase in failed generations.
Table 8: DB (validation set), Generation validity across 5,000 samples per temperature setting.
Valid samples
Invalid samples
Valid (%)
0.1
4,999
99.98
0.3
5,000
0
100.00
0.5
5,000
100.00
0.7
5,000
0
100.00
1.0
4,999
99.98
1.2
4,971
99.42
1.5
4,423
577
88.46
22

---

Published as a conference paper at ICLR 2026
Temperature sensitivity on DB. We study the effect of the sampling temperature initialization quality for the DB domain. For each temperature, we report the best-of-50 summed runtime across the standard validation set. We consider both -0TT--0M-N--071 18 and QwEN-2.5-7B. each fine- tuned on the same training data. As shown in Table higher temperatures can slightly improve performance by increasing proposal diversity. We utilize T 0.7, which is commonly used for LLM sampling and is close to optimal without affecting reported results.
Table 9: DB (validation set). Best-of-50 summed runtime (seconds; lower is better) vs. sampling temperature T.
T GPT-4O-MINI-0718
QWEN-2.5-7B
0.1
84.97
84.42
0.3
65.88
69.14
0.5
62.19
63.97
0.7
61.54
62.04
1.0
60.09
61.25
1.2
59.78
62.61
1.5
60.50
64.45
Temperature sensitivity on peptides. For the peptide task, we selected a slightly higher default temperature (T' 1.0) to increase the diversity of the 1,000 generated samples while maintain- ing adherence to the task's seed similarity constraint. Table 10 reports the uniqueness fraction, fraction, and summed MIC as function of T (1,000 samples per seed; 20 validation seeds). T = 1.0 maximizes the effective fraction peptides (both unique and within constraint) and yields the best summed MIC, confirming that it provides the best tradeoff between novelty and staying in-distribution.
Table 10: Peptide temperature ablation. For each sampling temperature T, we report the fraction of unique samples, the fraction satisfying the similarity constraint, the effective fraction (unique & in-constraint), and the summed best MIC across 20 validation seeds (lower is better).
Unique frac.
In-constraint frac.
Effective frac.
Summed best MIC
0.1
0.01620
0.87340
0.01285
155.91
0.3
0.12315
0,85800
0.09085
111.85
0.5
0.33190
0.83030
0.24170
103.94
0.56620
0.78585
0.40035
95.10
1.0
0.83990
0.69090
0.54350
93.30
1.2
0,93765
0,59700
0,53785
100.67
1.5
0.98655
0.39660
0.38425
103.56
23

---

Published as a conference paper at ICLR 2026
D.6 INITIALIZATION STAGE COMPARISONS AND A NO-FINETUNING BASELINE
We report initialization only quality as a function of the first A oracle calls on held-out tasks for both domains. We also include no-finetuning LLM baseline (BOLT-0) in the DB setting.
Peptides. We show summed (unnormalized) predicted MIC across 20 validation peptides at l {1, 100, 200, 500, 1000} oracle calls (lower is better).
Table 11: Peptides (20 validation tasks). Summed unnormalized MIC vs. oracle calls & at the initialization stage.
k
BOLT-10
BOLT-20
BOLT-50
BOLT-600
STBO/MTBO
1
1204.0525
1120.9886
1057.1308
564.4510
5727.7421
100
135.4973
147.2456
112.9047
100.5810
1551.4898
200
120.4763
134.8256
107.1327
96.7752
1111.7161
500
109,1782
122.1278
101.3638
94.2405
792.5464
1000
107.5492
119.7096
97.1403
92.1007
625.9521
Database queries. We show summed runtime across 10 validation queries at l: {1. 10, 20, 50} oracle calls. We include BOLT-O (no fine-tuning), BOLT-50, BOLT-1422, and STBO/MTBO.
Table DB (first 10 validation queries). Summed runtime (seconds) vs. oracle calls & at initial- ization. BOLT-0 uses untuned LLM.
k
BOLT-0
BOLT-50
BOLTT-1426
STBO/MTBO
1
53.2584
13.9788
15.1161
10
13.8501
7.6380
13.7863
20
11.5664
7.6343
13.3761
50
182.9500
10.3764
7.4340
12.0967
The fine-tuned BOLT initializations improve markedly with scale and outperform both STBO ini- tializations and the untuned BOLT-O baseline early in the budget (Tables 12
D.7 ADDITIONAL OUTER-LOOP AND ROBUSTNESS ABLATIONS
Outer-loop sensitivity on DB. Table 13 reports Best@ $50 for three disjoint BOLT-200 runs, which lie between BOLT-50 and BOLT-893, indicating that the particular traces used fixed T matter less than the overall scale. Table 14 varies K E 21,2,5, 5, 10, 20) at fixed T 893 and shows better performance K increases.
Table 13: DB outer-loop sensitivity which BO traces used. @5 summed runtime (sec- onds; lower is better) across the 99 validation queries for three disjoint B-20-220 runs and baselines (no self-augmentation).
Model
Best@ 50 runtime (s) J
BOLT-50
87.84
BOLT-200 (run
79.67
BOLT-200 (run
84.09
BOLT-200 (run 3)
84.21
BOLT-893
82.31
24

---

Published as a conference paper at ICLR 2026
Table 14: DB outer-loop sensitivity the number of top solutions per task. @ summed runtime (seconds; lower is better) for BOLT trained on the same 893 tasks with varying K.
Top-K Best@ 50 runtime (s) |
78.80
77.40
76.27
10
82.31
20
69.56
Context shuffling on DB. Table 15 shows that randomly shuffling the mapping between SQL contexts and BOLT-1426 plans degrades @e from the original BOLT score far worse than BAO, confirming that BOLT relies on aligned task descriptions rather than memorizing global pool of good plans.
Table 15: DB context shuffling ablation. Best-of-50 summed runtime (seconds; lower is better) across validation queries when breaking the alignment between SQL contexts and BOLT-11226 plans.
Method
Best-of-50 runtime (s) +
BAO initialization
106.72
BOLT-1426 (aligned contexts)
61.54
BOLT-1426 (shuffled contexts)
402.61
Random and local-search baselines. Table 16 summarizes random VAE-latent search, random query-space search, trust-region perturbations around prior best solutions, and local search around BOLT-1426 samples. Both random strategies and purely local perturbations underperform BOLT initialization, and even local search around BOLT-1422 remains weaker than running the full BO loop initialized from BOLT-1426, suggesting that cask-conditioned proposals plus BO are essential.
DB random and local-seacch-stlle baselines on the first 10 validation queries. Entries report summed runtime (seconds; lower better) under comparable or larger oracle budgets.
Table 16:
Method
Summed runtime (s) 4
BOLT-1426 + BO
6,43
ROL1T-1426 local search (latent perturbations)
6.89
RO1.T-1426 init only (50 LLM samples)
7.43
TR perturbations around prior best solutions (50 samples)
8.62
Query-space random (4,000 samples, capped)
9.42
VAE-latent random (4,000 samples, capped)
11.97
25

---

Published as a conference paper at ICLR 2026
E LLM USAGE.
LLMs were fine-tuned for generating improved initialization points for BQ runs as part of BOLT. LLMs were used to improve this paper's writing and presentation and assist in code implementation (e.g., co-pilot auto-completion). LLMs were not involved in generating or refining research ideas, experimental design, or theoretical developments.
26


Reviews:
[{"review_id": "lspBA8Q1G8", "reviewer": "Reviewer_AGbs", "summary": "This paper addresses the problem of scaling with multi-task Bayesian optimization and proposes the Bayesian Optimization with LLM Transfer (BOLT) method. BOLT is a simple approach that uses LLMs to generate candidates for new tasks, which provides strong initializations for BO. BOLT scales MTBO without saturation, effectively handling MTBO settings up to 1500 tasks.\nThe experiments on database query optimization and antimicrobial peptide design showed that BOLT is effective in providing high-quality initializations, which yield strong few-shot performance, outperforming LLM-based MTBO methods.", "questions": "- How would the performance of BOLT change if task contexts are perturbed? Could authors discuss robustness to context perfurbations? To test LLM's generalization performance vs. memorization?\n\n\n- See also the Weaknesses above.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "-  The proposed method BOLT is simple and modular, i.e., can be plugged into any BO loop.\n\n- BOLT avoids saturation observed in common-shared GP methods and provides scaling.\n\n- BOLT is empirically evaluated on diverse real-world use-cases, particularly, two high-throughput domains: database query optimization and antimicrobial peptide design.", "weaknesses": "-  BOLT requires a task description context that can be used in an LLM prompt to define the task. This excludes common MTBO settings.  Although the authors raise this point as a limitation, they still state BOLT as broadly scalable MTBO. A more precise scope statement or a counter-example domain would help to clarify this better.\n\n- How sensitive is BOLT's performance to the outer loop? The ablations on top-k size for fine-tuning and fine-tuning frequency would have provided a better sensitivity analysis.\n\n- It would strengthen the empirical effectiveness of BOLT better if the methods are analyzed under a fixed compute budget. That is, in terms of the compute budget, would BOLT still outperform if the total cross-task compute, including oracle evaluations and finetuning cost, is set to the same budget across methods? \n\n\n- MINOR:\n\n- - Line 64-65, the references for Optformer and LLAMBO should be added.\n- - Line 457: \"illustrates\" should be corrected as \"illustrate\".", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "XdI7YRoUNk", "reviewer": "Reviewer_KGBF", "summary": "The paper presents BOLT, a multi-task Bayesian optimization method that transfers information between tasks by fine-tuning a language model. Unlike other approaches, information transfer is done at the initialization level, such that initial values for the Bayesian Optimization procedure are chosen increasingly effectively as information from other tasks are incorporated into the language model. The authors demonstrate substantial improvements over earlier methods - in particular, they show better scaling with the number of tasks.", "questions": "### Questions\nline 190. *\"we extract the top-K observations from each of the T runs completed so far.\"* Don't you risk getting very similar solutions using this approach. Wouldn't it make sense to impose some diversity criterion?\n\nline 268. *\"we filter out characters that do not correspond to strings of integers or valid amino acids for the respective tasks.\"*. How frequently do such non-valid sample errors occur? It this a problem in practice?\n\nline 280. *\"l points are sampled using a temperature parameter of 0.7 unless otherwise specified.\"*. For the two problems you use two different temperature hyperparameters. How are these tuned, and is the performance very sensitive to this choice?\n\n### Minor comments\n\nline 257. I found it difficult to assess whether a budget of 200,000 oracle calls is reasonable in practice. You could consider adding a note on this. \n\nFigure 1. The labels in this figure are difficult to read, both due to the colours and the small font. Considering changing this.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "**Originality**. To paper introduces a novel way to use language models to improvement multi-task BayesOpt. \n\n**Quality**. The method is described in detail, the empirical results support the stated claims. \n\n**Clarity**. The paper is well-written, clearly explaining the idea. A reproducibility statement is included to clearly state what code will be made available upon acceptance.\n\n**Significance** The paper reports significant improvements over the state of the art. The method is also quite simple, and is thus likely to find real-world application.", "weaknesses": "Although the paper presents several succesful applications, it does not provide a clear picture of when it fails. Limitations are discussed in broad terms, but the paper would benefit from e.g. a more careful analysis of how sensitive the performance is to the similarity between tasks. For example, one could imagine an experiment on the peptide design task where performance was reported as a function of the degree of similarity between peptides, to see if it broke down with very low similarities.\n\nAs is explicitly stated in the paper, the method focuses on large-data domains. However, it remains unclear how much data is typically necessary, and whether the approach could find application in small-data regimes with more restricted fine-tuning procedures. Any insights on this matter would be useful for a reader that is considering implementing the method.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "K0c101nHQP", "reviewer": "Reviewer_Yjw8", "summary": "&nbsp;\n\nThe authors introduce Bayesian optimization with LLM transfer (BOLT), a framework for multitask Bayesian optimization (MTBO) that leverages supervised finetuning (SFT) of LLMs to propose initial candidates for unseen tasks. The method is applied to database query planning and peptide design. Although the idea is interesting, I have concerns regarding the overall experiment design as well as the implementations of the baselines. Furthermore, I have concerns over the reproducibility of the work given that the codebase is not released. If these issues can be addressed in the rebuttal I will be willing to raise my score.\n\n&nbsp;", "questions": "&nbsp;\n\n1. On line 155, the authors state that each training task comprises the top-K observations from the trajectory for each of $t$ tasks. Are these the top-K optimal observations or simply the top-K observations identified in a BO procedure. If the latter, how does one ensure consistency in the efficacy of the BO procedure used to generate the traces for each training task? Update: This is clarified to be the latter on page 4 of the paper. In this case it may be worth running a sensitivity analysis on the BO runs used to initialize the LLM with the top-K observations albeit this would be costly from a compute perspective.\n\n2. In Figure 1, it is not clear how different tasks are treated. How does the number of oracle calls given in the x-axis related to the number of test tasks? What are the errorbars reported over?\n\n3. I take it that BOLT-10, BOLT-20 etc. refer to the number of training tasks the LLM has been fine-tuned on? It might be worth emphasizing this for the reader in the experient section or the figure captions since the meaning of the BOLT-T notation appears only to be provided in Section 3. Update: This appears to be explained in the caption of Figure 2 but not in Figure 1.\n\n4. For self-augmentation I fail to see the point of saving time on BO computation under the assumption that querying $f(\\mathbf{x})$ (scoring under the problem's oracle in the authors' words) is a more expensive process relative to re-training the BO surrogate. Obviously the authors' BO runs were computationally intensive but is there an implicit assumption that for self-augmentation to be useful, evaluating $f(\\mathbf{x})$ must be relatively cheap compared to the cost of running BO?\n\n5. What is the task for the results reported in Table 4? The database task presumably?\n\n&nbsp;", "limitations": "", "rating": 4, "confidence": 5, "soundness": 2, "presentation": 3, "contribution": 3, "strengths": "&nbsp;\n\nThe method is novel and timely, leveraging advances in LLMs for multitask Bayesian optimization (MTBO). It is conceivable that LLMs can be strong meta learning tools for MTBO albeit the tradeoffs against existing methods in terms of performance and/or compute cost are unclear. \n\n&nbsp;", "weaknesses": "&nbsp;\n\nI summarize my comments in major and minor points. In particular points 2,3, and 4 are worth prioritizing for the authors to receive an upgraded score.\n\n&nbsp;\n\n**__MAJOR POINTS__**\n\n&nbsp;\n\n1. In terms of the structure of the paper, the background on the problem settings on page 3, namely antimicrobial peptide and database query plan optimization would be better placed in the appendix. Given that the authors are introducing a general purpose optimization methodology at a machine learning conference, readers are more likely to be interested in comparable optimization methods rather than the specifics of the applications considered. Given that the authors leverage LLMs, an expanded related work section on meta learning and/or transfer learning in BO as well as other attempts to use LLMs for BO may be more appropriate.\n\n2. The absence of an anonymous GitHub or similar in the submission raises concerns for the reproducibility  of the current work e.g. the description of the architecture for the network in Section C.1 is not complete. What was the choice of activation function?\n\n3. The problems considered by the authors feature optimization over structured input spaces via latent space BO. Following on from point 2 above some of the details of VAE training are missing. Do the authors periodically retrain the VAE? As I understand the authors use the same initialization set as the BAO algorithm for database query planning? Why not use random initializations and report errorbars? For the peptide problem the authors mention they curate L=1000 sequences partitioning 100 of these to the validation set. Is the VAE trained on the 900 sequences comprising the training set?\n\n4. It would be worth adding a random search baseline across the VAE latent space as a sanity check.\n\n&nbsp;\n\n**__MINOR POINTS__**\n\n&nbsp;\n\n1. In terms of the opening statement, \"Multi-task optimization seeks to use related, previously solved tasks to accelerate the optimization of new ones.\" I would disagree with this definition. Related tasks do not necessarily need to be \"solved\" to provide benefit to the optimization of a new task.\n\n2. When citing domains in which multitask optimization problems occur, it would be useful to provide reference works for the domains mentioned.\n\n3. Line 64, the acronym MTBO for multi-task Bayesian optimization is introduced before it is defined.\n\n4. Line 64/64, the source papers for Optformer and LLAMBO should be given upon first mentioning the methods.\n\n5. Line 86, when introducing Bayesian optimisation it would be worth citing the originating papers for the methodology [1,2] as discussed in [3] in place of the references given.\n\n6. Line 89, the notation $y$ should be defined as a noise-corrupted version of $f(\\mathbf{x})$.\n\n7. The source paper for the VAE [4] should be cited when introducing it on line 96.\n\n8. On line 97 when introducing the idea of latent space Bayesian optimization, the source paper should be cited [5], as well as close follow-up works on the topic [6, 7] which were published before Eissman et al.\n\n9. Line 124, the citation to Leis et al. should be parenthetical e.g. (Leis et al. 2015) as opposed to narrative since the author's name is not part of the sentence. It would be worth correcting this across the manuscript.\n\n10. When describing structured input spaces it would be worth explaining what the authors mean by \"structured\" as this may not be apparent to layreaders. Something akin to a \"non-numerical, discrete input\" may be appropriate together with examples such as images, molecules, or amino acid sequences.\n\n11. Line 148, the acronym for LLMs has already been defined earlier in the manuscript.\n\n12. In Algorithm 1, the notation $X_t^*$, $X_{init}$, $y$ etc. should be defined. $y_{init}$ should be denoted as a vector. Additionally, $y$ and $y_{next}$ should also be denoted as vectors for generality in the batch setting. On line 177, $x$ should also be a vector. $\\alpha$ should not be a subscript on the argmax.\n\n13. In Algorithm 2, it would be better not to use the variable $T$ for the number of iterations since it overloads the use of $T$ as the number of tasks. Additionally, $x$ and $y$ should be bolded as vector quantities $\\mathbf{x}$ and $\\mathbf{y}$.\n\n14. Line 198, $x$ should be a vector.\n\n15. There is a missing full stop at the end of Equation 2.\n\n16. There are missing capitalizations in the references e.g. \"bayesian\" in place of \"Bayesian\".\n\n17. There is a missing arXiv reference for Eggensperger et al. 2020.\n\n18. There are missing conference references e.g. Haluptzok et al. was published at ICLR 2023.\n\n19. It would help the reader if the authors described roughly what the DKT and FSBO methods were instead of forcing the reader to read the source papers.\n\n20. In the related work on language models as optimizers it would be worth discussing the relation of the current work to [8].\n\n21. The source paper for AdamW [9] should be cited given that it is used.\n\n&nbsp;\n\n**__REFERENCES__**\n\n&nbsp;\n\n[1] H.J. Kushner (1962). [A Versatile Stochastic Model of a Function of Unknown and Time Varying Form. Journal of Mathematical Analysis and Applications](https://www.sciencedirect.com/science/article/pii/0022247X62900112) 5(1):150–167.\n\n[2] H.J. Kushner (1964). [A New Method of Locating the Maximum Point of an Arbitrary Multipeak Curve in the Presence of Noise.](https://asmedigitalcollection.asme.org/fluidsengineering/article-abstract/86/1/97/392213/A-New-Method-of-Locating-the-Maximum-Point-of-an?redirectedFrom=fulltext) Journal of Basic Engineering 86(1):97–106.\n\n[3] Garnett, R., [Bayesian optimization](https://bayesoptbook.com/). Cambridge University Press. 2023.\n\n[4] Kingma and Welling, [Auto-encoding Variational Bayes](https://openreview.net/forum?id=33X9fd2-9FyZd&source=post_page---------------------------), ICLR 2014.\n\n[5] Gómez-Bombarelli, R., Wei, J.N., Duvenaud, D., Hernández-Lobato, J.M., Sánchez-Lengeling, B., Sheberla, D., Aguilera-Iparraguirre, J., Hirzel, T.D., Adams, R.P. and Aspuru-Guzik, A., 2018. [Automatic chemical design using a data-driven continuous representation of molecules](https://pubs.acs.org/doi/full/10.1021/acscentsci.7b00572). ACS Central Science, 4(2), pp.268-276.\n\n[6] Griffiths, R.R. and Hernández-Lobato, J.M., 2020. [Constrained Bayesian optimization for automatic chemical design using variational autoencoders](https://pubs.rsc.org/en/content/articlehtml/2019/sc/c9sc04026a). Chemical Science, 11(2), pp.577-586.\n\n[7] Kusner, M.J., Paige, B. and Hernández-Lobato, J.M., [Grammar variational autoencoder](https://proceedings.mlr.press/v70/kusner17a.html?ref=https://). ICML 2017. PMLR.\n\n[8] Ranković, B. and Schwaller, P., 2025. [GOLLuM: Gaussian Process Optimized LLMs--Reframing LLM Finetuning through Bayesian Optimization](https://arxiv.org/abs/2504.06265). arXiv preprint arXiv:2504.06265.\n\n[9] Loshchilov and Hutter, [Decoupled Weight Decay Regularization](https://openreview.net/forum?id=Bkg6RiCqY7), ICLR 2019.\n\n&nbsp;", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "AYBUUFLcda", "reviewer": "Reviewer_PwEA", "summary": "While this article claims to consider multi-task Bayesian optimisation, really it's about transfer learning within a class of Bayesian optimisation problems. The latter is really important, and the proposed solution is promising. The idea is to tune an LLM to map from task features to likely high performing parameter selections for that task, and to use these suggestions to initialise Bayesian optimisation when a new task comes along. The idea is good, and the results are interesting.", "questions": "Is my hypothesis correct on the consequence of initialising only with high value parameter values? If not, why not? Would it be better if we also transferred hyper parameters, or some other way of conveying the baseline performance level?\n\nGiven that we can find good initial points using the LLM, and apparently have access to a large number of samples, would local search not be likely to perform better? I'm not convinced that BO is the right approach once we have a well-fitting LLM.\n\nHow dependent is this on the particular BO method you have used? The BOSS algorithm (Moss et al, NeurIPS 2020) is explicitly designed for optimising over string spaces and would be a useful comparator. Is it that the *constrained* version of LOL-BO is really important (see my previous points). In which case, more should be made of it - perhaps an ablation where we use an unconstrained BO to demonstrate the difference.\n\nMinor quibbles:\n- I think you have a typo in line 177. You are not taking the argmax over alpha values. It should probably read argmax_x alpha(x,GP)?\n- I disagree with your analogy to self-play in RL in line 243. In RL self-play we bring in new data from the environment when the algorithm plays itself. In self-instruction, and in what you do here, there is no further extrinsic information introduces at all.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "In the problem sets considered, the LLM is able to learn the strongly performing parameters from early runs of Bayesian optimisation. It learns so well that the initial suggestions for later tasks are often better than the final discoveries by single task BO on those tasks. However further refinement by BO provides even better solutions.\n\nThe avoidance of any need to learn a cross-task performance model makes multi-task BO much more feasible than in the traditional settings we consider.", "weaknesses": "The paper does not fully explore why this works. My intuition is that if you provide a few really strong examples to initialise BO, the acquisition function should then take the performance at these points as a baseline and go and search in the areas where it does not yet have any examples (ie far from the initialisation). \n\nThe number of BO reps is high. We are initialising with 50 observations and the x axis in Fig 1 goes to 2000 (so far as my eyes can make out). This is a lot of samples.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)