Adaptive Acquisition Selection for Bayesian Optimization with Large Language Models
Authors: Giang Ngo
Dat Phan Trong
Dang Nguyen
Sunil Gupta
Svetha Venkatesh
Venue: ICLR 2026 Poster (2026)
Keywords: Bayesian Optimization
Large Language Models
Primary area: optimization

Abstract: Bayesian Optimization critically depends on the choice of acquisition function, but no single strategy is universally optimal; the best choice is non-stationary and problem-dependent. 
Existing adaptive portfolio methods often base their decisions on past function values while ignoring richer information like remaining budget or surrogate model characteristics. 
To address this, we introduce LMABO, a novel framework that casts a pre-trained Large Language Model (LLM) as a zero-shot, online strategist for the BO process. 
At each iteration, LMABO uses a structured state representation to prompt the LLM to select the most suitable acquisition function from a diverse portfolio.
In an evaluation across 50 benchmark problems, LMABO demonstrates a significant performance improvement over strong static, adaptive portfolio, and other LLM-based baselines. 
We show that the LLM's behavior is a comprehensive strategy that adapts to real-time progress, proving its advantage stems from its ability to process and synthesize the complete optimization state into an effective, adaptive policy.

Published as a conference paper at ICLR 2026
ADAPTIVE ACQUISITION SELECTION FOR BAYESIAN OPTIMIZATION WITH LARGE LANGUAGE MODELS
Giang Ngo; Dat Phan-Trong, Dang Nguyen, Sunil Gupta & Svetha Venkatesh Applied Artificial Intelligence Initiative, Deakin University Victoria, Australia {g ngo, d. rongtrong, d. nguyen, sunil. gupta, svetha. ge
ABSTRACT
Bayesian Optimization critically depends on the choice of acquisition function, but no single strategy is universally optimal; the best choice is non-stationary and problem-dependent. Existing adaptive portfolio methods often base their deci- sions on past function values while ignoring richer information like remaining budget or surrogate model characteristics. To address this, we introduce LMABO, a novel framework that casts a pre-trained Large Language Model (LLM) as a zero-shot, online strategist for the BO process. At each iteration, LMABO uses a structured state representation to prompt the LLM to select the most suitable ac- quisition function from diverse portfolio. problems, IMABO demonstrates significant performance improvement over strong static, adaptive portfolio, and other LLM-based baselines. We show that the LLM's behavior is comprehensive strategy that adapts to real-time progress, proving its advantage stems from its ability to process and synthesize the complete optimization state into an effective, adaptive policy.
an evaluation across 50 benchmark
INTRODUCTION
Bayesian Optimization (BO) is a preeminent methodology for the global optimization of expensive- to-evaluate, black-box functions, pervasive challenge across science and engineering (Shahriari jtt al. 2015). Its framework uses a surrogate model (often a Gaussian Process (GP)) to approxi- mate the objective function and acquisition function (AF) to intelligently select the next point to evaluate, balancing the trade-off between exploration and exploitation. A core challenge the selection of the AF. It is well-established that no single, fixed AF offers optimal performance across all problems (Hoffman et al.] 201T); T); the best strategy is highly contingent on the character- istics of the objective function and can even change dynamically throughout single optimization run. This has spurred the development of adaptive strategies that move beyond static AF selection to dynamically choose different AFs from a portfolio every iteration (Hoffman al. 2011).
BO is
Existing adaptive portfolio methods, however, suffer from a critical limitation: their decisions are guided almost exclusively by narrow view of the optimization process, typically relying on the history of observed function values. Portfolio-based strategies (Hoffman et al. 2011 Vasconce- los et al.) 2019) 2022) operate on a reward signal derived from the past performance of each AF, utilizing only the function evaluations and the surrogate model's output. These methods largely ignore a wealth of other critical state information, such as the remaining optimization budget, the distance between evaluated points, and insights about the function's 's complexity encoded in the sur- rogate model's own hyperparameters (e.g., GP lengthscales). The core challenge is that designing a principled, algorithmic approach that effectively reason over such diverse and complex set of strategic, tactical, and landscape-related information has proven immensely difficult.
This paper bridges this gap by leveraging Large Language Model (LLM) a dynamic optimization strategist. We are motivated by the fact that modern I.I.Ms, trained on immense corpora of scientific literature and code, possess rich, nuanced, and implicitly encoded understanding of optimization principles. Instead of hand-crafting a complex policy covering all state information, we tap into this
ccrrespoondigg author

---

Published as a conference paper at ICLR 2026
pre-trained knowledge, as well as the reasoning capabiliities of the LLM to guide the exploration- exploitation balance in an optimization process. We introduce a new formulation of adaptive BO: casting acquisition function selection trained LLM,, supported by a novel state serialization design. At each iteration, the complete, multi- faceted optimization state is serialized into structured textual prompt. The LLM then analyzes this rich state and select the most appropriate AF from diverse portfolio for the immediate next step.
an in-contett decision-making problem solved by pre-
Experiments show LMABO consistently outperforms the baselines across diverse optimization problems. Ablation studies confirm the LLM leverages all components of the state summary, with performance dropping when any part is removed. A behavior analysis reveals comprehensive strat- egy: LMABO prefers exploration when progress has stagnated, heavily exploits with low remaining budget, and aggresively switches between AFs; during early stages of optimization, the L.MM sensitive to all information; during middle stages, the performance history and process status are influential and towards the end, sensitive only incumbent values. This proves that LMABO's suc- cess stems from its ability to reason over complete set of strategic and tactical information and to successfully mirror the established best practices in BO similar to the intuition of a human expert.
The contributions of this work can be summarized as follows:
We recast BO's acquisition function selection in-context decision decision-making task with an LLM as a closed-Loop strategist to select the most appropriate AF.
We propose a structured representation of the BO state, shown via ablations to be essential for effective zero-shot decision-make maaing..
On 50 benchmarks, LMABO outperforms static, adaptive, and LLM- baselines, with analysis revealing emergent, state-aware strategies beyond simple heuristics.
2 BACKGROUND
Bayesian Optimization is sample-efficient framework for optimizing expensive-to-evaluat black-box functions (Frazier 2018). Formally, let f x X' - R be an unknown objective function defined over X C R BO constructs probabilistic surrogate model, typically Gaussian Process (Rasmussen & Williams, 2005), to approximate f and quantify uncertainty in unexplored regions. At each iteration, given the set of previously observed evaluations D,-1, BO selects the next query point V4 E X' by maximizing an acquisition function c(x:; D.-1). which balances exploration (sam- pling in uncertain regions) and exploitation (sampling near low predicted values (for minimization)), There are many well-studied and effective AFs, including Expected Improvement (EI) (Mockus) 1998), Thompson Sampling
(TS) (Chowdhury & Gopalan| 2017), and Upper Confidence Bound (UCB) (Srinivas et al. 2010), and each of them has their own advantages and disadvantages. Some AFs like TS and UCB (given appropriate hyperparameters) are more explorative, while others like El are more exploitative. However, no single AF performs optimally across all problems (Hoffman) ct al. 201T), and achieving sample efficiency often requires dynamically prioritizing exploration or exploitation at the appropriate stages of the optimization process. Therefore, using the right AF with the right focus on exploration or exploitation at the right time can be a key to better BO performance. More details about AFs used in this paper and their grouping are provided in Appendix A
Gaussian Process is a nonparametric prior over functions, defined such that any finite collection of function values follows joint Gaussian distribution. A GP is fully specified by its mean function mxx) and covariance kemel k(r, x'), where the kernel encodes assumptions about the smoothness and structure of the underlying function. An example of a kernel the squared exponential kernel:
(I.
k(xxx') 0f exp
where E, are the lengthscales, controlling function variation along each input dimension, and oî is the outputscale, determining the overall variance of the function values. A small lengthscale means the function varies rapidly with input changes, and small outputscale keeps predictions close the mean with little variation. These hyperparameters are typically learned by maximizing the marginal likelihood of observed data, enabling GPs adapt their complexity the underlying objective.
2

---

Published as a conference paper at ICLR 2026
3 RELATED WORKS
Portfolio-Basee Strategies These methods treated AF selection as a portfolio allocation problem. GP-Hedge (Hoffman et al. 2011) framed the task as multi-armed bandit problem, selecting AFs by first computing a reward signal derived from their past performance then randomly sampling an AF according to a probability distribution weighted by these rewards. Subsequent methods like No-PASt-BO (Vasconcelos et al.) (2019) and SETUP-BO (Vasconcelos et al.) (2022) sought to im- prove upon this by introducing memory factors to discount distant evaluations, but they still rely on the same fundamental reward mechanism. Shahhiirii et al. (2014) used an information-theoreet approach with Entropy Search Portfolio (ESP), which shifted the selection criterion to forward- looking measure of utility: the expected reduction in uncertainty about the location of the global optimum. However, a key limitation unites these strategies: their decisions are guided by a narrow view of the optimization state, focusing primarily on function values and uncertainty while ignoring other critical information like the remaining budget or characteristics of the surrogate model itself.
Learning-Base Based Strategies MetaBO (Volpp al. 2020) and FSAF (Hsich et al. 202 meta- learn a state-dependent policy, formalizing AF selection as a reinforcement learning problem. These are related but not applicable to our setting, as they are designed for transfer learning scenario, where a policy is learned on a distribution of source tasks and adapted to new target task.
LLM-Based Bayesian Optimization With the recent success of LLMs, efforts have emerged to incorporate them into the BO process, though their functional roles differ fundamentally from approach. FunBO (Aglietti et al. 2025) uses the LLM as an offline algorithm generator to discover new AFs with few function evaluations, but it does not participate in the adaptive process of AF selection. Other recent methods have integrated LLMs more directly into the optimization loop as component replacements. For instance, LLMP (Requeima al., 2024) focuses on enhancing the surrogate model with natural language priors, while LLAMBO (Liu et al. 2024) uses the LLM for multiple steps the optimization process including generating initial samples, surrogate modeling, and proposing candidate points. These approaches rely on the L.MM's inductive biases regarding nu- merical function shapes, effectively treating the text-based model as a numerical regression engine. LMABO establishes distinet paradigm: the semantic controller. Unlike FunBO, LMABO oper- ates online to adapt to real-time feedback. Unlike LLAMBO/LLMP, it does not replace the rigorous mathematical backbone (Gaussian Processes) with LLM predictions. Instead, it recasts the AF se- lection itself as a sequential pocision-making task solvable by in-context reasoning, representing a new paradigm for adaptive BO.
4 METHODOLOGY
4.1 LANGUAGE MODEL- ADAPTIVE BAYESIAN OPTIMIZATION (LMABO)
Our proposed method LMABO uses a pre-trained LLM to dynamically select the most appropriate acquisition function at each iteration of the BO process. The framework operates as a closed-loop system, where the LLM is prompted at each iteration with a rich representation of the optimization state to infer the most effective AF for evaluation. The entire process is detailed in Algorithm [
4.2 THE LLM AS A ZERO-SHOT STRATEGIST
A core tenet of our LMABO framework is to leverage the reasoning capabilities of a large, pre- trained L.M in zero-shot setting. This approach requires no task-specific fine-tuning to the LLM's weights Instead, the model's strategic behavior is guided entirely through in-context learning, con- ditioned on an initial prompt, Po. that structures the entire rrcision-making task. The initial prompt, Po. is static instruction set that establishes the context for the entire optimization run. It is com- posed of several key components designed to elicit an expert-like decision-makkin process:
1. Role-playing Instruction: The prompt begins by providing the LLM with an instruction to act as an "expert in Bayesian Optimization". This contextual framing is used to condition the model, leveraging the patterns it learned during pre- pre-training to emulate the decision- making process that human expert might follow when presented with similar data.
3

---

Published as a conference paper at ICLR 2026
Algorithm The LMABO Framework
Require:
Objective function f(x); Initial dataset Do {xx, yo), .......... }: Optimization budget T; ..........
Portfolio acquisition functions A = {a1, ......., Or }; Large Language Model W.
1: Construct initial prompt Po that defines the L.LM's i role BO expert and provides the set of available acquisition functions A.
b See Appendix C
2: Send Po to V/ establish context.
See Sec. 4.2
3: for = 1, ..,,...,7 'I' do
4: Fit a Gaussian Process model (-1
5: Generate state summary St from GP and optimization history.
D See Sec.
4.3
6: Construct the update prompt P. from St.
7:
Obtain the next acquísition function
((P)).
See Sec.
4.2
8: Optimize or to find the next point to evaluate: di +- arg max, (s.()).
9: Evaluate the true objective function: y/1 f(z)) + n with noise 7)--
10: Di + D U (Ze.
11: end for
12: return The point x" corresponding to the best function value in Dr.
2. Available Actions:
We explicitly define the portfolio available acquisition functions, A. Each function is listed with its abbreviation (e.g., .. EI, UCB) and full name. We refrain from giving a description for each AF to avoid biased interpretations and instead rely on the LLM's encoded knowledge. Note that default UCB if the LLM's output is invalid.
3. State Information Schema: The prompt describes the structure of the state summaries,
S, that it will receive each subsequent step, explaining what each piece of information (c.g., GP lengthscales, current performance) represents.
4. Output Formatting Constraint: Finally, the prompt specifies a strict output format ("Ac- Justification") to ensure responses can be reliably parsed.
quisition abbreviation:
Po is sent once at the beginning of the optimization date prompt P. is formed by appending the new state summary, S, (see the next section), to the established context of Po. The full text for Po is provided Appendix [o reproducibility.
the stage. At each iteration I, the up-
4.3 OPTIMIZATION STATE REPRESENTATION
A key component of our LMABO framework is the translation of the nigh-dimensional, numerical state of the BO process into concise, heaanlre textual summary, S.. This summary is de- signed to provide the LLM with a comprehensive, multi-faceted view of the optimization landscape and progress. The state summary S; at each iteration composed of the following elements:
To contextualize the current step within the overall process, we provide the number of evaluations performed so far (N), the remaining budget (Nrem), and the problem dimensionality (D). The remaining budget, in particular, is critical for balancing the long-term need for exploration against the short-term need for exploitation.
Process Status:
Performance History: To provide context the optimization's progress, include sev- eral performance indicators. These are the incumbent objective value (fmin), the observed function value range, and the shortest distance from the last evaluated point any pre- vious point (as an indicator of the last evaluation's 's exploration tendency).
GP Model Characteristics: To inform the LLM about the current understanding of the function landscape, we provide key hyperparameters from the fitted surrogate model GP.-1- This includes the kernel's outputscale and summary statisties of the lengthscales (minimum, maximum, mean, and standard deviation).
These components are formatted into a structured string that becomes the core the prompt P. sent to the LLM at each iteration. See examples of these state summaries in Appendix C The design of S, balances compactness with completeness, enabling the LLM to leverage domain--pecifit signals (budget, GP hyperparameters, exploration metrics) without requiring training. Our ablation results (Table 2) demonstrate that omitting any these elements significantly degrades performance, un- derscoring the importance of the representation.
4

---

Published as a conference paper at ICLR 2026
5 EXPERIMENTS
5.1
EXPERIMENT SETUP
Baselines We employ a comprehensive set of baselines spanning four categories:
Static Acquisition Functions: These are standard and popular BO methods that use a single, fixed AF throughout the process. We include all 12 AF's that constitute the portfolio from which LMABO can select (see Appendix for details).
These methods use simple, non-adaptive strategies to select among multiple AFs. They first include the naive strategy of uniformly random selec- tion within a portfolio of: 1) all 12 AFs, 2) most popular AFs in practice (EI, TS, UCB, PosMean), and 3) AFs that commonly selected by LMABO (i.e. EI, LogEI, and TS, as shown later in Figure 2a). We also include strategies that alternate between exploration (TS) and exploitation (EI) (i.e. Alt-EI-TS-k with 1,3, 5) and a strategy that explores first then exploits (i.e. TwoPhases-TS-ED,
Simple Meta-strategies:
Adaptive Acquisition Functions: These methods adapt their AF based on the optimization state, including GP-Hedge (Hoffman et al.) 201T). No-PASt-BO /Vasconcelos al. 2019). SETUP-BO (Vasconcelos et al. 2022), and ESP (Shahriari et al. 20144).
LLM-based Methods: State-of-the-an baselines that incorporate LLMs into the BO pro- cess, including LLAMBO (Liu et al. 2024) and LLMP (Requeima et al. 2024).
The evaluation is performed on a broad set of 50 problems to test for robustness and general applicability. These includes 30 synthetic benchmark functions from the COCO platform (Hansen et al., 2021] and the BoTorch library (Balandat et al., 2020). In addition, we use 20 real-world hyperparameter optimization problems from Bayesmark (Uber) (2020). This benchmark evaluates the practical applicability of LMABO on common and important task in machine learning. Details of these benchmark problems are provided in Appendix B
Benchmark Problems
Implementation Details We implement LMABO with Gemini-2.5 Flash. For surrogate modeling of GP-based methods, we use a GP with Matérn 5/2 kernel, and the implementations are built using standard modules from the BoTorch library. Each optimization run is initialized with 2D + points, where is the dimensionality of the problem. The optimization budget is set to 50 itera- tions for problems with fewer than 10 dimensions and 100 iterations for problems with 10 more dimensions. More implementation details are provided in Appendix
Evaluation Each experiment is repeated 10 times with different random seeds. For each method on each problem, averaged the Areas Under the Simple Regret Curves (AUCs) over all 10 repetitions. We then compute Relative Performance (RP): for each problem, the method with the lowest (best) total AUC receives RP of 1.0, and all other methods are assigned an RP equal to their total AUC divided by the best total AUC for that problem. This ensures aggregation across problems not affected by different absolute scales of AUCs. A rank of each method on each problem is determined by sorting all methods by their total AUC (lower is better), assigning rank to the best. Note that the ranking includes 25 baselines. 8 ablation variants of LMABO, and 4 adaptive portfolio variants, resulting in a maximum rank of 38. RP and rank help provide a clear and concise summary of comparative performance across all methods and problems instead of plotting all regret curves for 38 methods on 50 problems. Experimental results will undergo rigorous statistical analysis to ensure the validity of our findings. Details about the statistical tests are provided in Appendix B33
5.2 MAIN RESULTS
The results, summarized in Table [ demonstrate that LMABO achieves a substantial performance improvement over all baseline categories. Averaged across problems, LMABO's total AUC is 9.7% lower than the best static AF, 14.8% lower than the best simple meta-strategies, 54.7% % lower than the best LLM-based method, and 16.6% lower than the best adaptive portfolio method; conse- quently, LMABO ranks among the top four methods on average. LMABO's s low variation (CV = 0.37) indicates high consistency across different seeds. A 50-iteration with LMABO uses total
5

---

Published as a conference paper at ICLR 2026
Table 1: Overall performance comparison of LMABO against all baselines across 50 opti- mization problems. P-values from Friedman tests in the last row indicate statistically significant differences among methods for both RP and rank. The third and fifth columns show p-values of post- hoc pairwise comparisons between LMABO and each method, which confirm that the differences performance between LMABO and methods are significant. Exploitative AF's are marked in for details).
blue and explorative AFs are marked in magenta (see Appendix A
Mean RP
P-value (RP)
Mean Rank
P-value (Rank)
CV of
Method
(Interquartile Range)
(Min Max)
AUC
Static Acquisition Functions
1.53 (1.08-1.53)
3.71e-03
15.80 (1-36)
1.54e-04
0.45
LogPI
1.40 (1.10-1.47)
7.40c-03
14.52 (1-37)
3.89e-04
0,47
EI
1.34 (1.11-1.52)
2.85c-04
13.08 (1-34)
3.87e-05
0,44
LogEI
1.36 (1.15-1.48)
8.24e-06
13.92 (1-35)
1.49e-06
0,42
PosMean
1.42 (1.08-1.51)
1.05c-02
14.70 (1-37)
5.06e-04
0,45
PosSTD
7.02 (2.12-5.5))
3.37e-08
34.78 (3-38)
3.47e-08
0.48
UCB
1.75 (1.23-2.02)
2.59e-07
23.94 (1-37)
1.32e-07
0.37
TS
2.07 (1.32-1.92)
1.10e-07
25.46 (1-37)
7.05e-08
0.35
KG
1.66 (1.24-1.78)
7.19e-08
23.96 (4-36)
5.09e-08
0.40
PES
2.93 (1.68-3.24)
2.80c-08
31.92 (10-38)
2.74e-08
0.38
MES
2.80 (1.22-1.66)
5.16e-07
20.54 (1-37)
2.04e-07
0.40
JES
1.62 (1.30-1.70)
4.30c-08
24.64 (1-36)
4.79c-08
0.39
Simple Meta-strategies
Random (Full portfolio)
1.43 (1.23-1.54)
5.16e-07
17.36 (2-31)
1.91e-07
0.41
Random
1.45 (1.19-1.55)
1.82c-07
17.62 (1-34)
2.07e-07
0.42
(EI, TS, UCB, PosMean)
Random (EI, LogEl, TS)
1.42 (1.18-1.54)
7.91e-06
15.62 (1-33)
3.60e-06
0.40
Al--EI-TS-1
1.50 (1.27-1.54)
7.94c-08
18.72 (4-35)
1.38c-07
0.40
Al--EI-TS-3
1.62 (1.28-1.56)
1.27e-07
21.50 (2-33)
6.32e-08
0.38
Alt-EI-TS-5
1.58 (1.28-1.58)
7.78c-08
21.16 (3-35)
1.10c-07
0.39
TwoPhases-TS-EI
1.86 (1.30-1.63)
6.62e-08
24.06 (2-35)
6.32e-08
0.37
LLM-based Methods
LLAMBO
2.67 (1.14-2.65)
1.55e-04
23.74 (1-38)
1.49e-06
0.43
LLMP
2.78 (1.59-2.49)
3.70c-08
31.30 (1-38)
3.52c-08
0.33
Adaptive Portfolio Methods
GP-Hedge
1.45 (1.22-1.52)
1.82e-07
16.96 (1-34)
1.65e-07
0.42
No-PASt-BO
1.53 (1.20-1.74)
1.27c-07
19.08 (1-37)
1.65e-07
0.37
SETUP-BO
1.56 (1.21-1.62)
1.11e-07
19.06 (1-37)
8.91e-08
0.42
ESP
1.62 (1.29-1.67)
4.30e-08
22.98 (1-38)
5.53e-08
0.42
LMABO
1.21 (1.06-1.25)
5.62 (1-19)
0.37
P-values of Friedman Tests
1.380e-106
1.380e-106
of about 6000 tokens (z:$0.01); both this expense and the LLM call latency of roughly 1 second per iteration are negligible relative to the cost of evaluating expensive black-box functions (which often takes minutes or hours per evaluation) and are justified by the resulting performance gains.
Static AFs are inherently unreliable. While strong heuristics like El or LogEl are among the best in this class with relatively low RPs and ranks, their performance is brittle; some problems, their rank drops to as low as 35th, highlighting the risk of fixed strategy. In addition, adaptive portfolio methods, though achieving competitive results, frequently exhibit higher variability and are less robust across the heterogeneous problem suite. These approaches depend on heuristics for weighting past acquisition performance and thus be slow to adapt when the tpek-spec land-

---

Published as a conference paper at ICLR 2026
Table Ablation study on the components LMABO. We analyze the contribution of LMABO's key components by comparing the full model to multiple ablated versions. L.MABO-8B300 uses open-source LLMs (Qwen3-8B and Qwen3-30B-A33 -o---iin- 2507 (Team) 2025)). LMABO- 120B uses the open-weight model gpt-oss- 120b (OpenAl (025). The Mean RP and Mean Rank are calculated using the same global ranking of all baseline and ablation methods as Table
Thinking-
Mean RP.
P-value (RP)
Mean Rank
P-value (Rank)
CV of
Method
(Interquartile Range)
(Min Max)
AUC
LMABO without:
Remaining budget
1.40 (1.17-1.54)
4.25e-07
15.72 (3-34)
2.61e-07
0.39
GP model characteristics
1.50 (1.21-1.59)
4.30e-08
20.04 (4-34)
3.47e-08
0.40
Shortest distance information
1.50 (1.23-1.65)
1.62e-07
19.76 (1-33)
7.60e-08
0.39
Instruction avoid ineffective AFs
1.92 (1.45-1.91)
3.07c-08
28.30 (3-37)
3.02e-08
0.43
LMABÓ using other L.LMs
LMABO-8B
1.48 (1.24-1.62)
2.48c-07
18.94 (1-34)
1.72e-07
0.38
LMABO-30B
1.29 (1.15-1.35)
3.21e-04
10.70 (2-25)
6.28e-05
0.39
LMABO-120B
1.22 (1.07-1.24)
1.00++00
6.64 (1-31)
3.99c-01
0.39
LMABO (GPT-4o mini)
1.21 (1.11-1.26)
1.00c+00
7.16 (1-21)
1.84e-01
0.37
LMABO
1.21 (1.06-1.25)
5.62 (1-19)
0.37
scape changes or when surrogate uncertainty dynamics differ across problems. Given the same AF portfolio, L.MABO mitigates this shortcoming by considering other important factors including the process status, performance history and surrogate model characteristics, which enables faster, more consistent adaptation and yields significantly better average performance. LLM-based methods, while occasionally ranking among the top performers (c.g., LLAMBO is in the top three 12 out of 50 problems), generally exhibit inconsistent results and often fall behind other approaches. This highlights that simply incorporating exploitation trade-off is crucial for robust BO. LMABO addresses this by explicitly framing AF se- lection siciisonnmaking task in which the LLM can make informed, context-aware decisions to balance the trade-off effectively.
LLM is not sufficient; effective navigation of the exploration-
Similar to the aforementioned baselines, simple meta-strategies also struggle maintain consistent performance across diverse problems. Figures[2lannd 2nd show that LMABO uses EI, LogEl, and TS more frequently than other AFs and often switches between these three options. However, simple meta-strategies that mimic these behaviors (c.g., random selection between the three or alternating between EI and TS) fail deliver robust performance. Therefore, these results demonstrate that LMABO's behavior cannot be reduced to simple heuristic.
5.3 ABLATION STUDIES
Our ablation studies confirm that each component of the LMABO framework is crucial for achiev- ing its superior performance, with statistically significant degradation in performance compared to the full model once component is removed, shown in Table 2 However, these studies also demonstrate the robustness of our core framework, as the ablated versions still achieve respectable results, often performing on par with or better than many established baselines.
Firstly, LMABO's performance is inherently coupled with the underlying LLM. With small open- source model in our IMABO-SB experiment, we observed a noticeable performance drop, though it still achieves competitive results. LMABO-30B, using larger model with improved thinking capabilitics, recovers much of this drop. approaching that of the full LMABO with Gemini-2.5 Flash and outperforming all baselines. Both LMABO- 120B and LMABO with GPT-4o mini achieve performance comparable to the vanila version, demonstrating that LMABO's 's effectiveness is not tied to specific LLM, but rather benefits from the general reasoning capabilities of strong LLMs.
Removing the remaining budget leads to the smallest performance drop, which means that this information is the least critical of the three ablated inputs. GP model characteristics and shortest.

---

Published as a conference paper at ICLR 2026
Holderfoce
Shetel
not. Dre
For, clibete Decseriee
-22
-25
-00
-25
-2.9
-200
-20
Figure 1: Impact of task-specific context LMABO performance. Results are averaged over 10 runs with standard deviation shown as shaded regions.
distance information are equivalently impactful on the effectiveness of the LMABO. The large drop in LMABO-AB4 demonstrates that instructing the LLM avoid AFs that failed to improve the incumbent is essential. Without this guidance. we find that the LLM repeatedly selects ineffective AFs, leading to significantly worse optimization performance.
In addition to these ablation studies, we attempted to inject task-specific context into the initial prompt Po on 4 benchmark problems, such as characteristies of the synthetic functions or hyperpa- rameter optimization tasks. The hypothesis is that by exploiting prior knowledge about the problem domain, the LLM could make more informed AF selections, potentially enhancing the optimization performance. Details about these contexts provided in Appendix C.I. The results, shown in Figure [ suggest that providing context acts as a safeguard against stagnation in local optima, a benefit observed across both synthetic and real-world landscapes. Vanila LMABO may stall at sub-optimal plateau after the initial progress (e.g. between iterations 10 and 30 on HolderTable); however, being warned of "many local minima", the context-aware variant successfully identified this trap and bypassed it early, converging to the global minimum significantly faster. Based on these findings,
recommend providing a textual description of the objective function whenever such knowledge is available. In hyperparameter optimization scenarios involving well-known algo- rithms (e.g. Decision Tree or AdaBoost), this semantic warmstarting can significantly reduce the computational budget required to reach competitive performance (which was also reported in Liu al.)) (024) albeit for point initialization).
6 ANALYSIS
To understand LMABO's strategy, we performed an in-depth analysis of its AF selection behavior, which reveals a multi-faceted strategy with clear preferences, following distinct temporal patterns, and, most importantly, adapting its behavior in response to the real-time optimization state. Our findings provide strong evidence that LMABO effectively synthesizes the state information to execute a dynamic, context-aware policy that mirrors established practices in BO. Note that this section's results are aggregated across all repetitions on all problems from experiments in Section 5
Overall Preferences In Figure 2al we observe that LMABO exhibits clear preference for certain AFs (e.g. EI, LogEl, and TS). EI's usage often increases slightly at the beginning as a response to early improvements, while TS's usage decreases gradually near the end as the need for exploration diminishes. PosMean is heavily used near the end for LMABO as a last effort to find improvements. Another surprising observation is the high usage of MES and PES in the first few iterations, but this seems to align with the initial goal of quickly reducing uncertainty about the global optimum. How- ever, the strong performance of LMABO also involves other factors, as discussed later. On other adaptive portfolio methods, there is clear preference for any AF, and the selection is more uni- formly distributed. The only exception is No-PASt-BO with an increasing preference for PosMean as the budget runs out. From this figure, first insight is that LMABO's 's success partly stems from a well-calibrated preference for strong AFs, rather than a uniform exploration of all options, as well dynamic adjustment of these preferences over time.
On a separate note, the adaptive portfolio baselines can be at a disadvantage without knowing the strengths of El and LogEl. We conducted an additional experiment where these methods operate on
8

---

Published as a conference paper at ICLR 2026
LIADO
Nodga
PKG:00
SCT.7.0
100%
Pelteer
(a) AF selection frequeney the optimization process
LM480
setured
(b) AF switch frequency
IMADO
G2-tedge
No-PG--50
SETUP-B3
ESP
Agge
DG
7.00
11:4
943
441
6-4
7.30
8:p
2-J4
8-3
1:00
1641
135
176
19:03
13:38
PSS
3.0
1.39
822
263
721
2.31
5.00
(c) AF selection frequeney by Improvemen//stagnatioon status. "Stagnation" is defined as improvement in the incumbent, and "Improvement" is defined as any change in the incumbent.
Figure 2: LMABO's acquisition function selection behaviors. Note that these behaviors are aggre- gated across all runs on all problems.
only curated subset comprising of EI, LogEl, and TS the three most frequently selected AFs by LMABO. We indeed observe a performance improvement for these methods except for GP-Hedge (detailed results in Appendix but the curated versions are still strictly outperformed by LMABO.
notable difference is observed between LMABO and the adaptive portfolio baselines in terms of AF switching. LMABO the only method perform high number of switches between the group of explorative AFs (i.e. the first five AFs) and exploitative AFs (i.e. the remaining seven AFs) throughout the opti- mization process, seen in the bottom left and top right of the figures. The dynamic adjustment mentioned earlier is more evident here, as LMABO frequently switches between exploration and exploitation. Figure 2c shows increased usage of exploitative AFs during improvement phases for all methods, which aligns with the goal of refining the search around promising areas. However, combined with the findings from Figure 2b the adaptive portfolio baselines seem to demonstrate passive, one-directional learning by using more exploitative functions after success, but their sparse switching patterns reveal a "sticky" policy that is slow to abandon a strategy, even when failing. This is likely because of their reliance past successes guide future choices, which lead to overcommitment to a single AF. In contrast, LMABO employs an active, bi-directional strategy, not only learning exploit improvement but also decisively switching back to exploratory functions
Switching the AF in Response to the Optimization State In Figure 2b

---

Published as a conference paper at ICLR 2026
to escape stagnation. This demonstrates that LMABO's core advantage lies not just in identifying a good heuristic, but in its superior, more agile policy for knowing when that heuristic is no longer effective and different approach is required.
Linking AF Selection to Justification To verify consistent link between the LLM's AF choices and its justifications, we an- alyzed keywords appeared in the justifications across different AFs. In Figure 3 explorative AFs strongly associated with terms like "exploration" or "stagnation", and exploitative AFs are strongly associated with terms like "exploitation" or "improve- ment". This confirms that the LLM's selections are consistent with its justification, and that it is possible to use these justfica- tion to understand the decision of the I.I.M
Information Sensitivity Analysis To assess the sensitivity of LMABO to the information provided the prompt, we conducted a sensitivity analysis by perturbing the values fed to the LLM in the state representation across early, middle and late stages. Due to space constraints, we summarize the key findings here (detailed results are in Appendix G). During the early stage, the LLM is highly reactive to all information changes, including the process status, performance history, and GP model characteristics. During the middle stage, performance history and process status most influential. In the late stage, the LLM is less sensitive to perturbations and is mostly changing its decision in response to new incumbent values where it exhibits strong preference for exploitative AFs. While perturbing the values presents inherent limitations (e.g. the perturbed values may be unrealistic or inconsistent with other state variables), the results still provide valuable insights into the LLM's decision-making process. Overall, we find that LMABO is highly sensitive to tactical variables like performance history with evident signs of stagnation/mmprooeeeett and process status, while correctly showing robustness to changes in secondary GP parameters that do not alter the overall strategic context. This demonstrates that the LLM is synthesizing the state summary to weigh the relative importance of different inputs, a key feature of its effective, state-aware policy.
Figure 3: Word frequency in justifications. Red box (bottom left) the explorative group. Blue box (top right) is the ex- ploitative group.
7 CONCLUSION
We introduced LMABO, novel framework that successfully utilizes pre-trained LLM zero- shot, online strategist for selecting acquisition functions in Bayesian Optimization. By prompting the L.L with a comprehensive, real-time summary of the optimization state, LMABO dynamically controls the opploration-cxploitteee that this approach significantly outperforms strong static, adaptive, and other Ablation studies and analysis of the LLM's behavior confirm its success stems from a well-rounded state-aware strategy that adapts well the optimization's progress, demonstrating patterns that align closely with established BO best practices.
trade-off. Our extensive experiments on 50 benchmarks show
baselines.
This paper focused on standard BO with Gaussian Processes, and future work could adapt LMABO to other surrogate models and optimization settings. For instance, in constrained BO, the LLM could be leveraged to dynamically balance objective improvement against constraint satisfaction. Overall, LMABO opens new avenues for integrating LLMs into adaptive optimization frameworks, leveraging their reasoning capabilities for prcisionn-mking in complex tasks.
REPRODUCIBILITY STATEMENT
Implementation of LMABO and its variants along with the baselines and scripts to reproduce the http:p/:s/gg..ti.b.oooa/aaggg/// wtwp:///g...oiooiinooooooeee maboo
experiments can be found htttttss///gthhhhee
10

---

Published as a conference paper at ICLR 2026
LLM USAGE
Large Language Models were used for correcting grammar and improving writing clarity along with updating the related works. We used Gemini Pro and ChatGPT for these purposes.
REFERENCES
Virginia Aglietti, Ira Ktena, Jessica Schrouff, Eleni Sgouritsa, Francisco Ruiz, Alan Malek, Alexis Bellot, and Silvia Chiappa. FunBO: Discovering acquisition functions for bayesian optimization with funsearch. In Forty-second International Conference on Machine Learning. 2025.
Sebastian Ament, Samuel Daulton, David Eriksson, Maximilian Balandat, and Eytan Bakshy. Un- expected improvements to expected improvement for bayesian optimization. Advances in Neural Information Processing Systems, 36:20577-20612, 2023.
Maximilian Balandat, Brian Karrer, Daniel R. Jiang, Samuel Daulton, Benjamin Letham, An- drew Gordon Wilson, and Eytan Bakshy. BoTorch: A Framework for Efficient Monte-Carlo Bayesian Optimization. In Advances in Neural Information Processing Systems 33, 2020. URL
http::/arxiv..rr org/ /etp/9910.0000
Sayak Ray Chowdhury and Aditya Gopalan. On kernelized multi-armed bandits. In Proceedings of the 34th International Conference on Machine Learning-Volume Volume 70, pp. 844 853, 2017.
Peter Frazier. A tutorial bayesian optimization. arXiv preprint arXiv: 080711181711111 2018.
Nikolaus Hansen, Anne Auger, Raymond Ros, Olaf Mersmann, Tea Tušar, and Dimo Brockhoff. Coco: A platform for comparing continuous optimizers a black- box setting. Optimization Methods and Software, (16(1):114-11) 2021.
José M Hernández-Lobato Matthew W Hoffman, and Zoubin Ghahramani. Predictive entropy search for efficient global optimization of black-box functions. Advances in neural information processing systems, 27, 2014.
Matthew Hoffman, Eric Brochu, and Nando de Freitas. Portfolio allocation for bayesian optimiza- tion. In Proceedings of the Twenty-Seventh Conference on Uncertainty in Artificial Intelligence, pp. 327-336, 2011.
Bing-Jing Hsich, Ping-Chun Chug Hsich, and Xi Liu. Reinforced few-shot acquisition function learning for bayesian optimization. Advances in Neural Information Processing Systems, 34:7718-7731 2021.
Carl Hvarfner, Erik O Hellsten, and Luigi Nardi. Vanilla bayesian optimization performs great in high dimensions. In Proceedings of the 41st International Conference on Machine Learning, pp. 20793-20817, 2024.
Harold J. Kushner. A new method of locating the maximum point of an arbitrary multipeak curve in the presence of noise. Journal of Basic Engineering, 86:97-106, 1964.
Tennison Liu, Nicolás Astorga, Nabeel Seedat, and Mihaela van der Schaar. Large language models to enhance bayesian optimization. In The Twelfth International Conference Learning Repre- sentations, 2024. URL. htt.:htttt//ppeeeeeeeeee
net./ wtt.:eeteeeooomoooddooooooooooo
J. Mockus. The application of bayesian methods for seeking the extremum, 1998.
OpenAl. 12t- & gpt-oss-20b model card, 2025, URL https:: ptttps://arxiv.xr. wwaxii.org//bs.. 2508. 10925
C.E. Rasmussen and C.K.I. Williams. Gaussian Processes for Machine Learning. Adaptive Com- putation and Machine Learning series. MIT Press, 2005. ISBN 9780262182539
James Requeima, John Bronskill, Dami Choi, Richard Turner, and David K Duvenaud. Lim pro- cesses: Numerical predictive distributions conditioned on natural language. Advances in Neural Information Processing Systems, 37: 10976, 2024.

---

Published as a conference paper at ICLR 2026
Bobak Shahriari, Ziyu Wang, Matthew W Hoffman, Alexandre Bouchard-Côte and Nando de Fre- itas. An entropy search portfolio for bayesian optimization. arXiv preprint arXiv:1406.4625, 2014.
Bobak Shahriari, Kevin Swersky, Ziyu Wang, Ryan P Adams, and Nando De Freitas. Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE, 104(1); 148-175, 2015.
Niranjan Srinivas, Andreas Krause, Sham Kakade, and Matthias Seeger. Gaussian process optimiza- tion in the bandit setting: no regret and experimental design. In Proceedings of the 27th Interna- fional Conference on International Conference Machine Learning, pp. 1015-1022, 2010.
Qwen Team. Qwen3 technical report, 2025. httpl/tctss///aav..oorrr////////
09388
Ben Tu, Axel Gandy, Nikolas Kantas, and Behrang Shafei. Joint entropy search for multi-objective bayesian optimization. Advances in Neural Information Processing Systems, 35:9922-9992 9938, 2022.
Uber. Uber/bayesmark: Benchmark framework to easily compare bayesian optimization methods on real machine learning tasks. httpss//gitthie com www.///ww/wweeeaee......aaa/e 2020. Online; accessed 2025-09-09.
Thiago de P Vasconcelos, Daniel ARMA de Souza, César LC Mattos, and João PP Gomes. No- past-bo: Normalized portfolio allocation strategy for bayesian optimization. 2019 IEEE 31st International Conference on Tools with Artificial Intelligence (ICTAI), pp. 561-568. IEEE, 2019.
Thiago de P Vasconcelos, Daniel Augusto RMA de Souza, Gustavo C de M Virgolino, Cesar LC Mattos, and Joao PP Gomes. Self-tuning portfolio-based bayesian optimization. Expert Systems with Applications, 188: 15847, 2022.
Michael Volpp, Lukas P. Fröhlich, Kirsten Fischer, Andreas Doerr, Stefan Falkner, Frank Hutter, and Christian Daniel. Meta- acquisition functions for transfer learning in bayesian opti- mization. In ICLR, 2020.
Zi Wang and Stefanie Jegelka. Max-value entropy search for efficient bayesian optimization. In Proceedings of the 34th International Conference Machine Learning Learning-Volume Volume 70, pp. 3627- 3635, 2017.
Jian Wu and Peter Frazier. The parallel knowledge gradient method for batch bayesian optimiza- tion. In Proceedings of the 30th International Conference Neural Information Processing Systems, pp. 3134-3142, 2016.
A LIST OF ACQUISITION FUNCTIONS
The acquisition function a<;; D) guides the selection of query points in BO by quantifying the utility of evaluating f at x. By maximizing axx; D) over the search space, we identify points that balance exploration (sampling where uncertainty is high) and exploitation (sampling where the surrogate predicts low function values). Details of the AFs used in our experiments are follows:
Probability of Improvement (PI) Kushner 1964
((x) (σx) (
(Pp((:)
where t is the standard normal CDF, j((r) and (xx are the posterior mean and standard deviation, and T is target (e.g., the incumbent solution). PI selects points with high probability of improving T.
Expected Improvement (EI) (Mockus) 1998):
1(x) a (xx)
aEr (m(x) r) ((2) (()),
where is the standard normal PDF El measures the expected magnitude of improvement.
12

---

Published as a conference paper at ICLR 2026
Log Probability of Improvement (LogPl) (Balandat al. 2020) and Log Expected Im- provement (LogED) (Ament et al. 2 These are numerically stable variants and El, respectively, that operate in the log domain to handle vanishing values and gradients.
Upper Confidence Bound (UCB) (riiniaas et al. 2010):
QUCB (z) μ(x) + (o)
where & > 0 controls the exploration weight. UCB explicitly balances exploitation (mean) and exploration (uncertainty).
Thompson Sampling (TS) (Chowdhury & Gopalan 2017): Draws a sample f ~ p(f D)
and selects r maximizing fz). TS provides randomized exploration strategy consistent with the posterior.
Posterior Mean (PosMean) and Posterior Standard Deviation (PosStd): Using Omean (2) u(z) yields pure exploitation, while Oigdd (z) = (xx) performs pure exploration.
Knowledge Gradient (KG) (Wu & Frazier, 2016): This look-ahead function quantifies the expected increase in the maximum value of the function that results from collecting new observation at candidate point x.
Information-theorriicc AFs:
Predictive Entropy Search (PES) [Hernáneez-Lobiee Lobato et al. 2014): Selects points that maximize the expected reduction in entropy of the distribution over the location of the global optimum.
-Max-value Entropy Search (MES) WWang & Jegelka 2017): Focuses reducing un- certainty about the maximum function value rather than location.
- -oint Entropy Search (JES) (Tu et al. 2022): A recent AF that generalizes PES and MES by considering the joint entropy of both the location and value of the optimum.
We use the implementations of these AFs from the BoTorch library (Balandat et 2020 in our experiments. For KG, PES, MES, and JES, we use their available batch implementations with batch size of 1. We consider PosSTD, UCB, TS, KG, PES, MES, and JES to be the exploratory category, while the exploitative AFs include PosMean, PI, LogPI, EI, and LogEl.
B EXPERIMENTAL DETAILS
B. LIST OF BENCHMARKS
We evaluate LMABO on a diverse suite of 50 benchmark problems, including synthetic functions and real-world hyperparameter optimization tasks. The synthetic benchmarks isted in Table with 15 functions from the COCO suite (Hansen et al. [02T]) and 15 from BoTorch (Balandat et al. 2020 For hyperparameter optimization tasks, we follow the practice of Liu al. (2024) and use datasets and ML models available on Bayesmark (Uber (020) to form 20 dataset-model pairs. The datasets include Breast, Digits, Wines and Diabetes, and the ML. models include Decision Tree, Random Forest, SVM, AdaBoost, and MLPSGD. The dimensionality of these search spaces ranges from We use the empirical optimum found by all methods (as the true optima are unknown for many problems) to compute the simple regret each iteration. See Appendix D.1 of Liu et al. (2024) for full details about the hyperparameter optimization tasks.
B.2 IMPLEMENTATION DETAILS
We follow the standard practice optimize the GP hyperparameters by maximizing the log marginal likelihood at each iteration. We use the default implementation of GP regression in Bo'orrch (Ba- landat et al., 2020) which has hyperparameter priors from (Hvarfner et al. (024) to enhance perfor- mance on high-dimensional tasks. All acquisition functions are optimized using multi-start LBFGS- B, except for TS and PES. For both TS and PES, we use discrete optimization over a finite set of randomly sampled candidates due to the high computational cost of evaluating the acquisition func- tions. This practice of AF optimization applies to running experiments with all static acquisition functions, all adaptive portfolio baselines, and LMABO.
13

---

Published as a conference paper at ICLR 2026
Table COCO and BoTorch synthetic benchmark functions used in our experiments.
COCO benchmarks (15)
BoTorch synthetie benchmarks (15)
Name
Dimensionality
Name
Dimensionality
BucheRastrigin
5
Ackley
50
LinearSlope
5
Beale
2
AttractiveSector
5
Bukin
2
StepEllipsoid
5
Cosine8
8
Discus
5
DixonPrice
15
BentCigar
5
DropWave
2
SharpRidge
5
EggHolder
2
DifferentPowers
5
Griewank
9
Weierstrass
5
Hartmann
6
SchaffersllICond
5
HolderTable
2
CompositeGriewattRReebbce
10
Levy
13
Gallagher21
Michalewiez
10
Gallagher101
5
StyblinskiTang
21
Katsuura
5
Shekel
4
LunacekBiRastrn
5
SixHumpCamel
2
For all experiments, the LLMs were queried with a temperature of 0.0. Again, for invalid or failed LLM responses, we fall back UCB. The fallback rate is extremely small only about 0.1 1.11% of all queries (i.e. out of 10,000 queries, only 11 of them do not follow the "Acquistion abbrevia- tion: justfication" format). This indicates that LLM failures are rare and have minimal impact on overall optimization performance. All adaptive portfolio baselines, such as GP-Hedge, were imple- mented using their standard configurations as described in their respective publications, except that the portfolio of acquisition functions was expanded to include all 12 AFs listed in Appendix
B.3 STATISTICAL TESTS
Results in Tables 1 2 and d 6 follow a standardized statistical analysis to ensure the significance of results. This statistical analysis is conducted separately both mean RPs and ranks (across 10 repetitions) of 38 methods on 50 benchmarks. Firstly, assess the significance of performance differences between the methods, we first apply the Friedman test (a non-parametric test for data that is normally distributed) Friedman test that all methods perform equally, while the alternative hypothesis is that at least one method performs differently. the Friedman test indicates significant differences, we follow up with post-hoc pairwise comparisons using the Wilcoxon signed-rank test with Holm-Bonferroni correction control for multiple comparisons, specifically comparing each baseline method against LMABO. The null hypothesis for each pairwise comparison is that there is no difference in perfor- mance between LMABO and baseline, while the alternative hypothesis that the performances of LMABO and the baseline differ. We set significance level of 0.05 for all statistical tests.
the matrix of mean RPs (or ranks). The null hypothesis of the
C PROMPTS
Figure shows the complete, unabridged initial prompt (P,) used to instruct the LLM in our ex- periments. This prompt was developed through a very brief iterative process of 6-7 trials. The refinments were not aimed at tuning for performance accurate formatting of the LLM's responses and to encourage a full consideration of all information in the optimization state representation. Examples of follow up prompts during the optimization process are shown in Table For KG. PES, MES, and JES, we denote them by gKG, qPES, qMES, and qJES, respectively, in the initial prompt align with the naming conventions in BoTorch (Ba- landat et al. 2020). In constructing the initial prompt, we observed the following phenomena during preliminary experiments that informed the final design:
specific benchmark, but rather to ensure
14

---

Published as a conference paper at ICLR 2026
Table 4: Additional contexts provided the LLM verify the benefit of prior knowledge on LMABO.
Problem
Context
HolderTable
This is optimize black-box synthetic function defined on 2- dimensional bounded domain. The landscape is highly complex and non-convex, featuring a wavy pattern with many local minima. My prior analysis strongly suggests that the global minimum is not unique. This is to optimize a black-box synthetic function defined on 4- dimensional bounded domain. The landscape should be mostly flat, but it is punctuated by a small number of sharp, narrow, and deep hollows at some locations. The main challenge is not finding the gen- eral region of these minima, but precisely pinpointing the 'needle in haystack' global minimum at the very bottom of one of these steep basins.
Shekel
hpt digits DreisionTree
This is to optimize a 6-dimensional space for a Decision Tree classi- fier on the Digits dataset, where the objective is maximize validation accuracy (by minimizing the negated values). The landscape is rela- tively smooth but features several ridges and valleys due the com- plex interactions between parameters like max max_deppp _depth, max_ features, and dain_samplessppiit The main challenge lies balancing these pa- rameters to avoid overfitting while still capturing the underlying pat- terns in the data. My analysis indicates that certain combinations of these parameters can lead to similar accuracy levels, suggesting mul- tiple optimal regions in the parameter space.
Seeiabbtes_Deissioo
This is optimize 6-dimensional space for Decision Tree regres- sor on the Diabetes dataset, where the objective is to minimize mean squared error. The landscape is relatively smooth but features several ridges and valleys due to the complex interactions between parameters like _dephh, max_features, and min wwmplessspppiiiiiiceeeeeeee.ssse The main chal- lenge lies in balancing these parameters to avoid overfitting while still capturing the underlying patterns the data. My analysis indicates that certain combinations of these parameters can lead to similar error levels, suggesting multiple optimal regions in the parameter space.
A full history of past function values and AF choices was not necessary for each input prompt, as the I.L.M seems capable of inferring relevant optimization history from previous input prompts.
Specific examples of AF choices (e-g. "UCB: brief justification") in the prompt could bias the LLM towards particular choice, replaced them with placeholders. Before this change, we found that Qwen3-8B often mimicked the exemplar choices if specific AFs were mentioned as examples, and only stopped doing when the examples replaced with placeholders.
The LLM sometimes ignored certain details, such as the number of remaining iterations, added assurance in the prompt consider all provided context.
C.1 ADDITIONAL CONTEXTS
Table contains the task-specific contexts provided to the LLM to test how such prior knowledge about the problem would affect LMABO's performance. These contexts were added to the initial prompt Po between lines 3 and 5 in Figure
D RUNTIME COMPARISON
Table 5 shows the average runtime, in minutes, for 50 iterations of the methods across all tested benchmarks. LMABO only incurs moderate overhead compared static AFs due to the LLM
15

---

Published as a conference paper at ICLR 2026
query at each iteration, but it is significantly faster than adaptive portfolio methods that require optimizing multiple AFs at each iteration. Since the curated versions of adaptive portfolio methods only optimize AFs instead of 12, they much faster than their full versions while achieving better performance (as shown Table (6), suggesting that a smaller, well-chosen portfolio can be beneficial for these methods. Among L.LM-base methods, LLAMBO is comprable in speed to LMABO, while LLMP is much slower due to inferring with an open-source model locally. This higher runtime is also observed on LMABO-SB/30/11005 the ablated versions of LMABO that use an open-source model.
Table 5: Average runtime for 50 iterations of all methods across all benchmarks (in minutes).
Method
Runtime
Method
Runtime
PosSTD
2.20
GP-Hedge-Curann
12.01
PosMean
2.06
No-PASt-BO
1 113.67
2.62
No-PASt-BO-Curated
14.94
LogPI
2.09
SETUP-BO
104.42
EI
2.14
SETUP-BO-Curated
7.24
LogEI
2.15
ESP
50.61
UCB
2.07
ESP-Curated
6.01
TS
4.62
LMABO (with Gemini 2.5 Flash/GPP--40 mini)
7.42/7.36
KG
12.88
IMABO-ABI
6.69
PES
8.93
LMABO-AB2
8.17
MES
3.73
LMABO-AB3
7.79
JES
5.45
LMABO-AB4
6.46
LLAMBO
9.21
LMABO-8B
19.12
LLMP
29.35
LMABO-30B
14.87
GP-Hedge
109.18
LMABO-120B
15.61
E CURATED SET FOR ADAPTIVE PORTFOLIO BASELINES
In this experiment, the adaptive portfolio baselines (i.e. GP-Hedge. No-PASt-BO, SETUP-BO. and ESP) are restricted to a curated subset of acquisition functions: EI, LogEI, and TS. The curated methods also participated in the calculation of relative performance, rank, and statistical tests men- tioned in Section 5) No-PASt-BO, SETUP-BO, and ESP show a performance improvement when using the curated set, while GP-Hedge shows degradation.
LLM RESPONSE EXAMPLES
F. RESPONSES TO THE INITIAL PROMPT
Table shows the responses of Gemini-2.5 Flash the initial prompt across some different opti- mization problems. The responses confirm the LLM's understanding of the task and readiness to proceed with the optimization process.
F.2 RESPONSES DURING OPTIMIZATION
Table& example responses from Gemini-2.5 Flash at different stages of an optimization run of the Weierstrass function. As seen on the table, the LLM adapts its acquisition function choices based on provided context, including the number of remaining iterations, the current best objective value, the model's lengthscales, and the shortest distance between points. Contrary to the popular choice TS in early iterations other cases, the first response opts for El when observes wide range of function values. The second and fourth responses both select LogEl after new best value is found, showing the LLM's ability to recognize when an improvement-fooce appropriate. Although both El and LogEl are suitable in these contexts (as well as other exploitative AFs like PI), the LLM's s choice LogEl is influenced by the wide range of function values and the modest gain compared to the range (which were from -6.576 -9. 9.980 for the second response
acquisition is
16

---

Published as a conference paper at ICLR 2026
Table 6: Comparing adaptive portfolio methods between using large portfolio (of 12 AFs) and curated portfolio (of 3 AFs). The curated methods are denoted with a "-Curated" suffix.
Mean RP
P-value (RP)
Mean Rank
P-value (Rank)
CV of
Method
(Interquartile Range)
(Min Max)
(AUC)
GP-Hedge
1.422 (1.206-1.517)
1.239e-07
12.42 (1-25)
1.112e-07
0.418
GP-Hedge-Curated
1,487 (1.233-1.670)
3.126e-08
15.06 (2-28)
5.444e-08
0.372
No-PASt-BO
1,505 (1.186-1.722)
9.572e-08
14.06 (1-28)
1.112e-07
0.370
Ce-PASttBO-Curatte
1.469 (1.218-1:579)
0.656e-08
14.38 (1-24)
7.235e-08
0,409
SETUP-BO
1,536 (1.155-1.688)
8.609e-08
13.88 (1-28)
6.129e-08
0.420
SETUP-BO-Curated
1.503 (1.243-1.556)
3.126e-08
14.76 (1-26)
4.781e-08
0.374
ESP
1.601 (1.273-1.674)
3.126e-08
16.82 (1-29)
3.742e-08
0.419
ESP-Curated
1.506 (1.240-1.537)
9.572c-08
15.06 (2-26)
1.112e-07
0.422
Table Responses to the initial prompt Po-
Problem
Response
Cosine8
Yes, I understand the context. I am ready to receive the first sum- mary the Bayesian Optimization process.
Weierstrass
ready receive the summary of the Bayesian Optimization process and recommend the next acquisition function. I will strictly follow the specified output format. Yes, understand the task and the available acquisition functions. am ready to receive the first summary of the Bayesian Optimization process.
Okay, understand.
bLLWWDn_MLPSGD
and from -10.148 to 11.300 for the fourth response). This is helpful when the values of El may become very small, as LogEl more numerically stable. The third response chooses TS to escape stagnation phase in response to failed improvements and over-exploration signs (demonstrated by the smaller shortest distance). Finally, with only one itcration left, the LLM selects El maximize the chance of final improvement.
G INFORMATION SENSITIVITY ANALYSIS
and show the results of our information sensitivity analysis at early, middle, and late stages of optimization, respectively. For these experiments, we perturb each element of the state representation S, individually while keeping all other elements fixed their original values iteration 7' in an optimization run. Input prompts from previous iterations were not modified, so the LLM's memory of the optimization history remains intact.
Tables
While perturbing the values may present some noises in the results, we still observe some clear trends across all four tables. Firstly, information about the process status (e.g. number of points evaluated, remaining budget) and the performance history (e.g. incumbent objective value, function value range, shortest distance) have significant impact the LLM's acquisition function choices. Reducing the remaining budget or having new incumbent objective value tends to shift the LLM's preference towards exploitative AFs. However, in Tables and IO we observe that a very small improvement in the incumbent objective value (e.g. from 1.24 to 1.244 in Table | and from 9.98 to -9.982 in Table TT) can lead to a switch back an explorative AF. This suggests that the LLM is sensitive the magnitude of improvement relative the overall function value range. Secondly, information about the model state (e.g. lengthscales, outputscale) appears to have a more subtle influence the LLM's choices. While perturbing these elements does lead some changes in the selected AFs, the changes are less consistent and pronounced compared to the other state elements. This indicates that the LLM may prioritize information about the optimization progress and performance over the surrogate model's 's internal parameters when making decisions.
17

---

Published as a conference paper at ICLR 2026
You are an expert Bayesian Optimization, specifically tasked with recommending the most suitable acquisition function for the next iteration to minimize an objective function.
For context, Matern 5/2 kernel with ARD.
a Gaussian Process as the surrogate model with
will previde you with summary of the Bayesian Optimization process at each step. This summary will include the following information:
**N*** The total number of points evaluated so far. *Remmaining iterations:** The number of iterations left in the optimizaticn process.
**b:.* The dinensionality of the search space (namber of input parameters).
* rangeska The range of the objective function values observed so far
**f --__min:** The current best (lowest) observed objective value.
**Shortet distance:** The shertest distance from the last point to any other point, indicating whether is exploiting too much.
12
* lengthscales:** These are crucial hyperparameters of the
Gaussian Process model's kernel.
They describe how the model perceives the smoothness and relevance of each input dimension the objective function.
You will receive their range (min/max), mean, and standard deviation. **Model outputscale:** It detines the overall magnitude or amplitude of the functicn's variation.
15
Available acquisition functions you can choose from, with brief descripticns of their primary uses:
18 1. PI (Brobability of Improvement)
2. LogPI (Log Probability of Improvement)
3. EI (Expected Improvement)
21 4. LogEI (Log Expected Improvement)
22 5. UCB (Upper Confidence Bound)
23 6. FosMean (Posterior Mean):
7. PosSTD (Posterior Standard Deviation)
8. TS (Thompson Sampling)
26
9. qKG (Knowledge Gradient)
27
10. dPES (Predictive Entropy Search)
11. qMES (Max-valus Entropy Search)
12. QJES (Joint Entropy Search)
31
At each step:
32
+ Review the provided summary of the optimization process and consider the current state of the ogtimization.
**Select the acquisition functicn that you believe will be best for the opt imization process.
34
* *vvoic reusing acquisition functions that failed to improve the db objective function in previous iterations.**
When responding, select the acquisition function you deem appropriate.
37
Your justi fication should briefly explain why that function is suitable given the provided optimization summary, referencing relevant aspects like explorattion/explitteeee characteristics.
balance, remaining iterations, model
38 The response must strictly follow the format "Acquisition abbreviation: justification", similar to these examples:
"AF_ABBREVIATION: Your justification for choosing this specific function."
"xxx: brief reason explaining why хxx is the optimal choice now.
41
Firstly, just give brief confirmation that you understand the task and the available acquisition functicns.
Listing I: The unabridged initial prompt (Ph) provided to the LLM.
18

---

Published as a conference paper at ICLR 2026
Input Prompt
Response
Current optimization state:
Given the early stage (N=I1, iterations remaining) and the wide range of observed function values and vary- ing lengthscales, there is still significant uncertainty and po- tential for discovering better regions. El provides robust balance hetween exploration (considering areas of high certainty) and exploitation (seeking expected improvement over the current best, f.min). making an excellent choice for building better global understanding of the objective function while still prioritizing improvement.
N: 11
Remaining iterations: 50
D: 5
range: Range [-5.982, 83.765), Mean
26.466 (Std Dev 35.075)
- f_min: -5.982
Shortest distance: 0,455
Lengthscales: Range [0.261, 59.707),
Mean 27.581 (Std Dev 24.557)
Outputscale: 0.898
Current optimization state:
LogÉl: Thompson Sampling successfully discovered a new f_min. With this new best ohservation and 40 iterations re- maining, LogEl is strong choice. It is effective for finding further significant improvements, especially when dealing with potentially wide range objective values (as indi- cated by frange) or when the current best is challenging to beat. LogEl provides a more robust measure of improve- ment compared to standard EI, by focusing on the log of the improvement, which can help navigating the complex landscape suggested by the varying lengthscales and ensure continued progress.
- N: 21
Remaining iterations: 40
D::
range: Range [-9.980, 106.046]. Mean
23.896 (Std Dev 34.093)
[mmi: -9.980
Shortest distance: 0.427
Lengthscales: Range [0.513, 399.250),
Mcan 154.676 (Std Dev 188.785)
Outputscale: 1.212
Current optimization state:
TS: Both JES and PES failed to improve f_min. The length- scales have drastically shortened and hecome concentrated, indicating the model believes the function is very local- ized, yet this hasn't yielded improvements. The shortest distance is also very small (0.20), suggesting potential over- exploitation. With 21 iterations remaining, Thompson Sam- pling is the best choice break this persistent stagnation and localized search. By sampling a new function from the posterior, TS can explore regions that the current model might be overlooking or prematurely discarding, offering fresh, diverse exploration strategy find new global op- tima
- N: 40
Remaining iterations: 21
5
f_range: Range [-10.148, 132.117), Mean
15.999 (Std Dev 33.416)
f_min: -10.148
Shortest distance: 0.204
Lengthscales: Range [0.449, 3,865), Mean
1.786 (Std Dev 1.159)
Ouiputscale: 1.618
Current optimization state:
LogEl: El successfully improved f_min! With a new best value found and only 11 iterations remaining, it's crucial to build on this success with improvement sertrveeeet--fuusedddd acquisi- tion function. LogEl is excellent choice continues to seek expected improvement, but logarithmic scaling makes it particularly robust to the magnitude of objective function values. This allows it to efficiently pinpoint fur- ther significant improvements new f_min, ensuring we maximize the chances of find- ing an even better optimum in these final, critical iterations.
N: 50
Remaining iterations: 11
5
f.range: Range [-11.302, 132.117), Mean 12.749 (Std Dev 30.799)
- f_min: -11.302
Shortest distance: 0.346
refine the search around
Lengthscales: Range [0.494, 2.198), Mean
1.532 (Std Dev 0.61
- Outputscale: 2.069
Current optimization state:
El: With only iteration remaining, the primary goal is to make the absolute best attempt improve the current f min. El (Expected Improvement) is the robust and direct improvement- focused acquisition function. directly opti- mizes for the expected gain over the current best observed value, offering the most pragmatic strategy to find a final, better optimum, especially since it demonstrated success in finding the most recent f_min just few iterations ago.
N: 60
Remaining iterations: 1
- frrange: Range [-12.135, 132.117), Mean
11 11.070 (Std Dev 29,387)
f-min: 135.135
Shortest distance: 0.503
Lengthscales: Range [0.548, 2.468), Mean
1.615 (Std Dev 0.653)
Outputscale: 2.821
Table 8: Example responses from the LLM during an optimization run of the Weierstrass function. The left column includes the input prompts fed to the LLM. Each response includes the LLM's selected acquisition function along with its justification.
19

---

Published as a conference paper at ICLR 2026
Table 9: Information Sensitivity Analysis for Early Stage (7 5). We perturb the values of each element in the state representation St and observe the resulting changes in the output AF chosen by the LLM. The highlighted values indicate the original state element values at T in Griewank optimization run. We show explorative AFs in blue and exploitative AFs in magenta for better tracking of AF changes between the two groups. The original imput prompt and response are shown at the bottom of the table.
State element
Output AF given element value
Value AF
5
9
20
40
50
500
points evaluated
PES
LogEI
LogEI
PosSTD
PosSTD
PES
Value
1
10
40
46
50
100
Remaining budget
AF
PosMean
LogEl
PES
LogEl
PES
PES
Value
-100
-10
0
1.24
1.244
Incumbent objective value
AF
LogEl
EI
EI
KG
LogEl
Value
2
100
190
194.081
200
1000
Maximum function value
AF
LogEl
LogEl
UCB
LogEl
LogEl
PES
Value
2
20
50
56.326
60
100
Mean of function value
AF
PosSTD
UCB
LogEl
LogEI
LogEl
LogEl
Value
1
10
60
63.145
70
100
Std Dev function value
AF
JES
PES
PES
LogEl
PES
PES
Value
0.01
0.05
0.06
0.07
0.1
0.5
Shortest distance
AF
PES
LogEl
LogEl
LogEl
LogEl
MES
Value
0.01
0.1
0.2
0.231
0.3
0.45
Minimum of lengthscales
AF
LogEl
PES
LogEl
LogEl
LogEl
PES
Value
0.25
0.4
0.452
0.5
1.0
10.0
Maximum of lengthscales
AF
LogEl
EI
LogEl
PES
LogEl
EI
Value
0.24
0.3
0.342
0.4
0.45
Mean of lengthscales
AF
MES
LogEl
LogEl
LogEl
LogEl
Value
0.01
0.1
0.110
0.2
1.0
10.0
Std Dev of lengthscales
AF
PES
PES
LogEl
LogEl
LogEl
JES
Value
0.01
0.1
0.8
0.865
0.9
10.0
Outputscale
AF
LogEl
LogEl
LogEl
LogEl
El
KG
Original input prompt
Original response
Current optimization state:
LogEl: 'f min' has stalled for two iterations and the shortest distance is again very low (0.060), indicating po- tential over-exploitation or getting stuck in a flat region. LogEl is suitable here because it is less sensitive the absolute magnitude of improvement and focuses on rel- ative improvements, which helps escape local minima or flat regions where small absolute improvements are still valuable, while also maintaining strong exploration component given the many remaining iterations. We also need to avoid reusing EI which was used 2 iterations ago and did not improve min.
N: 9
Remaining iterations: 46
-D: 2
rrange: Range [1.244, 194.081),
Mean 56.326 (Std Dev 63.145)
f f.min: 1.244
SShortest distance: 0.060
Lengthscales: Range [0.231,
0.452), Mean 0.342 (Std Dev
0.1
Outputscale: 0.865
20

---

Published as a conference paper at ICLR 2026
Table 10: Information Sensitivity Analysis for Middle Stage (7 25). This table follows the same format as Table but for T' 25 in ComposieeGievannRosiibboro optimization run.
State element
Output AF given element value
Value
20
40
45
50
100
# points evaluated
AF
KG
TS
JES
LogPI
Value
20
26
30
100
Remaining budget
AF
PI
LogEl
KG
TS
TS
LogPI
Value
-1000
-200
-100
-92
-91.2
-91.176
Incumbent objective value
AF
PosMean
PI
LogPI
JES
JES
TS
Value
20
200
208.249
210
1000
Maximum function value
AF
JES
TS
KG
TS
JES
UCB
Value
-90
-50
400161
-40
0
200
Mean of function value
AF
PES
PES
TS
PES
TS
TS
Value
10
50
55.658
60
100
Std Dev of function value
AF
JES
TS
TS
TS
JES
KG
Value
0.01
0.1
0.7
0.714
0.8
1.0
Shortest distance
AF
UCB
TS
JES
TS
UCB
TS
Value
0.01
0.1
0.9
0.989
1.0
10.0
Minimum of lengthscales
AF
JES
JES
JES
TS
KG
KG
Value
30.0
100.0
200.0
208.249
210
1000.0
Maximum of lengthscales
AF
JES
PES
JES
TS
JES
TS
Value
1.0
10.0
20.0
29.547
30.0
200.0
Mean of lengthscales
AF
JES
JES
JES
TS
PES
JES
Value
0.1
10.0
60.0
62.256
70.0
200.0
Std Dev of lengthscales
AF
KG
TS
KG
TS
KG
JES
Value
0.1
3.0
3.731
4.0
10.0
100.0
Outputscale
AF
UCB
PES
TS
KG
JRS
PES
Original input prompt
Original response
TS: 'f_min' remains stagnant for four iterations, and "qMES' did not yield an improvement, despite the refined lengthseales. The lengthscales have slightly widened again, indicating some lingering uncertainty. With significant number of remaining iterations and persis- tent stagnation, "TS' (Thompson Sampling) is strong choice. It provides robust, probabilistic, and funda- mentally different exploration strategy by sampling from the GP posterior. This approach excellent for escap- ing local optima and discovering genuinely new, high- potential regions, especially when other more determin- istic informatioo-thheorttii methods have struggled to find a better 'f_min', and the model still has some uncer- tainty in its overall landscape understanding.
Current optimization state:
N: 45
Remaining iterations: 26
D: 10
f_range: Range [-91.176,
208.2491 Mean -40.161 (Std Dev 55.658)
-mmin: -91.176
Shortest distance: 0.714
Lengthscales: Range [0.989, 204.500), Mean 29.547 (Std Dev 62.256)
Outputscale: 3.731
21

---

Published as a conference paper at ICLR 2026
Table 11: Information Sensitivity Analysis for Late Stage 45) Part 1. This table follows the same format as Table 9 but for T = 45 in a hpt breast MLPSGD optimization run.
State element
Output AF given element value
Value
5
20
50
54
60
100
# points evaluated
AF
LogPI
JES
JES
JES
JES
JES
Value
5
10
50
100
Remaining budget
AF
KG
JES
JES
JES
JES
Value
-100
-10
-0.92
-0.917
-0.96
Incumbent objective value
AF
E1
EI
EI
LogEl
JES
Value
-0.7
-0.4
-0.360
-0.3
-0.1
0.0
Maximum function value
AF
JES
KG
JES
JES
JES
JES
Value
-0.9
-0.8
-0.738
-0.7
-0.4
Mean function value
AF
JES
JES
JES
KG
JES
Value
0.01
0.1
0.152
0.2
0.5
1.0
Std Dev function value
AF
JES
JES
JES
JES
JES
JES
Value
0.01
0.1
0.4
0.459
0.5
1.0
Shortest distance
AF
JES
JES
JES
JES
KG
JES
22

---

Published as a conference paper at ICLR 2026
Table 12: Information Sensitivity Analysis for Late Stage (T 45) Part 2. Continuation of Table
State element
Output AF given element value
Value
0.001
0.004
0.01
0.1
1.0
10.0
Minimum of lengthscales
AF
JES
JES
KG
JES
JES
JES
Value
1600
2000
4000
4234.465
5000
10000
Maximum of lengthscales
AF
JES
KG
JES
JES
JES
JES
Value
1.0
100.0
1500.0
1514.580 JES
1600
4000
Mean of lengthscales
AF
JES
JES
JES
JES
KG
Value
10.0
100.0
1600.0
1683.459
1700.0
4000.0
Std Dev of lengthscales
AF
JES
JES
JES
JES
JES
JES
Value
0.01
0.1
0.8
0.818
0.9
10.0
Outputscale
AF
JES
JES
JES
JES
JES
JES
Original input prompt
Original response
QIES: 'f_min" remains unchanged, meaning qMES did immediately lead to improvement. The length- scales are still extremely large and variable (Range [0.004, 4234.465], Mean 1514.580, Std Dev 1683,459), indicating persistent, profound uncertainty in the GP model's understanding of the objective function. With only iterations remaining (N=54), and 'f_min' still stag- nant despite various information-thhoree comprehensive information-gathenn to make the best final decision. qIES (Joint Entropy Search) is advanced acquisition function that directly aims to reduce the entropy the *joint* posterior distri- bution of the optimal value *and* its location. In these very late stages with high model uncertainty, a holistic understanding of both the value and location of the opti- mum is crucial for making the final, most informed de- cision. qJES provides a more complete information gain than qPES qMES alone, making it ideal for the lim- ited remaining budget to resolve uncertainty about the true optimum. (qJES was used at N=32, and its strength holistic uncertainty reduction makes it appropriate for this critical, late-stage, high-uncertainty scenario).
Current optimization state:
N: 54
Reeainiin iterations: 6
methods, strategy is needed
DD: 6
f_range: Range [-0.916, -0.360),
Mean -0.738 (Std Dev 0.152)
min: -0.916
-Shortest distance: 0.459
Lengthscales: Range [0.004,
4234.465), Mean 1514.580 (Std
Dev 1683.459)
Outputscale: 0.818
23


Reviews:
[{"review_id": "2BxGgwdUJN", "reviewer": "Reviewer_iyWi", "summary": "The paper studies meta–decision making in Bayesian Optimization. Specifically, how to select the acquisition function (AF) for a given task/budget—and proposes using a Large Language Model (LLM) as a source of prior knowledge to guide that choice. Concretely, the method extracts task context (e.g., problem description, dataset traits, search‑space information, etc) and queries an LLM to recommend an AF (e.g., EI, UCB). The authors evaluate this idea across a broad set of AutoML/HPO problems and report that making the right meta decision (AF selection) substantially affects BO performance.", "questions": "See above", "limitations": "", "rating": 4, "confidence": 4, "soundness": 2, "presentation": 1, "contribution": 2, "strengths": "The work tackles a meta design choice in BO/AutoML—which AF to use—rather than only tuning model hyperparameters. In practice this could translate in a more robust “out‑of‑the‑box” optimizer that requires less trial to find the optima. \n\nThe paper reports extensive experiments indicating that the meta decision is relevant. It is important to have flexibility in what AF to choose along the acquisition process.", "weaknesses": "Presentation. I think the presentation and how the paper is written is not good in general. To give some examples. 1) Some figures are hard to read or to use for significance judgments. In Figure 2, the variance of the result is  wide, making it difficult to visually assess the significance of the results. 2) Table 2 uses cryptic labels (“LMABO‑AB1…AB4”); these should be replaced or augmented with descriptive names.\nNovelty. The idea of leveraging LLM “knowledge” to steer BO is not brand‑new (e.g., LLAMBO, LLMP, FunBO; Sec. 3), although applying it specifically to the meta decision of AF selection is useful as the authors have shown empirically.\nAblation. The paper includes several ablation studies, which is valuable. However, this should also be examined at a more global level to demonstrate that the observed patterns consistently repeat across multiple runs and different experiments.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "A8c9kvCfhU", "reviewer": "Reviewer_HUN7", "summary": "This paper proposes LMABO, a zero‑shot strategy that delegates acquisition-function selection in Bayesian Optimization to Large Language Model. At each BO iteration, the model receives a state with: optimization state—process status (iteration count, remaining budget, dimensionality), performance history. This information is useful for the LLM to sample the best next acquisition function to be used.   Basically the method can be seen as an LLM doing a meta decision over an additional component of the acquisition process. The authors show strong result, which highlights the significance of the paper.", "questions": "See weaknesses", "limitations": "", "rating": 4, "confidence": 4, "soundness": 3, "presentation": 1, "contribution": 2, "strengths": "**Strong results.** Casting AF selection itself as an in‑context decision problem for an LLM shows that the meta decision of selecting the acquisition function is highly relevant for the whole acquisition process. The paper shows strong results.", "weaknesses": "**Novelty.** While the framing is neat, the algorithmic contribution largely reduces to a prompt + a state serialization + a portfolio choice. Many recent works have leveraged LLM inductive biases for decision selection or candidate generation in BO, so the conceptual step—“use an LLM to pick an AF given a textual state”—feels incremental without additional design elements.\n\n\n**What paper should have.** The paper positions an LLM as the decision-maker for selecting the acquisition function, with the main contribution lying in the experiments. Two questions should be addressed: (1) Can the LLM reliably choose an appropriate acquisition function at each point in time? and (2) Do the LLM’s choices align with—or diverge from—established expectations, given that its reasoning comes from in-context learning? The paper convincingly demonstrates (1). It attempts to address (2), but the presentation could be stronger: the Background section does not give readers enough context to evaluate why the LLM makes the choices it does.\n\n**Results.** The ablation studies are helpful but insufficient. The experimental setup around Figure 1 seems to be underspecified—it’s unclear whether the figure is illustrative or aggregates results across multiple runs. Results in Figure 2 show high variance, which makes it difficult to draw robust conclusions about the method’s behavior", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "ceB3GQOycA", "reviewer": "Reviewer_KYRT", "summary": "The paper introduces LMABO (LLM-Assisted Adaptive Bayesian Optimization), a novel framework that utilizes a pre-trained Large Language Model (LLM) as a zero-shot, online strategist to dynamically select the most suitable acquisition function (AF) at each iteration of the Bayesian Optimization (BO) process. LMABO addresses the limitation of existing adaptive methods by employing a structured state representation that translates the complete optimization context—including remaining budget, performance history, and surrogate model (GP) characteristics—into a text prompt for the LLM to reason over. Evaluated across 50 benchmark problems, LMABO demonstrated significant performance improvement over static, adaptive portfolio, and other LLM-based baselines (e.g., 55% lower total AUC than the best LLM-based method), confirming that its success is due to an emergent, robust, and state-aware policy for controlling the exploration-exploitation trade-off.", "questions": "* Could the authors provide a more detailed analysis of the economic costs associated with utilizing the LMABO framework?\n\n* The core, high-performing LMABO implementation relies predominantly on Gemini-2.5 Flash.   Given that ablation studies demonstrate a noticeable performance drop when using smaller open-source models, could the authors verify the framework's broad reliability and efficacy by testing its performance and execution speed with other competitive closed-source LLMs (beyond Gemini-2.5 Flash) or high-capability open-source models from different architectures?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "* The paper introduces a framework that successfully recasts the task of acquisition function (AF) selection in Bayesian Optimization (BO) as an in-context decision-making problem.   This approach leverages a pre-trained Large Language Model (LLM) to select the most appropriate AF at each optimization step based on its implicitly encoded knowledge of optimization principles.\n* A primary technical contribution is the design of a structured state representation that translates the complex, multi-faceted numerical state of the BO process into a concise textual summary.   This summary includes crucial strategic information, such as the remaining optimization budget and insights from the surrogate model's GP lengthscales, overcoming the limitation of prior adaptive methods that relied only on narrow performance history.\n* Extensive experiments across a diverse set of 50 benchmarks show that LMABO achieves a significant performance improvement over established static, adaptive portfolio, and other LLM-based baselines.   Analysis confirms that the framework’s effectiveness stems from an emergent, state-aware policy that executes active, dynamic switching between exploratory and exploitative AFs in response to real-time progress , demonstrating an agility beyond simple pre-defined heuristics.", "weaknesses": "* The LMABO algorithm requires calling the Large Language Model at every single optimization step. Although the financial cost per token is very low, the overall economic feasibility for extensive or complex problems using commercial LLM APIs requires more detailed analysis.\n* The framework's high performance is fundamentally dependent on the sophisticated reasoning capabilities of the underlying LLM. The core implementation relies predominantly on Gemini-2.5 Flash. Ablation studies confirm this sensitivity, showing a noticeable drop in performance when using smaller open-source models, alongside a dramatic decrease in time efficiency. To ensure the framework's broad reliability and generalizability, its efficacy requires additional, comprehensive validation across a wider spectrum of diverse and competitive LLMs.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "xqhopwnGQO", "reviewer": "Reviewer_Sjvn", "summary": "This paper introduces Language Model-assisted Adaptive Bayesian Optimization (LMABO), a policy for selecting, at each iteration of a BO algorithm, the most promising acquisition function to optimize for finding the next observation. LMABO involves a portfolio of acquisition functions and a Large Language Model (LLM) that is tasked to pick the most promising acquisition function given a prompt that summarizes the current optimization state.\n\nLMABO is evaluated against popular acquisitions functions, state-of-the-art strategies for acquisition function selection and heuristics on 50 benchmarks. The numerical results show a statistically significant improvement over the state-of-the-art.", "questions": "Here are a few questions to spark the discussion with the authors.\n\n* Do you know how a simple control strategy that picks an acquisition function at random in the portfolio would behave? What about a strategy that randomly picks one of the most popular acquisition functions (e.g., a reduced portfolio with EI, TS, UCB, PosMean)?\n\n* How would you plan to add a new, unknown acquisition function to the portfolio?\n\n* For each real-world experiment, have you tried to exploit some prior knowledge of the LLM on this particular task? For example, for an hyperparameter optimization task involving a gradient step $\\epsilon$ and a batch size $b$, have you adapted the initial prompt $P_0$ to describe this particular problem and to indicate to the LLM that it was working on the optimization of a gradient step and a batch size?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 4, "contribution": 2, "strengths": "* A new strategy for selecting an acquisition function would likely spark interest in the BO community.\n\n* The paper is easy to read and well-written.\n\n* Extensive numerical evaluation with relevant statistical tests.", "weaknesses": "* The main weakness is probably the lack of understanding of how the LLM takes its decision. Although LMABO requires a justification from the LLM, there is no guarantee that this justification is actually valid. In that sense, I think an additional control solution, consisting of picking the acquisition function uniformly at random in the acquisition function portfolio, would be very interesting to consider.\n\n* Another worry that I have is that LMABO builds on implicit knowledge of the LLM about acquisition functions in the portfolio. Therefore, integrating a brand new acquisition function in LMABO would probably prove to be challenging, as the LLM would not have any prior knowledge about it.\n\n* Minor presentation issues: a few typos (e.g., \"miultiple\" on line 125) and the captions of Figure 1(c) and 1(d) are hard to read.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)