Large Language Models to Enhance Bayesian Optimization
Authors: Tennison Liu
Nicolás Astorga
Nabeel Seedat
Mihaela van der Schaar
Venue: ICLR 2024 poster (2024)
Keywords: bayesian optimization
LLMs
Primary area: optimization

Abstract: Bayesian optimization (BO) is a powerful approach for optimizing complex and expensive-to-evaluate black-box functions. Its importance is underscored in many applications, notably including hyperparameter tuning, but its efficacy depends on efficiently balancing exploration and exploitation. While there has been substantial progress in BO methods, striking this balance remains a delicate process. In this light, we present \texttt{LLAMBO}, a novel approach that integrates the capabilities of Large Language Models (LLM) within BO. At a high level, we frame the BO problem in natural language, enabling LLMs to iteratively \emph{propose} and \emph{evaluate} promising solutions conditioned on historical evaluations. More specifically, we explore how combining contextual understanding, few-shot learning proficiency, and domain knowledge of LLMs can improve model-based BO. Our findings illustrate that \texttt{LLAMBO} is effective at zero-shot warmstarting, and enhances surrogate modeling and candidate sampling, especially in the early stages of search when observations are sparse. Our approach is performed in context and does not require LLM finetuning. Additionally, it is modular by design, allowing individual components to be integrated into existing BO frameworks, or function cohesively as an end-to-end method. We empirically validate \texttt{LLAMBO}'s efficacy on the problem of hyperparameter tuning, highlighting strong empirical performance across a range of diverse benchmarks, proprietary, and synthetic tasks.

Published as a conference paper at ICLR 2024
LARGE LANGUAGE MODELS TO ENHANCE BAYESIAN OPTIMIZATION
Tennison Liu, Nicolás Astorga, Nabeel Seedat & Mihaela van der Schaar DAMTP, University of Cambridge Cambirdge, UK
{t 1t1522, 4 6) cam. ac. uk
ABSTRACT
Bayesian optimization (BO) is powerful approach for optimizing complex and expensive-to-evaluate black-box functions. Its importance is underscored in many applications, notably including hyperparameter tuning, but its efficacy depends on efficiently balancing exploration and exploitation. While there has been substan- tial progress in BO methods, striking this balance remains delicate process. In this light, we present LLAMBO, a novel approach that integrates the capabilities of Large Language Models (LLM) within BO. At high level, frame the BO problem in natural language, enabling LLMs to iteratively propose and evaluate promising solutions conditioned on historical evaluations. More specifically, we explore how combining contextual understanding, few-shot learning proficiency, and domain knowledge of LLMs can improve model-based BO. Our findings illus- trate that LLAMBO is effective at zero-shot warmstarting, and enhances surrogate modeling and candidate sampling, especially in the early stages of search when observations are sparse. Our approach is performed in context and does not re- quire LLM finetuning. Additionally, it is modular by design, allowing individual components to be integrated into existing BO frameworks, function cohesively as an end-to-end method. We empirically validate LLAMBO'S 's efficacy on the prob- lem of hyperparameter tuning, highlighting strong empirical performance across a range of diverse benchmarks, proprictary, and synthetic tasks.
INTRODUCTION
Black-box optimization. Expensive black-box functions common in many disciplines and ap- plications including robotics 2 experimental design 3 drug discovery 4 interface design [S] and, in machine learning, hyperparameter tuning 66
8) Bayesian optimization (BO) an efficient model-based approach for globally optimizing these functions [9, [0]. BO's s effectiveness lies in its ability to operate based on limited set of observations without the need for direct access to the objective function or its gradients. It does so by using observed data to learn a surrogate model to approximate the black-box function and candidate point sampler to iteratively propose potentially good points. In each trial, the acquisition function selects the proposed point with the highest utility, based on surrogate evaluations. This chosen point undergoes evaluation, and the cycle continues.
7
Challenges of search efficiency. For BO, the name of the game is efficient search, but this efficiency largely depends on the quality of the surrogate model and candidate point sampler to quickly identify high-potential regions (II). Given that BO is designated for scenarios with limited observations, constructing an accurate surrogate model with sparse observations is inherently challenging. Additionally, the model can be sensitive to misspecification, and even slight misrepresentations of the model can introduce undesired bias, skewing the & sampling of potential solutions [12]. A further challenge arises when considering the integration of prior knowledge, especially in effectively transferring knowledge about correlations in the optimization space to new tasks.
At the core, these challenges pertain to accurately learning the objective function and effectively generating candidate solutions with limited data. This scenario is typically framed as the sew-shot
"Equal contribution

---

Published as a conference paper at ICLR 2024
LLAMBO
Hypothenis Generator
Data
Model
Instruction
0. W Warmstarling prompi
Warmstarting bg LLAMBO
Riee javente)
Riee
Hypothesis Evalaation
Add to
1. Candidate sampler prompi
1. L.Candddat Sampler
Perforance: (nd tywepaomseer: tuui
f(*)
LLAMBO(S Pa)
(t))
(hai
Perforaace rpporsrmmstees:
2. Surzogabe model prompt
3. Max. Aoq. Function
Rstimaze score
2. Surzogate Model
Hypxppreetee (aaj parformance (tai
h"
argmax a(h,
LLAMBO D.)
Hypappeeettee thyi pettmmncee (ta)
RrM
Hppapapeeteee f perforaacoe?
Figure I: Overview of LLAMBO. In order: LLAMBO can initialize BO through zero-shot warm- regions given past observations
starting, efficiently sample candidate points from and problem description, and evaluate these candidate points via surrogate model.
setting, context that demands swift learning and generalization from very few examples 13 In-
terestingly, such challenges of the few-shot paradigm align with the proficiencies of Large Language Models (LLM). Contemporary LLMs, which have been pre-trained on Internet-scale data, showcase an exceptional capacity to generalize from sparse data, enabling them excel in few-shot prediction, [5
generation [4
16 17 18), and contextual understanding 19 20 They achieve this remarkable sample-efficient performance, in part, by exploiting encoded priors (2 [2].
Key considerations. This study examines the potential of extending the capabilities of LLMs be- yond standard natural language tasks to enhance model-based BO. Our approach is grounded in representing BO components using natural language, introducing novel methods to effectively cap- ture LLM's distinct strengths. This exploration gives rise to two key questions:
[Q1] Can L.Mss, with their encoded knowledge and few-shor learning abilities, enhance key elements of BO, including the surrogate model and candidate point sampler? [Q2] How effectively can LLM-augmented BO components operate cohesive, end-to-end pipeline? In answering these questions, we chose hy- perparameter tuning (HPT) as our initial area of investigation. This for two main reasons: firstly, the extensive knowledge potentially acquired by LLMs about HPT during pretraining, coupled with its relatively low-dimensional nature, makes an ideal test bed to probe the applications of LLMs within BO. Secondly, HPT is practically important and core enabler in many applications.
Contributions. We present LLAMBO, a novel approach for integrating the capabilities LLMs into BO. To understand the performance gains from this integration, we execute a systematic investiga- tion, exploring the aforementioned questions. Our primary contributions are:
We propose LLAMBO, a novel approach to enhance components of model-based BO with LLMs, We systematically investigate the enhancements of LLAMBO throughout the BO pipeline, show- casing significant improvements to zero-shot warmstarting, the efficacy of the surrogate model, and the efficiency of candidate sampling.
We empirically investigated the end-to-end performance of LLAMBO for hyperparameter tuning, demonstrating strong performance on diverse benchmarks.
2 LLAMBO: LLMs TO ENHANCE BO
Figure illustrates the LLAMBO framework. Fundamentally, our methodology translates different components in the BO pipeline into natural language. This allows the LLM iteratively suggest and evaluate solutions, informed both by the BO problem description and search history.
2.1 THE INTEGRATION OF LLMS INTO BO
Preliminaries. To aid with exposition, we introduce the following notation. Let consider an objective function, ʃ : H + S, where h E H C Rd is the d-dimensional input, and S E R is the output space. We aim find h" H that minimizes this objective function:
h" arg min (h) NEH
where f is a costly black-box function without accessible gradient information. To overcome these limitations, BO employs surrogate model to approximate and candidate sampler to generate
2

---

Published as a conference paper at ICLR 2024
h H.. In general terms, surrogate model can be viewed as machine learning (ML) method producing the predictive distribution of given h and some observed data D,, = {h,, s:)]=
p(s(h;
p; D,) do
Here, the marginalization is over 0, a latent variable that captures the underlying structure between and h. Specifically, (r, h)) x p(D, (,, h(d( describes the posterior distribution after ob- serving some data, with p(0) being the prior knowledge of this underlying structure. The candi- date point sampler can be viewed similarly, as generating samples from a posterior distribution: p(h(D,) = Se phh D,, (()(() do. These priors, p(0), can play significant role, especially given the typically sparse observations in BO adopt non-informative priors, potentially missing out on valuable domain-specific knowledge. The challenge lies not just in the inclusion of prior knowledge, but also in accurately learning the asso- ciated predictive distribution with ML methods, especially in settings with limited observations.
23 However, in practice, many BO applications
Synergy of LLMS and BO. In this light, LLMs can offer significant enhancements due the fol- lowing capabilities: (1) Prior knowledge: Recently, [25] explained LLM in-context learning (ICL) as performing implicit Bayesian inference (14). This raises the interesting prospect of using ICL to tap into an LLM's encoded knowledge for BO. In this framework, p(0) represents the priors related the optimization problem and domain- speccifc correlations absorbed through pretraining 15 26 (2) ICL: Learning generalizable models given only limited observations is highly challeng- ing. LLMs have demonstrated the capacity to generalize from few in-context examples, an ability
that can directly complement BO's needs for sample-efficient exploration [4 27 28 (3) Contex-
tual understanding: LLMs are adept at processing contextual information, especially via natural language [29]. This offers a versatile interface to incorporate meta-feature about optimization tasks, search spaces, and auxiliary details that can improve search performance.
Operationalizing this synergy. Despite these hypothesized advantages of LLMs, effectively cap- italizing on them an iterative optimization framework like BO is challenging. Recently, [30) explored the of LLM- BO for molecules. While this work primarily focused the surro- gate model, we introduce novel methods for LLM enhancement of multiple components of BO and conduct systematic investigation to understand the performance gains offered by this integration. Specifically, we employ ICL to enhance three key components of BO (Figure I::
Warmstarting initializes the optimization process with pre-identified set of points, denoted as {h:]i= which are evaluated first to build up meaningful representation of f- We propose a strategy identify promising initializations through zero-shot prompting.
Warmstarting:
Sampling candidates: The sampling process proposes points {hx } that are considered for future evaluations. Drawing inspiration from TPE (6) propose mechanism conditionally sample candidates based target objective value s': hyg. p(hls';;,)) Here, we employ ICL by providing the optimization history as few-shot examples.
Surrogate modeling: The surrogate model, denoted as p(s[h; D,),, is an approximation of f and is trained using Dn.- Specifically, we introduce two methods, leveraging ICL on the optimiza- tion history: discriminative approach that produces regression estimates with uncertainty, and generative approach that scores via binary classification.
Overview of investigation. Having outlined our approach for leveraging LLMs in BO, we now de- scribe the structure of our investigative study. The framework is presented below, and centres around the two aforementioned questions [Q1-2]. We begin by analyzing each component isolation while keeping other factors consistent whenever possible. We conclude our study with an assessment of LLAMBO'S performance as an end-to-end BO method.
Section
Method
Goal and Method
Q's
Section
Warmstarting
Enhancing optinization warmstarting from LLM prior Improving quality of surrogate model in shot settings through ICI.
[Q1]
Section
Surrogate model
[Q1]
Section
Candidate sampling
Conditional sampling of iigtentttetial points for desired s" viu ICL
[Q1]
Section
End-to-end BO
Augmenting end-to-end BO performance
[Q1][Q2]
It is worth mentioning the different approaches to encode priors: Gaussian Processes
? embed prior
distributions over functions p(f), while Bayesian NNs use prior distributions over weights p(w)
24
3

---

Published as a conference paper at ICLR 2024
Experimental setup. We conduct our investigations using 74 tasks extracted from Bayesmark and HPOBench [31] 32] and OpenAT's GPT-3.5 Language Model (see Appendix D for detailed exper- imental procedures). While it is important to recognize that the choice of LLM can substantially influence the results of optimization, we note that the overarching methodology and fundamental insights covered in this work are broadly applicable beyond the specifies of any single L.M.
2.2 BO PROMPT DESIGN
The proposed integrations are realized through structured natural language queries to the LLM. While the specifics of each query differ (e.g. for surrogate modeling and sampling), they are con- structed from three essential elements. For the complete prompts, please refer to Appendix C.
Problem description. This includes information of the input space H. the output space S, and the objective function f. Specifically for HPT, this entails <MODEL CARD>, describing the ML model being optimized (f), the hyperparameters (H), and the scoring metric (S). We also include
a <DATA CARD> containing dataset attributes.
Optimization history. The history contains the sequence of points and scores observed during the optimization process, captured in D,, The observed points are provided as few-shot examples for ICL of the surrogate model and candidate point sampler.
Task instructions. For each component under consideration (e.g. surrogate model), we include task-specific instructions desired inference and guidelines on the format of the response.
3 RELATED WORKS
Bayesian optimization. At its core, BO relies on probabilistic modeling. One widely adopted tech- nique is the Gaussian Processes (GP) due to their flexibility and analytical tractability 33 Recent works have sought to enhance their expressiveness through deep kernel GP B4 35 and manifold GPs [3). On another front, NN-basce and tree-based surrogates have been considered, particularly due their flexibility in high-dimensional
hierarchical optimization problems 24
38 39)
Tree-structured Parzen Estimator (TPE) is an alternate approach based on the generative surrogate 6 40 [1 Recent trends have also leaned towards Transformers surrogate models BO is commonly used to optimize expensive, black-box functions, including in robotics and experimental design
model p (h)s)
42 43
and most prominently for autoML 40 45 46
Transfer learning for BO. Recent research in BO has explored transfer learning to improve op- timization across similar domains. Prominent among these are multitask GPs, designed to opti- mize several related black-box functions by leveraging common structures or patterns across tasks Other approaches have sought to transfer learnings from previously optimized functions to new functions [50] (51. However, these approaches only consider inductive transfer over fixed search space, i.c. all tasks share the same search space. More recently, [42] introduced a pretrained Transformer for meta-learning across tasks with different search spaces. In our work, explore lightweight alternative by harnessing prior knowledge contained in generalist LLMs, which does not require dedicated pretraining and structured results collected from related optimization problems.
LLMs and optimization. Recent works have explored the use of LLMs for optimization tasks, no- tably for prompt optimization 52 53 54 and as genetic search operators in evolutionary algorithms Of particular note is the research that delves into L.M for BO of molecules in 330, which primarily focused surrogate modeling. In contrast, introduce novel methods for LLM enhancement of multiple components of model-based BO and conduct a systematic investigation to understand the performance gains offered by this integration.
55 56 57].
4 WARMSTARTING THE BO PROCESS
Motivation. We start by analyzing whether LLMs transfer prior knowledge about optimiza- tion problem through warmstarting. While warmstarting can accelerate convergence by supplying more insightful initial points, conventional approaches require prior results collected from similar optimization problems [58 59]. This data collection can be resource-intensive and might not be feasible for certain applications In contrast, we explore the use of LLMs for warmstarting as a more efficient and lightweight alternative, allowing the acquisition of warmstarting points without explicitly requiring data collection from related problems.
4

---

Published as a conference paper at ICLR 2024
Method. LLAMBO employs zero-shot prompting sample points for warmstarting. We explore three distinct settings, each providing different levels of information about the op- timization problem. No context: the LLM is prompted recommend good initial hyperparameters for given MI. model, but no dataset details are provided; A Partial con- text: provides meta-features about the dataset through the <DATA CARD>, including the number of samples, features, the type of features (categorical vs continuous), and the learn- ing task (e.g. classification); Full context: further augments the <DATA CARD> with information on marginal distribu- tions, inter-feature correlations, and feature-label correlations.
Avg Regret trials
Handom
Sobal
ICCabe
0.3
No Context
Partial Contect
Full Context
0.2
0.1
10
15
25
30
Random
No Context
Experimental setup. To evaluate the impact of warmstarting, employ two widely adopted BO methods: Gaussian Pro- cesses (GP) [7] and Tree Parzen Estimator (TPE) (6. We com- pare these against random initialization techniques, namely Random, Sobol, and Latin Hypercube (HCube) sampling. Each search begins with 5 initialization points and proceeds for 25 trials, and we report average results over ten seeded searches. Our evaluation metries focus on two aspects: search performance and the diversity of the initialization points. To assess search performance, we adopt the normalized regret metric, defined as min (h) Smin)/ (s -min), ) where H4 denotes the points chosen up trial and sman and Smaz represent the best and worse scores, respectively [60]. To assess diversity, we use the generalized variance: (E), with Σ being the covariance matrix of the hyperparameters.
Avg corr: 0.2546
Avg corr: 0,3006
Partial Context Avg 0.3905
Full Context
Avg 0,4301
Full Contest
Pastial Contexe
No Context
Empirical insights. (1) Performance: Figure 2 (Top) visual- izes the average regret across all tasks. We begin our analysis with a sanity check-name surpasses the performance of random initialization techniques. This verifies that our LLM possesses a basic knowledge of generalizable correlations (independent of specific problems) between hyperparameters. Interestingly, we observe that pro- viding additional information about the dataset improves the search performance when warmstarting for both partial context and full context. This is partic- ularly prominent in the early stages of the search (i.e. trials < 5). However, these initial gains are maintained as the search progresses. (1a) Correlations: To explore deeper, we compute the correlation matrix sampled warmstarting points depicted (Middle) (with further analysis in Appendix B Our findings reveal that the points recommended by the LLM exhibit considerably greater correlations between hyperparameters compared to those from random initialization. More strikingly, the correlation matrices computed for different tasks reveal different correlation struc- tures, suggesting that the LLM is dynamically adjusting its suggestions to different optimization problems. (2) Diversity: A closer look at the diversity of warmstarting points in (Bottom) reveals that their generalized variance is typically lower than that of randomly initialized points. This trend aligns with our expectations: higher correlations often lead decreased determinant of the co- variance matrix due to 'redundant" information. Since random initialization methods sample each hyperparameter independently, they exhibit lower correlation levels, resulting in higher diversity.
Handom
0.0
0.2
0.4
0.6
0.8
Generalized Variance
warmstarting using no context
Figure 2: Warmstarting. (Top)
average regret, (Middle) correla- tion of sampled points, (Bottom) diversity of initial points sampled with different methods.
& Warmstart initialization via zero-shot prompting is an efficient strategy to transfer knowledge about correlations in the optimization landscape, enhancing search performance.
SURROGATE MODELING
Motivation. Surrogate modeling, core component of BO, aims learn accurate representations of complex functions using only a limited set of evaluations. The efficacy of these models depends on their capacity to generalize and make accurate predictions from sparse observations. Recent
5

---

Published as a conference paper at ICLR 2024
NRA3SD (U)
04
Reggot CA)
Scooo
GP
SMAC
30
LLAMBO
10
0.5
(MC)
LPD
Clowmaage
0.6
LLAMBO
0.6
Number Observed Points
"Dusted groy five regroeesis desicd esuengee st. att dee
Figure 3: Discriminative surrogate models. (Top) prediction performance measured in NRMSE and R2, and regret; (Bottom) uncertainty calibration evaluated using LPD, coverage, and sharpness.
studies have underscored LLM's remarkable ability to perform few-shot learning 25 61.. Building on this, we propose two tailored approaches to surrogate modeling via ICL: (1) a discriminative approach to predict the mean and uncertainty of the objective value of given candidate point, Le. p(s h:; D.)) in Section 5.1, and (2) generative approach that scores each point based on the probability that its objective value is better than some performance threshold т, i.e. p(s S T|h; Dn)) in Appendix [ These represent two distinct approaches, with (1) framing surrogate modeling as a regression problem, while (2) views surrogate modeling probabilistic binary classification.
5.1 DISCRIMINATIVE SURROGATE MODEL
One of the main approaches to surrogate modeling involves learning the conditional probability of the output 8 given the input is using data D,,, expressed p(s]hı; (,,)), D discriminative approach. An effective surrogate model should produce an accurate mean prediction of the objective function's 's central tendencies. and well-calibrated uncertainty estimates to balance exploration and exploitation.
Method. We serialize the observed optimization trajectory into natural text. For example, with h; as an RF's hyperparameters and s; the accuracy, the serialization would read: "max depth is 15, min maiples -spiit is 0.5...., accuracy is 0.9" [22 63). These text representations, for all n observed samples, are concatenated into few-shot examples, symbolized as D Here, we use the superscript ni to mean representations of observations in natural text. Together with the problem description and query example hit they form the input to the LLM. For each query, the LLM outputs a response: (šy, p(š))) denoting the predicted score and associated probability, respectively: (&k, p(.))) LLAMBO(AE De To obtain probabilistic estimates, this prediction step is repeated times, from which we compute the empirical mean and standard deviation. This Monte Carlo-based approach is termed LLAMBO (MC), and mirrors the method proposed in 30
Our empirical observations revealed that the MC implementation often achieved suboptimal calibra- tion of uncertainty estimates. After further explorations, we found the sensitivity to the ordering of in-context examples likely cause. As L.MSs process inputs in left-to-right manner, the predic- tions are sensitive to permutations within the prompt [64 65). To enhance robustness, we introduce shuffling mechanism that randomly permutes the few-shot examples within D which is com- bined with MC sampling. We acknowledge that while this approach is not grounded in principled probabilistic reasoning-simin similar to the popular SMAC method [8 can be an effective technique to obtain probabilistic estimates. This improved method is hereby referred to as LLAMBO.
Experimental setup. We compare LLAMBO against GP and SMAC, two established surrogate mod- els. We evaluate these probabilistic discriminative models via prediction performance and uncer- tainty calibration. For performance metrics, we use NRMSE (4) and R2 (t). Calibration assessed using the scoring rule, log predictive density (LPD) (L), empirical coverage (where the desired cov- erage for I standard deviation, assuming Gaussianity, is 0.68), and sharpness (J) (66). We also include normalized regret of the point acquired using expected improvement (EI) [1O]. Our goal is to assess the surrogate model's efficacy when a different number of evaluations are available (n). We evaluate each task when E [5, 10, 20, 30), and test predictions against 20 unseen points.
Empirical insights. (1) Prediction performance: Figure 3 (Top) plots the NRMSE and R² against the number of observed samples. L.LAMRO consistently outperforms in prediction across all sam-

---

Published as a conference paper at ICLR 2024
ple counts, particularly with fewer observed samples. Moving on, examine normalized regret: notably, all methods show increased regret at n = 5, this reflects greater uncertainty across unex- plored regions, leading to heightened levels of exploration (and higher regret). For > 5, LLAMBO attains lower regret, demonstrating better exploitation than other methods. (2) Uncertainty quan- tification: (Bottom) assesses uncertainty quantification. We find that, in this aspect, GPs, with their probabilistic grounding, produce the best uncertainty estimates, particularly in LPD and empirical coverage. GPs maintain good coverage even with low number of samples, while LLAMBO only approaches similar performances as increases. In this regard, our approach exhibits performance more similar SMAC, a frequentist method that also makes use of empirical variance. Interest- ingly, we note that the sharpness of uncertainty intervals for GPs remains consistently higher, while in LLAMBO, the sharpness decreases as the coverage improves. This likely due to the better predic- tion performance, enabling the predictions to be more confident (lower sharpness) while achieving improved empirical coverage. (3) LLAMBO vs LLAMBO (MC) The purely MC-driven approach ex- hibits subpar uncertainty calibration, evident through worse LPD and coverage metrics. Coupled with low sharpness values, this suggests the predictions are overly confident, tending to underesti- mate uncertainty. We also observe that LLAMBO consistently achieves better prediction performance. As such, empirical evidence supports that permuting few-shot ex- amples, while straightforward in implementation, improves both uncertainty quantification and prediction performance, both criti- cal aspects of balancing exploration and exploitation. (4) Role of prior knowledge Lastly, we investigate the importance of prior knowledge to LLAMBO'S few-shot performances. To this end, we introduce an ablation setting LLAMBO (UnInf) where the problem description (containing the <DATA CARD> and <MODEL CARD>) are omitted, and the hyperparameter names are substi- tuted with "X,". Figure 4 reveals better prediction performance and calibration when compared to the uninformative ablation. This reveals the crucial role of prior knowledge in enhancing surrogate modeling, especially in few-shot settings [25 63
1.5
LLABB
(Uniaf)
NRMSE
0.4
LLAMBO
LPD
0.2
Number of Observed Points
Figure 4: Ablation. LLAMBO (UnInf) omits problem de- scription and hyperparameter names.
@ Discriminative surrogate models implemented through ICL can produce effective regression estimates with uncertainty, although there is tradeoff of stronger prediction performance with worse calibration than probabilistic methods. The LLM's ' encoded prior is crucial to improving the efficacy of such surrogate models.
6 SAMPLING OF CANDIDATE POINTS
Motivation. The sampling of candidate points is another crucial component of BO, as high-potential points can speed up convergence to the optimal solution. In this context, we present novel mecha- nism to conditionally generate candidate points based on desired objective values through ICL.
Method. Our proposed sampling mechanism draws inspiration from TPE. While TPE focuses on sampling candidate points, denoted as hm from 'good' regions in the search space (i.e. hm I(h) = p(h]s Σ t; D,)). we sample from regions of high potential by directly conditioning on a desired objective value s': hi ~ p(h s'; Dn). This distinction is fundamental as it allows us to target specific objective values, something TPE's binary categorization cannot achieve. The few-shot generation capabilities of LLMs are crucial here, as learning such a conditional generator through conventional means poses significant challenges due to the limited number of observations.
We define the desired objective value using the equation: s' = Smin a (Smaz "maz--mmin) Smin), ), where smoz and Smin are the worst and best objective values observed up until that point. Intuitively, s' is defined relative to the best objective value, with the difference proportional to the observed variability in s. The exact value is controlled by a, the exploration hyperparameter. essentially extrapolating, which cannot be achieved through conventional
positive a sets s' to improve
over 5min. Here, TPE. Conversely, a negative a (i.c. -1 < a < 0) results in a more conservative target value that is within the observed objective value range. To operationalize this, we implement p(hll D,,) through ICI. We generate M candidate points independently, i.e. hig LLAMBO(S' 8', D. after which,
7

---

Published as a conference paper at ICLR 2024
Gen Variance (1)
Reellhooo (1)
TPE (Ind)
0.6
-10
Avg Regret
0.6
Best Regret
0.2
-1n
TPE (Mult)
Randon
0.4
LLAMBO
(Uninf)
0.2
LLAMBO
Number Observed Points
Figure 6: Candidate point sampling. Quality is evaluated using average regret and best regret.
Diversity is assessed using generalized variance and loglliiklihood.
select the point that maximizes the acquisition function the point to evaluate next. Thus, our approach, like TPE, uses a sampling-bas -bseed approximation to optimize the acquisition function.
Experimental setup. We compare our proposed sampler against TPE (Ind), TPE (Multi), and ran- dom sampling (Random). As before, we also include ablation of our method LLAMBO (UnInf), which omits problem description and hyperparameter names. Our analysis examines two aspects: candidate point quality and diversity. To evaluate quality, we compute the average regret (L) and use generalized vari- ance (T) to evaluate the spread of candidate points and log-likelihood (f) to assess the probability of candidate points being sampled from observed points. We start by investigating the effect of a on sampling performance. Then, following the experimental procedure outlined previously, we evaluate sampling performance when a different number of observations available.
best regret (L) among the M sampled points [60]. For assessing diversity,
Empirical insights. (1) Effect of a: In Figure 5 we observe that as increases from -0.5 to 0, both erage regret and best regret improves. However, as a
Gmn Vatiance
Tong Liloliooo
0.4
increases beyond the average regret increases as the candidate points are increasingly sampled from beyond the observed distribution, compromising the reliability of these points. Interestingly, the optimal best regret emerges at a=0.01, hinting at our mechanism's ability to extrapolate from observed distributions. The gen- cralized variance decreases with increasing a, this is reasonable as the candidate points are sampled from smaller regions in the search space. Similarly, the log- likelihood decreases increasingly sampled away from observed points. To confirm that this is indeed the case, visually examine t-SNE projections of sampled points, localizing them against good (top 20% of samples) and bad points [67]. We note that when a=-0.2, the candidate points cover similar region as good points, but when o=0.01, the sampled points are observed outside the regions of good points. (2) Quality: Figure 6 compares the quality of our sampled points against baselines, with our method set at 0.2. We observe that LLAMBO consistently achieves the lowest average and best regret varies, but is especially notable
0.0
-05
0.00
0.01
0.1
Regred
Regnes
0.30
62
Exploration Hyporparannter
10
10
increases, the points
Good Points
Bud Points
LLAMBO
Figure 5: (Top) quality and diversity of points sampled with different a. (Bot- tom) projection of sampled points at 0.2 and a=0.01.
5. This gain also present when compared against the ablation, suggesting the crucial role of prior knowledge in proposing nigh-potential candidates. (3) Diversity: An examination of generalized variance reveals that TPE (Ind) proposes more diverse points. In contrast, the spread of LLAMBO-sampled points is similar to that achieved by TPE (Multi). This is reasonable, as both LLAMBO and TPE (Multi) model correlations, while TPE (Ind) models each dimension independently (and higher correlation decreases generalized variance). Furthermore, the log-likelihood of LLAMBO proposed points the highest, indicating that they are more plausible given the observed points.
@ Sampling candidate points by direct conditioning desired target value can generate high- quality points, although this can sacrifice diversity among sampled points. The a exploration hyperparameter allows balancing of this trade-off.
8

---

Published as a conference paper at ICLR 2024
7 END-TO-END DEMONSTRATION OF LLAMBO
Motivation. Having examined the integration of LLMs into key components of BO, we now holis- tically evaluate the performance of LLAMBC as end-to-end BO algorithm. Here, we instantiate LLAMBO with our discriminative surrogate model, as this is the most classic form of surrogate modeling.
Public
Private + Synthelic
Ang Regret
02
0.2
10
25
30
15
20
30
Experimental setup. We evaluate BO performance on 25 tasks extracted from Bayesmark [31], a con- tinuous HPT benchmark. Here, a task is a dataset- ML model pair. and we consider all 5 included datasets and 5 ML models. Additionally, we intro- duce 3 proprietary and 2 synthetic datasets into the benchmark- these are datasets for which the LLM would not have seen during pretraining, and thus serve to check for any memorization concerns. This results in a total of 50 HPT tasks, where for each task, executed 5 seeded searches, each with 25 trials. Baselines. We compare LLAMBO against 4 established baselines commonly used in production: GP-DKL (34), SKOpt (GP) [6]. Op- tuna (TPE) [1). and SMAC3 (RF) [8] To ensure a fair comparison, we do not use warmstarting and initialize all methods with the same set of 5 randomly sampled points in each run. We describe
Trials
Trials
Random
SMACS
Optuna (TPR))
GP DEL (BoTbeck)
5KO (CI)
LLAMBO
Figure 7: End-to-end performance of LLAMBO. (Left) average regret on public datasets and (Right) private and synthetic datasets evaluated on Bayesmark.
complete experimental details in Appendix D
Empirical insights. (1) Performance: Figure shows the average regrets across all HPT tasks on both public Bayesmark datasets, and private and synthetic datasets. We note that in both settings, LLAMBO achieves the best tuning performance. Additionally, we observe that, consistent with prior findings, LLAMBO excels in earlier stages of the search, when fewer observations are available. (2)
Additional results: In the interest of space, we include additional results in Appendix E Specifi- cally, we evaluate LLAMBO with our generative surrogate model; P compare against additional baselines on Bayesmark; evaluate BO performance on 24 additional tasks from HPOBench [32] and report individual task search results (by task metric, average regret, and average rank).
& LLAMBO performs effectively as an end-to-en end pipeline, exhibiting sample- eeffcceen search. Its modularity further enables individual components be integruted into existing frameworks
8 DISCUSSIONS
In summary, we introduced LLAMBO, novel framework that integrated LLM capabilities to enhance model-based BO. Our approach introduced three specific enhancements: zero-shot warmstart- ing to initialize search, generative and discriminative surrogate models of the objective function via ICL, and a candidate point sampler that can conditionally generate for specific target values. Our investigative study the problem of HPT uncovered performance improvements across all three integrations, which was especially notable when fewer samples were available. Additionally, we found that LLAMBO to be an effective stand-alone BO method, exemplified through superior performance on diverse benchmarks.
Limitations & future works. While LLAMBO does not perform any finetuning, performing infer- ence through LLMs incurs a much larger computational footprint than traditional BO algorithms. Our findings indicated that LLAMBO trades off this computational complexity for improved sam- ple efficiency, an especially desirable property in black-box optimization tasks. This suggests the potential fusion of LLAMBO with more computationally efficient methods. For instance, deploying LLAMBO in earlier stages of the search, only leveraging an individual component to complement existing BO frameworks. Additionally, while we have demonstrated the potential for integrating LLM in BO with GPT-3.5, it is important to recognize the choice of LLMs significantly influ- ence optimization results. A promising future direction involves benchmarking various L.LMs, to understand their strengths and limitations in different BO problem settings. Our study has primarily focused on HPT tasks, which are relatively low-dimensional. However, a notable area for future re- search is the expansion LLAMBO'S application higher-dimensional BO tasks with more complex search spaces, such as neural architecture search and robotic control 39 699

---

Published as a conference paper at ICLR 2024
ETHICS AND REPRODUCIBILITY STATEMENTS
Ethics. In this work, we evaluate both public benchmarks and private datasets. The private datasets are de-identified and used following the guidance of the respective data providers. We follow rec- ommendations use the Azure OpenAl service when using GPT models, where via the agreement we ensure the medical data is not sent for human review or stored, hence respecting the guidelines given by the dataset providers.
Reproducibility. Experimental investigations are described in Sections to with further details of the method, experimental setup, and datasets included in Appendix Appendix [ We provide the code to reproduce our results at tps: /ttpt:::::gttttbpbppp http://www...oeeemeeoooasaaaaaae wider lab repository / com/ wwtp://eerreaaaaaaaaaaaaa...
and the
ACKNOWLEDGMENTS
We thank the anonymous ICLR reviewers, members of the van der Schaar lab, and Andrew Rashbass for many insightful comments and suggestions. TL would like to thank AstraZeneca for their spon- sorship and support. NA thanks W.D. Armstrong Trust for their support. NS is funded by the Cystic Fibrosis Trust. This work supported by Microsoft's Accelerate Foundation Models Academic Research initiative.
REFERENCES
[1] D. J. Lizotte, T. Wang, M. H. Bowling, D. Schuurmans et al., "Automatic gait optimization with gaussian process regression." IJCAI, vol. 7, 2007, pp. 944 949.
[2] R. Calandra, A. Seyfarth, J. Peters, and M. P. Deisenroth, "Bayesian optimization for learning gaits under uncertainty: An experimental comparison dynamic bipedal walker," Annals of Mathematics and Artificial Intelligence, vol. 76, pp. 5-23, 2016.
[3] S. Greenhill, S. Rana, S. Gupta, P. Vellanki. and S. Venkatesh, "Bayesian optimization for adaptive experimental design: A review," IEEE access, vol. 8, pp. 13 937-13 948, 2020.
[4] K. Korovina, S. Xu, K. Kandasamy, W. Neiswanger, B. Poczos, J. Schneider, and E. Xing, "Chembo: Bayesian optimization of small organic molecules with synthesizable recommen- dations." " in International Conference on Artificial Intelligence and Statistics. PMLR, 2020, pp. 3393-3403. 3403.
[5] E. Brochu, T. Brochu, and N. De Freitas, "A bayesian interactive optimization approach to pro- cedural animation design," in Proceedings of the 2010 ACM SIGGRAPHH/Samggraphicc Sym- posium on Computer Animation, 2010, pp. 103-112.
[6] J. Bergstra, R. Bardenet, Y. Bengio, and B. Kégl. "Algorithms for hyper-parameter optimiza- tion," Advances in neural information processing systems, vol. 24, 2011.
[7] J. Snock, H. Larochelle, and R. P. Adams, "Practical bayesian optimization of machine learning algorithms," " Advances neural information processing systems, vol. 25, 2012.
[8] M. Lindauer, K. Eggensperger, M. Feurer, A. Biedenkapp. D. Deng, C. Benjamins, T. Ruhkopf, R. Sass, and F. Hutter, "Smac3: A versatile bayesian optimization package for hyperparameter optimization," The Journal of Machine Learning Research, vol. 23, no. 1, pp. 2475-2483, 2022.
[9] H. Kushner, "A new method of locating the maximum point an arbitrary multipeak curve in the presence of noise," Journal of Basic Engineering, vol. 86, no. pp. 97-106, 1964.
[10] D. R. Jones, M. Schonlau, and W. Welch, "Efficient global optimization of expensive black- box functions." Journal of Global optimization, vol. 13, pp. 455-492, 1998.
[11] T. Desautels, A. Krause, and J. W. Burdick, "Parallelizing in gaussian process bandit optimization," Journal of Machine Learning Research, vol. 15, pp. 3873-3923, 2014.
copration-expllitation tradeoffs
10

---

Published as a conference paper at ICLR 2024
[12] P. I. Frazier, "A tutorial on bayesian optimization," arXiv preprint
2018.
[13] Y. Wang, Q. Yao, J. T. Kwok, and L. M. Ni, "Generalizing from few examples: A survey few-shot learning," ACM computing surveys (csur), vol. 53, no. 3, pp. 1-34, 2020.
[14] Brown, B. Mann, N. Ryder, M. Subbiah,
Kaplan, P. Dhariwal, A. Neelakantan, P. Shyam, G. Sastry, A. Askell al., "Language models are few-shot learners," Advances in neural information processing systems, vol. 33, pp. 1877-1901, 2020.
[15] A. Chowdhery, Narang, Devlin, M. Bosma, G. Mishra, A. Roberts, P. Barham, H. Chung, C. Sutton, S. Gehrmamn al., "Palm: Scaling language modeling with pathways," arXiv preprint arXiv:2204.02311, 2022.
[16] G. Penedo, Q. Malartic, D. Hesslow, R. Cojocaru, A. Cappelli, H. Alobeidli, B. Pannier, E. AI- mazrouei, and J. Launay, "The refinedweb dataset for falcon Ilm: outperforming curated cor- pora with web data, and web data only," arXiv preprint arXiv:2306.0/116. 2023.
[17] N. Seedat, N. Huynh, B. van Breugel, and M. van der Schaar, "Curated Ilm: Synergy of Ilms and data curation for tabular augmentation in ultra low-data regimes," arXiv preprint arXiv:2312.1 12112, 2023.
[18] S. Holt, M. R. Luyten, and M. van der Schaar, "I "1.2mac: Large language model automatic com- puter for unbounded code generation," in The Twelfth International Conference on Learning Representations, 2023.
[19] J. Wei, X. Wang, D. Schuurmans, M. Bosma, F. Xia, E. Chi, Q. Le D. Zhou et al., "Chain- of-thought prompting elicits reasoning in large language models," Advances in Neural Infor- mation P Processing Systems, vol. 35, pp. 24 824-24422450 247,
[20] J. Wei, Y. Tay, R. Bommasani. C. Rafel, B. Zoph, S. Borgeaud, D. Yogatama, M. Bosma, D. Zhou, D. Metzler al., "Emergent abilities of large language models," arXiv preprint arXiv:2206.00 07682, 2022.
[21] V. Sanh, A. Webson, C. Raffel, S. Bach, L. Sutawika, Z. Alyafeai, A. Chaffin, A. Stiegler, A. Raja, M. Dey, M. S. Bari, C. Xu, U. Thakker, S. S. Sharma, E. Szezechla, T. Kim, G. Chhablani, N. Nayak, D. Datta, J. Chang, M. T.-J. Jiang, H. Wang, M. Manica, S. Shen, Z. X. Yong, H. Pandey, R. Bawden, T. Wang. T. Neeraj. Rozen, A. Sharma, A. Santilli, T. Fevry, J. A. Fries, R. Teehan, T.. L. Scao, S. Biderman, I.. Gao, T. Wolf, and M. Rush, "Multitask prompted training enables zero-shot task generalization." in International Conference on Learning Representations, 2022. [Online]. Available: http:/ppeeeeieee..ec/ouummmm/doo
[22] L. Ouyang, J. Wu, X. Jiang, D. Almeida, C. Wainwright, P. Mishkin, C. Zhang, S. Agarwal, K. Slama, A. Ray et al., "Training language models to follow instructions with human feed- back," Advances in Neural Information Processing Systems, vol. 35, pp. 27 27700-27 744, 2022.
[23] O. Berger, Statistical decision theory and Bayesian analysis. Springer Science & Business Media, 2013.
[24] J. Snoek, O. Rippel, K. Swersky, R. Kiros, N. Satish, N. Sundaram, M. Patwary, Prabhat, and R. Adams, "Scalable bayesian optimization using deep neural networks," in International Conference Machine Learning, 2015, pp. 2171-2180.
[25] S. M. Xie, A. Raghunathan, P. Liang, and T. Ma. "An explanation of in-context learning as implicit bayesian inference," International Conference on Learning Representations, 2022. [Online]. Available:
[26] K. Singhal, S. Azizi, T. Tu, S. S. Mahdavi, J. Wei, H. W. Chung, N. Scales, A. Tanwani, H. Cole-Lewis, Pfohl et al., "Large language models encode clinical knowledge." Nature, pp. 1-9, 2023.
[27] J. Wei, J. Wei, Y. Tay, D. Tran, A. Webson, Y. Lu, X. Chen, H. Liu, D. Huang, D. Zhou al., "Larger language models do in-context learning differently." arXiv preprint arXiv:2303.00 038466 2023.
11

---

Published as a conference paper at ICLR 2024
[28] S. Mirchandani, F. Xia, P. Florence, B. Ichter, D. Driess, M. G. Arenas, K. Rao, D. Sadigh, and A. Zeng, "Large language models as general pattern machines," arXiv preprint arXiv:2307.04221, 2023.
[29] C. Yang, X. Wang. Y. Lu, Liu, Q. Le, D. Zhou, and X. Chen, "Large language models as optimizers," arXiv preprint FrXiv:2309003099 2023.
[30] M. C. Ramos, S. S. Michtavy, M. D. Porosoff, and A. D. White, "Bayesian optimization of catalysts with in-context learning," arXiv preprint arXiv:2304.0534/ 2023.
[31] Uber, "Uber/bayesmarkk: Benchmark framework to easily compare bayesian optimization methods on real machine learning tasks," 2020. [Online]. Available: https///gthub..co//bberr bayesmark
[32] K. Eggensperger, P. Müller, N. Mallik, M. Feurer, R. Sass, A. Klein, N. Awad, M. Lindauer, and F. Hutter, "Hpobench: A collection of reproducible multi-fidelie benchmark problems for hpo." in Thirty-fitth Conference on Neural Information Processing Systems Datasets and Benchmarks Track (Round 2), 2021.
[33] C. E. Rasmussen, C. K. Williams et al., Gaussian processes for machine learning. Springer, 2006, vol.
[34] A. G. Wilson, Z. Hu, R. Salakhutdinov, and E. P. Xing, "Deep kernel learning," in Proceedings of the 19th International Conference on Artificial Intelligence and Statistics, ser. Proceedings of Machine Learning Research, A. Gretton and C. C. Robert, Eds., vol. 51. Cadiz, Spain: PMLR, 09-11 May 2016, pp. 370-378. [Online]. Available: https:/proceedings.mlprrsss// httpresss/i/wwwloso........
[35] A. Hebbal, L. Brevault, M. Balesdent, E.-G. Taibi, and N. Melab, "Efficient global optimization using deep gaussian processes," in 2018 IEEE Congress on evolutionary computation (CEC). IEEE, 2018, pp. 1-8.
[36] R. Calandra, J. Peters, C. F. Rasmussen, and M. P. Deisenroth, "Manifold gaussian processes for regression," 2016 International joint conference on neural networks (IJCNN). IEEE, 2016, pp. 3338-3345.
[37] F. Hutter, H. H. Hoos, and K. Leyton-Brown, "Sequential model-based optimization for general algorithm contiguration," in International Conference on Learning and Intelligent Optimiza- tion. Springer, 2011, pp. 507-523.
[38] J. T. Springenherg, A. Klein, S. Falkner, and F. Hutter, "Bayesian optimization with robust bayesian neural networks," in Advances in Neural Information Processing Systems, D. Lee, M. Sugiyama, U. Luxburg. I. Guyon, and R. Garnett, Eds., vol. 29. Curran Associates, Inc., 2016. [Online]. Available:
a96d3afec 84766bfecc 7t17766bbeca7799989968 7e http:://pcceeddnnge.eeerpc.cpppp
[39] Z. Dai, Y. Shu, B. K. H. Low, and P. Jaillet, "Sample-then-optimnz" batch neural thompson sampling," Advances in Neural Information Processing Systems, vol. 35, pp. 23 331-23 344, 2022.
[40] J. Bergstra, B. Komer, C. Eliasmith, D. Yamins, and D. D. Cox, "Hyperopt: A python library for model selection and hyperparameter optimization." Computational Science & Discovery, vol. 8, no. 1, 014008, 2015.
[41] T. Akiba, S. Sano, T. Yanase, T. Ohta, and M. Koyama, "Optuna: A next-generation hyper- parameter optimization framework," in Proceedings of the 25th ACM SIGKDD International Conference Knowledge Discovery and Data Mining, 2019.
[42] Y. Chen, X. Song, C. Lee, Z. Wang, R. Zhang, D. Dohan, K. Kawakami, G. Kochanski, A. Doucet, M. Ranzato et al., "Towards learning universal hyperparameter optimizers with transformers." Advances in Neural Information Processing Systems, vol. 35, pp. 350 32 (668, 2022.
12

---

Published as a conference paper at ICLR 2024
[43] S. Müller, M. Feurer, N. Hollmann, and F. Hutter, "Pfns4bo: In-context learning for bayesian optimization." arXiv preprint arXiv:230S.17535. 2023.
[44] S. Falkner, A. Klein, and F. Hutter, "Bohb: Robust and efficient hyperparameter optimization at scale," " in International conference on machine learning. PMLR, 2018, pp. 1437-1446.
[45] D. Jarrett, B. C. Cebere, T. Liu, A. Curth, and M. van der Schaar, "Hyperimpute: Generalized iterative imputation with automatic model selection," " in International Conference on Machine Learning. PMLR, 2022, pp. 9916-9937.
[46] Imrie, B. Cebere, E. F. McKinney, and M. van der Schaar, "Autoprognosis 2.0: Democ- ratizing diagnostic and prognostic modeling in healthcare with automated machine learning," PLOS Digital Health, vol. 2. no. 6, p. e0000276, 2023.
[47] K. Swersky, J. Snoek, and R. P. Adams, "Multi-task bayesian optimization," Advances in neu- ral information processing systems, vol. 26, 2013.
[48] M. Wistuba and J. Grabocka, "Few-shot bayesian optimization with deep kernel surrogates," in International Conference Learning Representations, 2021.
[49] M. Feurer, B. Letham, and E. Bakshy, "Scalable meta-learning for bayesian optimization," arXiv preprint htt.:0822002299 2018.
[50] R. Bardenet, M. Brendel, B. Kégl, and M. Sebag, "Collaborative hyperparameter tuning," in International conference on machine learning. PMLR, 2013, pp. 199-207.
[51] M. Poloczek, J. Wang, and P. I. Frazier, "Warm starting bayesian optimization," in 2016 Winter Simulation Conference (WSC). IEFE, 2016, pp. 770-781.
[52] H. Sun, A. Hüyük, and M. van der Schaar, "Query-dependens prompt evaluation and optimization with offline inverse RL," in The Twelfth International Conference on Learning Representations, 2024. [Online]. Available:
[53] C. Yang, X. Wang, Y. Lu, Liu, Q. Le, D. Zhou, and X. Chen, "Large language models as optimizers," arXiv preprint arXiv:2309.0344 2023.
[54] Q. Guo. R. Wang. J. Guo, Li, K. Song, X. Tan, G. Liu. J. Bian, and Yang, "Connecting large language models with evolutionary algorithms yields powerful prompt optimizers," in The Twelfth International Conference on Learning Representations, 2024. [Online]. Available: http://opennevvee.eeeffommmm Rddee//o
[55] E. Meyerson. M. J. Nelson, H. Bradley, Gaier, A. Moradi, A. K. Hoover, and J. Lehman, "Language model crossover: Variation through few-shot prompting," arXiv preprint arX1v:2302.122170 2023.
[56] J. Lehman, J. Gordon, S. Jain, K. Ndousse, C. Yeh, and K. O. Stanley, "Evolution through large models," in Handbook of Evolutionary Machine Learning. Springer, 2023, pp. 331-366.
[57] A. Chen, D. Dohan, and D. So, "Evoprompting: Language models for code-level neural archi- tecture search," Advances in Neural Information Processing Systems, vol. 36, 2024.
[58] M. Feurer, J. T. Springenberg, and F. Hutter, "Initializing bayesian hyper-parameter optimiza- tion via meta-lcarning: in Twenty-Ninth AAAI Conference on Artificial Intelligence, 2015.
[59] H. S. Jomaa, features," Data Mining and Knowledge Discovery, vol. 35, no. 3, pp. 964-985, 2021.
Schnidt-Thieme and J. Grabocka, "Dataset2vec: Learning dataset meta-
[60] S. P. Arango, H. S. Jomaa, M. Wistuba, and J. Grabocka, "Hpo-b: A large-scale reproducible benchmark for black-box hpo based on openml," in Thirty-fifth Conference on Neural Infor- mation Processing Systems Datasets and Benchmarks Track (Round 2), 2021.
[61] C. Han, Z. Wang. H. Zhao, and H. Ji, "In-context learning of large language models explained kernel regression," arXiv preprint arXiv: 12766, 2023.
13

---

Published as a conference paper at ICLR 2024
[62] T. Dinh, Y. Zeng, R. Zhang, Z. Lin, M. Gira, S. Rajput, J. Y. Sohn, D. Papailiopoulos, and K. Lee, "Lift: Language-interfacee interfaced fine-tuning for non-language machine learning tasks," in Advances in Neural Information Processing Systems, A. H. Oh, A. Agarwal, D. Belgrave, and K. Cho, Eds., 2022.
[63] S. Hegselmann, A. Buendia, H. Lang, M. Agrawal, X. Jiang, and D. Sontag, "Tabllm: Few- shot classification of tabular data with large language models," in International Conference Artificial Intelligence and Statistics. PMLR, 2023, pp. 5549-5581.
[64] Z. Zhao, E. Wallace, Feng, D. Klein, and S. Singh, "Calibrate before use: Improving few- shot performance of language models," in International Conference Machine Learning- PMLR, 2021, pp. 12 627-12 706.
[65] Y. Lu, M. Bartolo, A. Moore, S. Riedel, and P. Stenetorp, "Fantastically ordered prompts and where to find them: Overcoming few-shot prompt order sensitivity," in Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), 2022, pp. 8086-8098.
[66] P. D. Arendt, D. W. Apley, and W. Chen, "Quantification of model uncertainty: Calibration, model discrepancy, and identifiability" Journal of mechanical design, vol. 134, no. 10, 2012.
[67] L. Van der Maaten and G. Hinton, "Visualizing data using t-sne." Journal of machine learning research, vol. 9, no. 11, 2008.
[68] F. Pedregosa, G. Varoquaux, A. Gramfort, V. Michel, B. Thirion. O. Grisel, M. Blondel, Prettenhofer,
Weiss. V. Dubourg, J. Vanderplas, A. Passos, D. Coumaaeea,, M. Brucher, M. Perrot, and E. Duchesnay, "Scikit-learn: Machine learning in Python," Journal of Machine Learning Research, vol. 12, pp. 2825 2830, 2011. [Online]. Available: pttpppppes/vollumeee///rrrrggggo 12/pedregosal la/pedregosal la.pdf
[69] K. Kandasamy, W. Neiswanger, J. Schneider, B. Poczos, and E. P. Xing, "Neural architec- ture search with bayesian optimisation and optimal transport," Advances in neural information processing systems, vol. 31, 2018.
[70] N. Srinivas, A. Krause, S. Kakade, and M. Seeger. "Gaussian process optimization in the bandit setting: no regret and experimental design," in Proceedings of the 27th International Conference International Conference on Machine Learning, 2010, pp. 1015-1022.
[71] M. Sugiyama. T. Suzuki, and T. Kanamori, Density ratio estimation in machine learning. Cambridge University Press, 2012.
[72] M. A. Duggan, W. F. Anderson, Altekruse, L. Penberthy, and M. E. Sherman, "The surveil- lance, epidemiology and end results (SEER) program and pathology: towards strengthening the critical relationship." The American Journal of Surgical Pathology, vol. 40, no. 12, p. e94, 2016.
[73] S. J. Pocock, C. A. Ariti, J. McMurray, A. Maggioni, L. Køber, I. B. Squire, K. Swedberg, Dobson, K. K. Poppe. G. A. Whalley al., "Predicting survival in heart failure: risk score based on 39 372 patients from 30 studies," European Heart Journal, vol. 34, no. 19, pp. 1404-1413. 2013.
[74] C. U. PCUK, "Cutract," hutp://pposaeeecnceenk.rog 2019.
[75] S. Watanabe, "Trre-structured Parzen estimator: Understanding its algorithm components and their roles for better empirical performance," arXiv preprint arXiv:2304.11 11..000.1112... 127, 2023.
[76] D. Eriksson, M. Pearce, J. Gardner, R. D. Turner, and M. Poloczek. "Scalable global optimization via local Bayesian optimization," in Advances in Neural Information Processing Systems, 2019, pp. 5496 5507. [Online]. Available: http://papers.miss.ccppppp httg:saaabbeggbbllliiiaaaaaooooa
14

---

Published as a conference paper at ICLR 2024
[77] A. I. Cowen-Rivers W. Lyu, R. Tutunov, Z. Wang, A. Grosnit, R. R. Griffiths, A. M. Mar- aval, H. Jianye, J. Wang, J. Peters et al., "Hebo: Pushing the limits of sample-efficiee hyper- parameter optimisation," Journal of Artificial Intelligence Research, vol. 74, pp. 1269-1349 1349, 2022.
[78] D. Duvenaud, "Automatic model construction with gaussian processes," Ph.D. dissertation, University of Cambridge, 2014.
15

---

Published as a conference paper at ICLR 2024
A BAYESIAN OPTIMIZATION BACKGROUND
A.1 PRELIMINARIES
Consider objective function: f H S, where h E H. is the imput (which could be R for d-dimensional input) and E is the output space. We aim to find h" E H that minimizes the objective function:
h" arg min (h) hCH
However, f is assumed to be a black-box function and evaluations of f might be expensive. To ad- dress these limitations, BO is promising method that constructs a surrogate model approximate f and iteratively proposes potential points. In more detail, the core components are:
1. Surrogate model: BO methods typically construct surrogate approximation of using avail- able samples. We denote the surrogate model p(s]h: D,,), where D,, == {(h,, 8,)] are the n observed input-output pairs. Some commonly used models include the Gaussian Process (GP)
7 and random forests (SMAC) [] An alternative approach is the Tree Parzen Estimator (TPE)
[6] which uses two hierarchical processes ((x) and g(z) to model input distributions when the objective function is above or below a specified quantile т: p(h|s; s; D..) = I(h) if < т, gh).
2. Candidate point sampler: The sampler proposes a set of candidate points H == [h+ to query next. We denote the sampler (n)) where each candidate point is sampled indepen- dently h, ~ p(h)D,). For GPs, the candidate points are typically randomly sampled but then further optimized directly using the acquisition function. In SMAC, candidate points are sampled using a combination of random search and local search in the good regions found by the random forest. For TPE, the candidate points are sampled directly from the density of "good" points, g()).
3. Acquisition function: The acquisition function a H - E scores and selects the candidate points using the surrogate model. One of the most popular acquisition functions is expected improvement (EI): a(h) E []max(p(ss (shh) (hbest). 0)], where f(hees)) is the value of the best observation so far [1O]. The point that maximizes the acquisition function is selected as the next sample to evaluate h = arg max (R) a(h). Other popular acquisition functions include the probability of improvement 9 and upper confidence bound 770)
The BO process thus operates by first updating the surrogate model with existing data, and then sampling a set of promising candidate points. Using the surrogate model, the acquisition function scores each candidate point and selects the best point for evaluation. The new point and observed value are appended to available observations, and the cycle continues.
B GENERATIVE SURROGATE MODEL ( MOVED DOWN FROM MAIN PAPER]
Method. An alternative approach to surrogate modelling is to learn the generative process of inputs given the output, represented as p(h|s; D.). This approach is exemplified in TPE-based methods ( which constructs two hierarchical processes formulated as p(h] s; D.,) = I(h) if s I T else g(h). Here ((h) the generative model of good points, and 9(h) is the model for bad points. In this context, good points refer to those with score S т, where 7 defines the threshold of the top quantile of observed
Conversely, bad points correspond to scores s > T, or the bottom quantile of results. TPE evaluates each point using the acquisition function a(h) x ((h)/ which intuitively scores each point based on their likelihood of being good over bad. At first glance, it appears challenging to learn the two densities I(h) and gh) directly with an I.I.M.
Density ratio estimation. To address this, we employ Bayes' rule, transforming the density ratio estimation into probabilistic binary classification (71. In introducing TPE, [6] showed that the El of each point is defined as such:
ET(h) x 7 g(h) T(h) (1 7)
16

---

Published as a conference paper at ICLR 2024
In other words, to maximize EI, we would like h that has high probability under i(h) and a low probability under ghh). By simple application of Bayes rule, we can rewrite this as:
y
g(h) ((h)
(1 1 7))
p(h|s > T)
p(h|s T) (1 -7)
pls > =|h)p(s S +)
(1 -1)
p(s S =//)p(s > T)
p(s =1h)p(s <T)
-1
y
p(s S hhh)
p(s S 7(h) p(s S 7(h)
Thus, the expression is equivalently rewritten pp-s S r(h), which is proportional to p(s s T[h), i.e. the probability of h belonging to the good points, Intuitively, this gives us a scoring func- tion that scores each sample according to the probability of producing good objective values. Using this reformulation, we can now estimate the score a(h) through ICL, by obtaining the probabilis- tic classification p(s S 7 rlh). We recategorize the observed samples, such that 2 1(s; S T), meaning the label is 1 if the performance exceeds the desired threshold. As before, we trans- form the observed samples to text Din = {(hpt, ; obtaining K predictions from the LLM,, (2,, p())) LLAMBO(HE D..)), and computing empirical average to estimate a(h).
Experimental setup. We compare the performance of the proposed generative surrogate model vs two variants of TPE: TPE (Ind) which models each dimension indepen- dently, and TPE (Multi) which models joint multivariate densities [44]. We evaluate the scoring performance and regret attained by these surrogate models. To assess scor- ing performance, we report the correlation between esti- mated scores a(h) with ground-truth scores. Additionally, we calculate regret with respect to the point that was as- signed the highest score. As before, we evaluate each task when E [5, 10, 20, 30). and we test predictions against 20 unseen samples.
Correlation (t)
0.50
0.25
0.00
10
20
30
Regret (1)
0.4
0.2
Empirical insights. (1) Scoring performance. Figure (Top) visualizes the correlation between surrogate model predicted scores and ground truth objective values. We observe that LLAMBO achieves notably higher correla- tions, especially at n=5, where the TPE variants generated scores that are only weakly correlated with the ground truth performance. We note that increases, the cor- relation of baseline scores improves, but LLAMBO is still superior. (2) Regret (Bottom) examines the regret of the point picked from the 20 available points. We find that while all methods show higher regret at low sample sizes when levels of exploration are higher, LLAMBO quickly identifies good regions in the search space, leading to lower regret as n increases.
g
10
20
30
Number of Observed Points
TPE (Ind)
LLAMBO
TPE (Multi)
Figure model. (Top) score correlation and (Bottom) regret.
Generative surrogate
& Generative surrogate modeling via ICL predicts scores that are more highly correlated with ground-truth scores, leading to better identification of high-potential points.
C PROMPT DESIGNS
In this section, aim to shed light the prompt design process. We start by supplying the complete prompts used in LLAMBO in Appendix [C.]] before performing an ablation study in Ap- pendix C.2 to analyze the effects of different components of the prompt.
17

---

Published as a conference paper at ICLR 2024
C.1 COMPLETE PROMPTS
In this section, we include the complete prompts used for each component of BO, including:
1. Zero-shot prompts for warmstarting with No Context (Figure
Partial Context (Fig-
ure [[]; and > Full Context (Figure [I].
2. ICL prompts for discriminative surrogate model (Figure 13 and
generative surrogate
model (Figure [.
3. ICL. prompts for target value conditioned candidate sampling in Figure 15
Note that in all figures. { is used to indicate placeholders. Each prompt is constructed with the four
key components highlighted in Figure [and Section 2.2
<Model Card> describing the ML model being optimized:
<Data Card> providing information about the dataset;
Instructions task-specific guidelines on the format and requirements of the response;
Observations of current optimization trajectory.
You are assisting me with automated machine learning using {model). I'm exploring a subset of hyperparameters detailed as: {configurations, type, and ranges]. Please suggest (number of rec- ommendations) diverse yet effective configurations to initiate a Bayesian Optimization process for hyperparameter tuning. You mustn't 't include "None" in the configurations. Your response should include only list of dictionaries, where each dictionary describes one recommended configuration. Do not enumerate the dictionaries.
Figure 9: Prompt for warmstarting with No Context.
You are assisting me with automated machine learning using {model} for a {task} task. The [task} performance is measured using {metric]. The dataset has {number of samples } sam- ples with fnumber of features} total features, of which fnumber of continuous features] } are numerical and fnumber of categorical variables} are categorical. Class distribution is {class distribution J. I'm exploring a subset of hyperparameters detailed as: {configuration and type]. Please suggest {number of recommendations] diverse yet effective configurations initiate Bayesian Optimization process for hyperparameter tuning. You mustn't 't include 'None' in the configurations. Your response should include only list of dictionaries, where each dictionary describes one recommended configuration. Do not enumerate the dictionaries.
Figure 10: Prompt for warmstarting with Partial Context.
You are assisting me with automated machine learning using {model} for a {task) task. The [task} performance is measured using {metric}. The dataset has fnumber of samples} } sam- ples with {number of features} total features, of which {number of continuous features] are numerical and (number of categorical features) } are categorical. Class distribution is (class distribution). {statistical information] I'm exploring a subset of hyperparameters detailed as: fconfiguration and type]. J. Please suggest {number of recommenaation) diverse yet effective con- figurations to initiate a Bayesian Optimization process for hyperparameter tuning. You mustn't 't include 'None" in the configurations. Your response should include only a list of dictionaries, where each dictionary describes one recommended configuration. Do not enumerate the dictio- naries.
Figure 11: Prompt for warmstarting with Full Context.
Considering one-hot encoding for categorical features the total amount input's features of the random forest is (total of features after one hot encoding]. J. We are standarizing numerical values to have mean 0 and std 1. The Skewness of each feature is {skewness of each feature). The number of features that have strong correlation (defined > 0.5 or <-0.5) with the target fea- ture is {number of "strong correlations" with the target feature). Of the {number of pairwise feature relationships pairwise feature relationships, fnumber of "strongly correlated" pairwise features} } pairs of features are strongly correlated (>-0.5, <00.5).
Figure 12: Prompt of (statistical information) } used in Full Context.
18

---

Published as a conference paper at ICLR 2024
The following are examples of the performance of a {model} measured {metric} and the cor- responding model hyperparameter configurations. The model is evaluated on tabular {task] task containing {number of classes} classes. The tabular datasct contains {number of sam- ples} samples and {number of features } features (f (fnumber of categorical features categorical, (number of continuous features} numerical). Your response should only contain the predicted accuracy in the format ### performance ##..
Hyperparameter configuration: (configuration }}
Performance: {performance }}
Hyperparameter configuration: {configuration n}
Performance: {performance n}
Hyperparameter configuration: (configuration to predict performance)
Performance:
Figure 13: Prompt for discriminative surrogate model.
The following are examples of the performance of a {model} measured {metric} and the cor- responding model hyperparameter configurations. The model is evaluated tabular {task] task containing {number classes} classes. The tabular dataset contains {number of sam- ples} samples and fnumber of features) } features (fnumber of categorical features] categorical, (number of continuous features} } numerical). The performance classification uration is in the best best-performing 25.0% of all configurations, and 0 otherwise. Your response should only contain the predicted performance classification in the format ## performance clas- sification ##
if the config-
Hyperparameter configuration: (configuration }}
Classification: {classification }}
Hyperparameter configuration: (configuration n}
Classification: {classification n}
Hyperparameter configuration: (configuration to classify)
Classification:
Figure 14: Prompt for generative surrogate model.
The following are examples of the performance of {model} measured in {metric} and the cor- responding model hyperparameter configurations. The model evaluated on a tabular {task] } task containing (number of classes} classes. The tabular dataset contains {number of sam- ples} samples and {number of features} }or features (fnumber of categorical features} } categorical, {number of continuous features] numerical). The allowable ranges for the hyperparameters are: {configuration and type). Recommend a configuration that can achieve the target performance of (target score}. Do not recommend values at the minimum or maximum of allowable range, do not recommend rounded values. Recommend values with the highest possible precision, as requested by the allowed ranges. Your response must only contain the predicted configuration, in the format ## configuration ##.
Performance: {performance 1}
Hyperparameter configuration: configuration 1
Performance: {performance n}
Hyperparameter configuration: (configuration n}
Performance: {performance used to sample configuration) Hyperparameter configuration:
Figure 15: Prompt for candidate sampling.
19

---

Published as a conference paper at ICLR 2024
C.2 ABLATION STUDY
Our prompt design aligns with the framework outlined in Section components: (1) the description of the optimization problem, (2) the optimization history, and (3) explicit task instructions. To evaluate the influence of each component performance, conducted an ablation study with the following configurations:
2.2 encompassing three essential
LLAMBO: Represents the standard LLAMBO configuration employed in our experiments.
LLAMBO [No context]: This variant assesses the impact of the optimization problem description
(refer to component (1)) on performance. Specifically, it omits metadata about the underlying dataset from the prompts (see Figures [6]] 17 for details).
LLAMBO [No instructions] In this setting, we exclude additional, non-formatting-rlle in- structions from the prompts (refer to component (3)). This includes removing guidelines in candidate point samplers regarding the types of points be sampled. As such, the instructions that are maintained are purely to ensure adherence to the required format for regex processing (see
Figure 18
The following are examples of the performance of {model) measured in {metric} and the cor- responding model hyperparameter configurations. The model is evaluated on a tabular (task) task containng samples and
elasses, The tabular dutaset contains fentoren) eategerieal: numerieal)) Your response should only contain the predicted
features (fwembee
accuracy in the format ### performance ##..
Hyperparameter configuration: {configuration I}
Performance: {performance }}
Hyperparameter configuration: {configuration n}
Performance: {performance n}
Hyperparameter configuration: (configuration to predict performance)
Performance:
Figure 16: LLAMBO [No context]. Prompt for the discriminative surrogate model with parts of the optimization problem description removed. Note that in this setting, meta-data about the underlying datasets are provided. Strikethrough indicates the component that removed.
The following are examples of the performance of {model} measured in ( {metric} and the cor- responding model hyperparameter configurations. The model evaluated on a tabulur [rask)
task containnn fuumber of elusves) elasses. The-tabulat dataset contains (number of samples and fammeer of fentures) features (fmamber entegorient fenteres) categorieal; (aumberr- contieuous features) numerical). The allowable ranges for the hyperparameters {configuration and type]. Recommend a configuration that can achieve the target performance of [target score]. Do not recommend values at the minimum maximum of allowable range, do not recommend rounded values. Recommend values with the highest possible precision, as requested by the allowed ranges. Your response must only contain the predicted configuration, in the format ## configuration ##
Performance: {performance 1}
Hyperparameter configuration: (configuration 1}
Performance: (performance n}
Hyperparameter configuration: (configuration n}
Performance: {performance used to sample configuration) Hyperparameter configuration:
Figure 17: LLAMBO [No context]. Prompt for the candidate point sampler with parts of the opti- mization problem description removed. Note that in this setting, no meta-data about the underlying datasets provided. Strikethrough indicates the component that is removed.
20

---

Published as a conference paper at ICLR 2024
The following are examples of the performance of a {model} measured {metric} and the cor- responding model hyperparameter configurations. The model is evaluated on tabular {task] task containing {number of classes} classes. The tabular dataset contains {number of sam- ples} samples and {number of features) } features (fnumber of categorical features f categorical, fnumber of continuous features) numerical). The allowable ranges for the hyperparameters are: {configuration and type). Recommend configuration that can achieve the target performance of {target score]. J. Do not recommend values at the minimum or maximum of allowable range, de-net recommend rounded values, Recommend values with the highest pesesible preeision, requested by the allleallwwddddeeeo ranges- Your response must only contain the predicted configuration, in the format ## configuration ##.
Performance: pperformance }}
Hyperparameter configuration: (configuration 1}
Performance: {performance n}
Hyperparameter configuration: {configuration n}
Performance: (target score}
Hyperparameter configuration:
Figure 18:
LLAMBC [No instructions]. Prompt for the candidate point sampler with additional instructions removed. Strikethrough indicates the component that is removed.
Empirical analysis. We assessed the end-to-end per- formance of our ablation configurations on Bayesmark tasks containing RandomForest models. The out- comes are illustrated in Figure 19 Our findings re- veal that the standard LLAMBO configuration outper- forms other variants, underscoring the significance of each prompt component enhancing overall perfor- mance. Additionally, note that our ablated settings achieve competitive optimization performances when compared against the baselines. This is more notable on LLAMBO [No context), which demonstrated simi- lar optimization behavior without any meta-data about the underlying task. On one hand, this suggests the important role of meta-data in guiding the optimiza- tion process. For instance, specific hyperparameters better suited for large-p-smalle datasets can mitigate overfitting, and datasets with a higher proportion of categorical features may benefit from deeper decision trees. On the other hand, the robust performance of LLAMBO [No context], despite the lack of metadata, signifies the model's effectiveness beyond mere reliance data memorization or leakage, as operates without access to specific dataset information.
0.20
Avg Regnet
15
1V
0.0
T1Hads
LLAMBO NNo Iserartioo)
GP (forthwch)
SMAC2
5800pe (G2)
(TPR)
Figure 19: Ablation of prompt designs, eraged over 5 seeds.
Furthermore, investigations into the role of candidate generation instructions indicate worse per- formance for LLAMBO [No instructions). To understand this, we examined the acceptance rate of proposed points, defined as the proportion of points that both meet search space constraints and unique (not duplicates of existing or other proposed points). LLAMBO [No instructions] recorded an acceptance rate of 69. 20. + 0.79%. significantly lower than LLAMBO (91.60% + 0.45%) and LLAMBO [No context] (88.8% 8% + 0.39%). This reduced rate limits the effective pool of candidate points for evaluation and selection by the surrogate model. These findings underscore the impor- tance of detailed task instructions in enhancing the quality and efficiency of the candidate sampling process, ultimately contributing the overall optimization effectiveness.
D DETAIL OF EXPERIMENTAL PROCEDURES
In this section, we outline the benchmarks employed in our evaluations as well as the implementation details of method and considered baselines. To evaluate the performance of ILAMBO HPT,
21

---

Published as a conference paper at ICLR 2024
considered 25 built-in tasks from Bayesmark [1] (Appendix D.I and 24 built-in tasks from HPOBench [32] A.ppendixD.22). Additionally, we included three synthetic and three private datasets into Bayesmark, resulting additional 30 tasks (Appendix D.3). We used 5 tasks extracted from Bayesmark (all 5 datasets on RandomForest) to evaluate each component, and used all tasks for end-to-end evaluations.
D.1 BAYESMARK
We utilize Bayesmark 31 as continuous HPT benchmark We included the public datasets that came with the benchmark and 5 ML models, including RandomForest, SVM, DecisionTree, MLP, and AdaBoost. This makes a total of 25 tasks, with each task defined as the (dataset, model) pair. We execute all tasks using five different seeds for 25 trials, ensuring that all models share the same initialization for each seed. This approach guarantees consistency across results. For classification and regression tasks, the scoring function is accuracy and MSE, respectively.
Hyperparameter space. We follow the search space designated in Bayesmark, including the hy- perparameter type, space, and range (lower and upper bound). We use this specified search space in all baselines, performing the transformations before the optimization process. The search space for each ML model is summarized below, i.c. (hyperparam
[[space), [Lower.bound]
SVM [3d]: {C: [log, 1, 1e3], y: [log, le-4, le-3], tolerance: [log, le-5, le-1]}
DecisionTree [6d]: (max_depth: [linear, 1. 15], min.samples.split [logit, 0.01, 0.99], lcf [logit, 0.01. 0.49], min_weighh
wiin.weight.fraction...e. fraction [logit, 0.01. 0.49], max features. [logit, 0.01, 0.99], min impurity decrease: [linear, 0.0, 0.5]
RandomForest [6d]: fmax_depth. [linear, 1, 15]. min www.ssmmmlsspppiii [logit, 0.01, 0.99], min samples Jeaf: [logit, 0.01, 0.49], min Bein__eiggh__ttttee fraction leaf: [logit, 0.01, 0.49], max max_features:
[logit, 0.01, 0.99], min impurity decrease: [linear, 0.0, 0.5]
ML.P [8d]: (hidden layer sizes: [linear, 50, 200), alpha: [log, 1e-5, 1e1], batch size: [linear, 10, 250], learning_rate. init [log, le-5, le-1], power_t: [logit, 0.1, 0.9]. tol: [log, le-5, le-1], momentum: [logit, 0.001, 0.999], validation fraction: [logit, 0.1, 0.9]]
AdaBoost [2d]: (n estimators: [linear, 10, 100), learning rate: [log, le-4, lel]}
D.2 HPOBENCH
We included HPOBench, specifically the tabular benchmarks for computationally efficient evalu- ations (32]. We included all 8 OpenML datasets with pre-computed tabulated results, and 3 ML models, including XGBoost, RandomForest, and MLP. In total, we included 24 tasks. We execute all tasks using five different seeds for 25 trials, ensuring that all models share the same initialization for each seed. For all tasks, the scoring function is validation loss.
Hyperparameter space. The search spaces for all ML models, including their dimensionalities and search ranges, are provided below. The search spaces are discretized (see App D.3 in (32) for more details), allowing efficient tabular look-up operations for different configurations. For all baselines, the hyperparameters are treated as ordinal variables, following recommendations in the benchmark.
XGBoost [4d]: {colsample.bytres [0.1, 1.0], eta: [2e-10, I], max_depthh [1, 50), lambba [2e-10, 2e10]]
RandomForest [4d]: {max_deptte [1. 50], max [0.0, 1.0], min mampssmples..eal leaf: [1. 2]. min_samples_sge [2. 128]}
MLP [5d]: (alpha: [le-8, 1], batch_size: [4, 256), depth: [1, 3], learning.rate_ine init: [le-5, 1],
width: [16, 1024]]
D.3 PRIVATE AND SYNTHETIC DATASET
Additionally, we introduce 3 proprictary (SEER (72), MAGGIC (73). and CUTRACT (74) and 3 synthetic datasets. These are datasets for which the LLM would not have seen during pretraining,
15b788 3e..444ccaaa. F2338
22

---

Published as a conference paper at ICLR 2024
and thus used address any memorization concerns. The synthetic datasets were generated from the complex multimodal functions, specifically: Rosenbrock, Griewank, and KTablet (see [75] for full simulation parameters). We selected the input dimension as 15, where each dimension is uniformly sampled in the range of [0, 1]. Subsequently, we used the designated functions [75] to generate the corresponding output. All 6 datasets were introduced into Bayesmark, where the same set of ML models was evaluated (see Appendix D.I). leading to total of 30 tasks. The other aspects of the evaluation were identical, with the same seeding, trials, scoring functions, and search spaces.
D.4 IMPLEMENTATIONS
End-to-end LLAMBO. The end-to-end procedure iteratively performs three steps: (1) sample M candidate points (hm IM (2) evaluate M points using the surrogate model, i.e. p(s(fmm to ob- tain scores {a(hm) } }m according to an acquisition function. We use expected improvement (EI), a(hm) E[max (p(s]hm) [(hbest), 0)]. (3) select point with the highest score to evaluate next, h arg max mm-1 a(h).
Setting of LLAMBO. For our instantiation of LLAMBO, we sample M 20 candidate points, and set the exploration hyperparameter to a = -0.1. M = 20 is similar to the default number of candidates sampled by popular TPE implementations, including HyperOpt, and Optuna f We set 0.1 based on observations in Figure 5 as value that
the exploration hyperparameter to balanced exploration (diversity) and exploitation. For the surrogate model, we sample = 10 MC predictions to compute the empirical estimates. For experiments, we used gpt-3. 3. 5-tubb, version 0301 with default hyperparameters temperatus = 0.7 and top-p = 0.95.
Baselines. We select the following baselines:
We used the github. wt//io///ab/a httpsttaaeeemmelllleeeee ht////ollleee es/ generated/ wwtp:////w..ppptp.......iaeeeeee html Addition- http://aaaaappiillo/e////etettta
SKOpt (GP-based) 668
omiinie..
ally, we optimize the acquisition of candidates for better results, acq.optimizer = lbfgs. Version 0.9.0.
GP (Deep Kernel Learning)
We use the implementation of GPytorch htttps://occs Powocch_PPchhhhh_ oooee/ NN Integrat iinn _tn_ionnno//____ KISSGP SisssPPDeeep_e Bee_e__Dee__Keeee___ee
48
www.torch.al/e//staaae
CUDA. html in combination with the optimization
of acquisition function used in for better performance. (BoTorch version 0.8.5).
DNGO: We used public implementation
SMAC3 8]:
We used version 1.4.0. from htmmtttps://qqiieia http:/u...oommattom////////
Turbo [6)
We used the implementation found in https::/github.coo com
uber- www.ereeeeaaarreeeeee
НЕBО 77 We used the available pip version of HEBO bttps://pypp..orgprrroo
HE30/ Version 0.3.5.
Optuna 41. We consider the implementation offtttps://pttuaaco
We used the mul-
tivariate version, which was found to have better performance in [44]. Version 3.3.0.
TPE 77 We incorporated an additional implementation of TPE, essentially a heavily op-
timized variant of TPE designed to achieve competitive results hhttps://ithub..co
nabenabe0928/tpa
STO (39].
We used the official implementation provided by the authors
com/ homm/daazhhonxiaag////ee bnts
E ADDITIONAL RESULTS
E. ADDITIONAL WARMSTARTING RESULTS
As part of our investigation into warmstarting, we investigated the effect of varying the amount of
information provided the zero-shot prompts on the quality of warmstart initialization. Specifi-
uptuna.samplerse wttp:aammmmeerr..sssammm..a. |t
23

---

Published as a conference paper at ICLR 2024
Privade (TPE)
Public (TPE)
Private (GP)
Public (GP)
0.6
Avg Regre
Avg Regres
Avg Regges
Avg Regres
0.4
0.2
0.3
15
20
Triaee
Traale
Telale
Triak
Random
Sobal
HCube
Partad
Concest
No Context
Full Context
Figure 20: Fine-grained warmstarting results.
0.225
(Holbed))
GP DKL (BoTorch)
SIO
SMACH
Itggees
0.150
SKOPpt (GP)
0.125
Tecoo
Oponaa (TPE)
HERO
DNGO
ILAMRO (Ths)
TLAMBO (Gen)
0.030
0.025
10
20
Trials
Figure 21: End-to-end performance of LLAMBO (generative surrogate model) on Bayesmark. Average regret on public, private, and synthetic datasets.
cally, No Context, Partial Context and Full Context. We evaluate the effectiveness of warmstarting by evaluating the search performance GP and TPE under different initialization. In Figure 20 plot average regret across tasks for both BO models. We observe similar trends, that increasing the informativeness of the prompts led to improved search performance. In Figure we visualize the correlation matrices between sampled warmstarting points, observing higher correla- tions between points recommended by an LLM.
E.2 LLAMBO WITH GENERATIVE SURROGATE MODEL
Due to budgetary constraints, we did not evaluate LLAMBO with both proposed surrogate models in Section 7 To determine the stronger surrogate model for end-to-end evaluations, tested both methods in a constrained setting. This meant assessing both models on a single run across all 25 tasks within the public Bayesmark benchmark. The outcomes of this preliminary evaluation depicted in Figure[21] Our findings indicated that the LLAMBO with discriminative surrogate model outperformed its counterpart. Consequently, we opted for the discriminative surrogate model for comprehensive evaluation. However, it's worth noting that the LLAMBO using a generative surrogate model showcased competitive results when compared with baseline measures. Both approaches exhibited swift search and convergence in the initial trials, especially when n < 10. The generative instantiation was close second until the last few trials.
Drawing definitive conclusions from single seed can be challenging. However, we postulate that our model's 's generative version might exhibit sensitivity to the r hyperparameter, which dictates the boundary between good and bad points. A recent study inherent LLM biases in ICL by [64] underscored the majority label bias-a tendency of the model to favor the majority label in the given few-shot examples. This bias is intrinsically tied to generative surrogate model's 's design, where т determines the proportion of points labeled as good versus bad. We see a thorough examination of this potential bias, in-depth analysis of the hyperparameter's sensitivity, and the exploration of cias-correction techniques as key future research avenues to fully explore the generative surrogate model's capabilities.
24

---

Published as a conference paper at ICLR 2024
Public
Private + Synthetic
0.20
0.30
0.25
Avg Regrot
0.15
0.10
0.15
0.05
0.10
10
25
30
20
Trials
Trials
TPE
GP [BoTocch)
DKL (BoTorch)
STO
SMAC3
SKOpt (GP)
Tarbo
Optena (TPE)
HEBO
DNGO
LLAMRO
End-to-end performance LLAMBO on Bayesmark. (Leftt) Average regret on public datasets and (Right) private and synthetic datasets aggregated on all tasks.
Figure 22:
Table I: Average rank (4) achieved for different
Table 2:
Average rank (4) achieved for different ML methods on private and synthetic datasets.
ML methods on public datasets.
DTT
MI.P
SVM
ADA
MI.P
RF
SVM
ADA
5.12
5.34
4.85
3.98
GP (BoTorch)
6.72
5.62
5.94
6.06
4.74
GP (BeTorch)
6.70
6.18
7.08
7.10
5-45
DKL (BoTorch)
1.30
1.63
5.52
5.60
6.26
GP DKL (BoTorch)
4.78
4.25
4.60
SMAC3
3.82
6.42
5.40
5.66
6.02
SMAC3
5.12
5.33
5.13
6.15
5.40
SKOpt (GP)
3.66
4,40
4.90
4.84
5.12
SKOpt (GP)
4.83
4.98
4.52
2.32
4.95
Turbo
5.56
4,40
5.60
5.38
5.78
Turbo
5,80
5.07
5.77
4.58
Optuna (TPE)
6.58
5.34
5.48
5.08
5.08
Optuna (TPE)
4.98
5.75
5.37
5.12
5.52
HERO
4.88
5.10
6.50
DNGO
7.74
8.62
8.08
6.62
7.14
DNGO
7.83
8.17
9.12
7.32
7.62
LLAMBO
2.02
5.06
3.64
4.82
3.62
LLAMBO
3.35
4.10
2.08
4.88
6.45
E.3 ADDITIONAL RESULTS ON BAYESMARK
In this section, we include additional results to supplement our end-to-end evaluation of Bayesmark.
Additional baselines. We include additional baselines in Figure 22 including an optimized version
of TPE from (75). DNGO (a Bayesian neural network approach [24), STO (a neural network ap- proach (39]] Turbo 766 (a GP approach that identifies 'trusted regions' of the input space search for improvements), and HEBO (77). This presents a more comprehensive evaluation against wide array of BO approaches.
Search performance across ML models. In Tables I and 2 we compare the tuning performance of BO methods for different types of ML models. We show the average rank achieved by methods the end of each search. Although LLAMBO consistently demonstrates the best overall performance, exhibits model-dependent variability. Notably, LLAMBO excels in tuning Decision Tree and Ran- domForest across both public and private benchmarks. However, it does less well on SVMs. The underlying cause this remains speculative, but one plausible explanation is the inherent character- istics of the black-box function being optimized. This sensitivity and such nuances common to all BO techniques, given their sensitivities to black-box functions with different attributes (c.g. the choice of kemeel directly affects the effectiveness of GP for specific functions) [78). Exploring the characteristics of black-box functions where our method shines is a crucial avenue for future re- search and remains beyond the scope of our current study.
Individual task results. We plot individual task results by optimization metric (Figure 26), regret (Figure 27 and average rank of BO methods (Figure 28) (all results averaged over 5 runs).
25

---

Published as a conference paper at ICLR 2024
Random
0.12
GP DKL (Do Torch)
SKOgp (GP)
0.10
Optona (TPE)
Ang Regett
LLAMDO
0.08
0.08
0.04
10
15
20
25
Trials
Figure 23: End-to-end performance of LLAMBO (discriminative surrogate model) on HPOBench. Average regret on all tasks across five seeds.
E.4 ADDITIONAL RESULTS HPOBENCH
In this section, we include additional results on HPOBench to supplement our end-to-end evaluation. This included total of 24 tasks (where each task is a dataset-model pair, see Appendix D.2 for details). For each task, we executed seeded search. Figure 23 shows the average regret across all tasks, we note that LLAMBO achieves the best tuning performance. Additionally, we observe that, consistent with prior findings, LIAMBO excels in earlier stages of the search, when fewer observations are available.
Individual task results. We plot individual task results by optimization metric (Figure regret (Figure 30 and rank of LLAMBO and other BO methods (Figure 31
E.5
CLOCK TIME
We analyze the runtimes of various algorithms. Fig- illlutttssss the average clock time per iteration function of the number of observed points. It is important to note that the times recorded repre- sent solely the surrogate model's computation dura- tion, excluding any black-box query time. For con- text, all runtime measurements were conducted on an Intel i7- 170P (a consumer-grade laptop). Our observations reveal that LLAMBO incurs a higher av- erage clock time per iteration. However, it's essen- tial to highlight that this increased time is predom- inantly influenced by external factors such as in- ternet connectivity and API latency. Additionally, in black-box optimization scenarios, querying the black-box function is typically the primary compu- tational expense, overshadowing the time required for Bayesian Optimization (BO). Therefore, opti- mizing the search process to minimize the number of iterations performed, and consequently, the num- ber of black-box queries (i.e. sample efficiency), is arguably more significant than the time taken in each iteration of a BO algorithm.
Clock (soe) 3.
10
15
20
30
Number of observations
GP (BoTorch)
SKOpt (GP)
(BoTorch)
Optuna (TPE)
SMAC3
L.LAMRO
Figure 24: Average clock time (s) of BO al- gorithms.
26

---

Published as a conference paper at ICLR 2024
Tade digit-
Beakes
No Costoo
Pictiad Cuaret
Fei Chstod nemarae:
Aeurage capolitkes 223C
Averpp
Ansoao somatties 2234
(a) Task: DT on Digits.
dight
Ranker
No Cocted
Pactal Coreet
Fat Octtod
Aeeaag
(b) Task: RF on Digits.
Tale
DT
Banker
Xu Oottoo
Pactal Chaanl
Fii Outtod
(c) Task: DT on Wine.
Tral
Beakes Awcrag ounclikee 22NT
X6 Oatoo
Pactal Curazi
Fiil Custod
(d) Task: RF on Wine.
Baokes Average ounolikee 13
Pictad Cureei
Fii Cuettdd
(e) Task: DT on Breast.
Hankes
Pictidd Univer
Pii Custiod
Ameraee
Averyg
Aneaag
Task: RF Breast
Figure 25: Comparison of correlations between sampled initializations. Correlation matrix cal- culated on 50 initialization sampled for each task.
27

---

Published as a conference paper at ICLR 2024
20 Iewe
DV Inwel
VIP and lenel
<de idelow
abell
MiPllcs axed
601
H1.
DI Blwddeec
201 Eleid
TPE
CF (BuTand)
GF DKL (BcTlad))
STO
9K0y (CF)
Oykons (TPF)
BEDO
DSGO
LLAMBO
Figure 26: Bayesmark: individual HPT task results (task metrie). Evaluations according to task metrics, i.e. accuracy (1) for classification tasks, and negative MSE (1) for regression tasks, averaged over 5 runs.
28

---

Published as a conference paper at ICLR 2024
XP
MiPoog
DT Ailee
2035
2V32
KTAAI
DV ATtakl
Papl Klilder
KTalll
D2 Draved
HIP ol Draved
TPE
[B.Tecc)) SK0 (GP)
36L (B/Tied))
SSAAC
Thil
Optana (TPB)
DNGO
Figure 27: Bayesmark: individual HPT task results (regret). Evaluations according to normal- ized regret (L), averaged over 5 runs.
29

---

Published as a conference paper at ICLR 2024
10P 2.0000
2533 didlere
SVN
117
12
Boenboookk
DV Scbaie
Micheed Scaade
MtlP Conwnd
akOomal
TPS
(BdTedd)
CP DKL Reci))
STU
SMUCC
(TPP)
HEBO
DXCD
LLAYBO
Figure 28: Bayesmark: individual HPT task results (rank). Evaluations according to average
rank (L) during each search, averaged over 5 runs.
30

---

Published as a conference paper at ICLR 2024
pistraliin
oustmoiin
austallic
on blod, coandiilio
#1 NoorStorriiiii
xdh cransferi
02000
4.770
02703
4.00
300
cnalllg
500 colll
6.70
18
xobb lec
0107
D0 pionenso
35 plumeete
xgh pheneme
3.00
0.09
seganent
Xigh reginent
0.002
wallde
27 whidd
vellde
CP DRL (BoTudh) Optins (TP8)
SMLACH
S100p (GP)
Figure 29: HPOBench: individual HPT task results (task metric). Evaluations according task metrics, i.e. accuracy (t), averaged over runs.
31

---

Published as a conference paper at ICLR 2024
onstralion
cistrolan
xub outralio
100
103
2007
Jn blod tranecision
blodd Froncusica
xgb blod transheiic
1003
400
030
3000
6:15
300
dedii
585 coaii
4.12
K00
pionenze
3gb pheneme
0.2
6.19
0.8
6:0
segmennt
022
waidd
vddidd
xol walade
010
1007
100
CP DXL (BoTards) Optima (TPE)
S3EMC3
8800pt (CIP)
Figure 30: HPOBench: individual HPT task results (regret). Evaluations according to normal- ized regret (L), averaged over 5 runs.
32

---

Published as a conference paper at ICLR 2024
pistralian
austcaiion
on blod trandtiiion
28 Nood, Tressusoon
sgb baood, trensfusion
sgb crddi
n0 phoneme
1f ('ghoomee
XgD phocemn
sogmnen
pollde
25 vebicle
3go vebide
CP DKL (BoTond) Oplans (TPE)
SMAC
SROO (G22)
Figure 31: HPOBench: individual HPT task results (rank). Evaluations according average
rank (4), averaged over 5 runs.
33


Reviews:
[{"review_id": "ouKv1q2zGu", "reviewer": "Reviewer_ywEG", "summary": "This paper proposes a novel approach for enhancing Bayesian optimization using LLMs. The approach targets several sub problems in BO such as\n\n1) Selecting initial points for warm starting. LLMs being effective at transferring knowledge are can produce a more promising initial set when provided with the problem setup.\n2) Surrogate modeling, where the LLM is used to provide prediction and uncertainty estimates for a new design provided past observations.\n3) For sampling new candidates to observe.\n4) Finally all the steps are augmented for an end-to-end BO approach.\n\nExperimental results on benchmark datasets are promising. Traditional BO work have mainly focussed on black box optimization from scratch, and works on transfer learning are relatively recent. As such the work in this paper is interesting and strongly relevant to the BO community.", "questions": "It is not clear why an LLM should have any domain knowledge about hyper-parameter tuning to start with. When provided with past observations as part of the input prompt, it is true that the model may be able to generalize (#). However does the model have any additional knowledge about hyper-parameter tuning as a part of its training data?\n\nIs it obvious that comment (#) is true? What is the mechanism behind the LLM being able to parse numbers and compare them to get a decent understanding of the loss domain?", "limitations": "", "rating": 8, "confidence": 3, "soundness": 4, "presentation": 4, "contribution": 4, "strengths": "To summarize, this paper successfully demonstrates the utility of LLMs in Bayesian optimization. The paper makes several strong contributions\n- The paper demonstrates the utility of the LLM in all stages of BO from warm starting to candidate selection, and effectively utilizes the power of LLMs in knowledge transfer to novel problems.\n- Experimental results show improved performance on several benchmark problems. Experimental evaluation is reasonably extensive including several public and private datasets.", "weaknesses": "An obvious weakness of this method is that it has only been evaluated on relatively simple benchmark datasets. The great utilization of this approach would be to select sophisticated neural architectures for novel datasets. However, it is understandable that this may be out of scope for the current work.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "psyQbS7QM4", "reviewer": "Reviewer_x2Jx", "summary": "This paper introduces LLAMBO, an interesting approach that integrates large language models (LLMs) into Bayesian optimization (BO). The authors highlight the challenges of efficiently balancing exploration and exploitation in BO and propose LLAMBO as a solution to enhance various components of model-based BO. LLAMBO leverages the contextual understanding, few-shot learning proficiency, and domain knowledge of LLMs to improve surrogate modeling, candidate sampling, and zero-shot warm-starting. The authors empirically validate LLAMBO's effectiveness in hyperparameter tuning, demonstrating strong performance across diverse benchmarks.", "questions": "1. In page 8, could the author explain why a negative alpha will help improve the average regret? Intuitively, a negative alpha indicates that the sampling candidates usually perform no better than the best one when compared with the existing candidates, which therefore indicates that no improvement should be made through a negative alpha.\n2. What's the dimension of the empirical experiments that have been conducted in this paper? Any high-dimensional ones?", "limitations": "", "rating": 8, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. Generally, this paper is well written and easy to follow.\n2. This paper has well shown the feasibility of introducing LLM into BO for further performance improvement through extensive empirical experiments.\n3. This paper has conducted extensive experiments to show how and why the performance is improved.", "weaknesses": "1. The proposed method may failed to deal with high dimensional optimization problems due to the limited token of LLM.\n2. This paper mainly compares with the LLAMBO with standard BO algorithms whereas many neural network-based BO algorithms have been developed recently to improve the modeling of standard BO algorithms, e.g., [R1]. This paper may also compare with it to further support the advantages of using LLM for BO.\n\n[R1] Dai, Z., Shu, Y., Low, B. K. H., & Jaillet, P. (2022). Sample-then-optimize batch neural thompson sampling. Advances in Neural Information Processing Systems, 35, 23331-23344.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "c6WViH5zij", "reviewer": "Reviewer_LVtK", "summary": "This paper devises a method to enhance Bayesian optimization using a large language model.  By employing GPT-3.5, the authors investigate GPT-3.5's abilities to warm-start Bayesian optimization, model a surrogate function, and sample query points, and eventually conduct the entire process of Bayesian optimization.  In this paper various scenarios are conducted using in-context learning and prompt engineering, and some important messages discovered by the authors are delivered.", "questions": "* In prompt examples, do colors have some consistent indication?  For example, texts in orange have some meaning?  I think you can make consistency across examples and it can help understand the prompt examples.\n* Why is no context for warm-starting Bayesian optimization better than random, sobol, or hcube?  The results of no context should be similar to the random initialization methods.\n* In discriminative surrogate modeling, a method of LLAMBO utilizes both Monte Carlo sampling and shuffling of examples, or it only uses the shuffling?\n* Could you elaborate how $\\alpha$ is used in candidate point sampling?\n* Could you explain the details of end-to-end demonstration of LLAMBO?  I think it is missing in the paper including the appendices.\n* As I mentioned earlier, I would like to see the design process of prompts.", "limitations": "", "rating": 8, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "* It solves an interesting topic in Bayesian optimization or active search.  Many optimization researchers were curious about the topic handled in this work.\n* Many questions on how it works and which factor makes it work are answered and discussed.\n* Extensive analyses are provided.\n* Paper is generally well-written.", "weaknesses": "* I think this is timely work, but I am not sure that it can be presented at ICLR, which is a conference that focuses on representation *learning*.  This work did not do learning explicitly.  I am leaning towards a positive side, but it should be carefully discussed with authors, reviewers, and area chairs.  I think that this paper is more suitable for a natural language processing conference such as ACL, EMNLP, or others.\n* Standard deviation (or confidence interval) is not reported for every experiment.  I think this is an important component for this kind of studies.  If some results are statistically meaningless, the analyses are not much meaningful.\n* I am curious about how the authors design prompt templates.  Since I tried similar experiments, I could understand why some sentences are included.  However, the analysis on prompts and prompt designs, and potentially failure cases, should be included in the paper.  For example, if you do not include \"Do not recommend values at the minimum or maximum of allowable range, do not recommend rounded values\" in the prompt, what happens?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "iOtyWsSxeJ", "reviewer": "Reviewer_8Jea", "summary": "The authors present the new approach LLAMBO, which integrates large language models into Bayesian optimization for the case of hyperparameter optimization. The integration is done by translating knowledge about the problem, algorithm, and optimization history into natural language prompts. The modular approach comprises a warmstarting component, a candidate sampler, and a surrogate model. Two alternatives for the surrogate model are contrasted: a generative and a discriminative variant. The authors claim strong empirical performance to be shown in experimental evaluations. Each component is evaluated separately, and an evaluation of using all components together for an end-to-end approach in comparison to existing methods is carried out additionally.", "questions": "1. How were the Hyperparameters of LLAMBO selected?\n1. How were the hyperparameters of the other optimizers selected?\n1. Which versions of the libraries for the other optimizers were used?\n1. Which exact version of GPT-3.5 was used?\n1. In Figure 7, for three of the four plots, the random approach does not seem to do anything, which seems questionable. Could you elaborate on why this might be the case?\n1. Why was the end-to-end variant of LLAMBO not evaluated with its own warmstarting component?\n1. In Figure 3, SMAC's random forest surrogate seems to beat the GP-based surrogate model, which based on previously published results seems unlikely given that we talk about a small-dimensional, continuous space. Could you elaborate on why this might be the case?\n1. In Figure 8, SMAC seems to do nothing for the first few iterations, which, especially in comparison to the previous evaluation of the surrogates (e.g. vs GP) seems strange, could you elaborate on why this might be the case?\n1. The performance of HEBO looks surprisingly bad compared to SMAC based on previously published results, could you elaborate on why this might be the case?", "limitations": "", "rating": 8, "confidence": 5, "soundness": 3, "presentation": 4, "contribution": 4, "strengths": "Novelty & Significance: To the best of my knowledge, the approach is the first of its kind to integrate LLMs into the process of HPO/BO in this all-encompassing manner and could be interesting for the community to build upon.\n\nQuality: In fact, the authors discuss many very interesting approaches for how to combine LLMs and BO. Eventually, they show that by combining all these approaches, they get a very strong system overall. Although I have some concerns regarding some details, the wealth of ideas in this paper and the corresponding experiments are impressive.\n\nIn the limited evaluation, the approach shows very promising results.\n\nClarity: The paper is written up nicely and illustrated with many figures and plots, making it relatively accessible.", "weaknesses": "### Approach\nMy main criticism is that the authors used an LLM that models data as a sequence of tokens. However, when passing HPO data to an LLM, they face the problem that this data has no sequential structure (see Section 4.1). They get around by permutating the data and thus even derive some kind of uncertainty. However, this seems flawed to me. The authors mention the SMAC method which is built on a random forest. Although random forests are not great for deriving uncertainties, the bootstrapping of random forests is at least statistically motivated. I missed any good argument as to why non-sequential data should be fed into a sequential model and then applying a hack trying to fix it again.\n\nFurthermore, it is well known that the strength of LLMs is based on good prompts. Therefore, I would actually expect some kind of insights and ablation studies on how to do the prompts for this problem. Reading the exemplary prompt templates, I strongly wonder whether the authors came up with these prompts in their very first trial. \n\n### Clarity\n\nThe paper is very dense and includes many nice results. However, the main paper (without appendix) should still be self-contained. However, the authors decided to move the discussion of related work into the appendix. I oppose that decision and deem it very important to have a discussion of related work in the main body of the paper s.t. readers can very well understand how to situate the paper. As a concrete proposal, I recommend moving the generative surrogate model into the appendix since it does not perform better than the discriminative model anyway.\n\n### Experiments\n\nFirst of all, all papers using closed-sourced GPT models have an inherent flaw: We know nothing about how these models are trained exactly, e.g., training data. Therefore, we lack any scientific understanding of how to use them. The authors tried to get around this by using some private and artificial datasets, but how does this relate to the training distribution used for the GPT model? We cannot make any real claim regarding meta-data leakage. \n\nFurthermore, since GPT-3.5 is not publically available and might even be updated from time to time, the chance of reproducibility of these results is more or less not given at all. I strongly recommend using (at least) publicly available LLMs such as Llama 2.\nThere are many further doubts I will raise as questions below. These might translate to direct weaknesses if not properly answered in the rebuttal. \n\nSome answers could relate to the use of Bayesmark. This benchmark library includes many rather simple HPO benchmarks, i.e., low-dimensional, continuous spaces and traditional ML models. Given state of the art in HPO, I would not consider these reasonable benchmarks anymore. In fact, quite some of the results in this paper conflict with insights into comparing HPO tools (see questions). Therefore, I suspect that results might look very different, if the authors would have used more challenging benchmarks (e.g, from HPOBench).\n\nThere are several further points that diminish the strength of the presented experimental results and undermine soundness:\n\n* All plots use iterations instead of wall-time on the x-axis, which does not allow the reader to draw any conclusions on the overhead incurred through querying the LLM, which might negate any benefits incurred by its inclusion depending on the benchmark.\n* No uncertainty is shown in any plot, even though results are averaged over several runs/ seeds.\n* Especially in the end-to-end demonstration, the number of iterations chosen is very small, and it would be interesting to see how the curves continue.\n* No fully random baselines were provided (warmstarting and end-to-end).\n* There are some results that seem contradictory to previously observed behaviors of approaches (see questions)\n* Accessibility: Often, only colors indicate which curves belong to which approach, and colors like red and green are mixed.\n\n### Minor Points\n* P. 15 (Appendix): 2. Candidate Point Sampler H, is \"]\" intentional?\n* In Appendix E, Figure 16 the description hints to two graphs but only one is shown.\n* In Figure 8, all curves start at the same point in iteration 0. This is a bit strange in comparison to the warmstarting, where there are clearly different starts depending on the methods. While this is probably due to all methods using the same warmstarting point, it seems puzzling to not include this at the start of the graph or make it clear from the description.\n* In the plots, when regret or performance over time (iterations) is shown, the functions should be step functions since there are no interpolations in-between possible.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (poster)