Adaptive Kernel Design for Bayesian Optimization Is a Piece of CAKE with LLMs
Authors: Richard Cornelius Suwandi
Feng Yin
Juntao Wang
Renjie Li
Tsung-Hui Chang
Sergios Theodoridis
Venue: NeurIPS 2025 poster ()
Keywords: Kernel design
Gaussian processes
Bayesian optimization
large language models
Primary area: probabilistic_methods
TL;DR: We introduce an adaptive kernel design method that leverages LLMs as genetic operators to dynamically evolve Gaussian process (GP) kernels during Bayesian optimization (BO)

Abstract: The efficiency of Bayesian optimization (BO) relies heavily on the choice of the Gaussian process (GP) kernel, which plays a central role in balancing exploration and exploitation under limited evaluation budgets. Traditional BO methods often rely on fixed or heuristic kernel selection strategies, which can result in slow convergence or suboptimal solutions when the chosen kernel is poorly suited to the underlying objective function. To address this limitation, we propose a freshly-baked Context-Aware Kernel Evolution (CAKE) to enhance BO with large language models (LLMs). Concretely, CAKE leverages LLMs as the crossover and mutation operators to adaptively generate and refine GP kernels based on the observed data throughout the optimization process. To maximize the power of CAKE, we further propose BIC-Acquisition Kernel Ranking (BAKER) to select the most effective kernel through balancing the model fit measured by the Bayesian information criterion (BIC) with the expected improvement at each iteration of BO. Extensive experiments demonstrate that our fresh CAKE-based BO method consistently outperforms established baselines across a range of real-world tasks, including hyperparameter optimization, controller tuning, and photonic chip design. Our code is publicly available at https://github.com/richardcsuwandi/cake.

Adaptive Kernel Design for Bayesian Optimization Is a Piece of CAKE with LLMs
Richard Cornelius Suwandi' Feng Yin Juntao Wang'
Renjie Li Tsung- Boin--Huidnn Chang' Sergios Theodoridis'
School of Artificial Intelligence (SAI), The Chinese University of Hong Kong. Shenzhen
2 School of Science and Engineering (SSE), The Chinese University of Hong Kong, Shenzhen
HHolonyak Micro & Nanotechnology Lab, University of Illinois at Urbana-Champaign
HEERON CoE for Robotics and AI-ATHENA R.C., Greece frichardsuwandi, juntaowang) @link. cuhk edu. edu, stheodor@di. ura.gr
fyinfeng, changtsunghui) Herraunnggguuuuuukkeeee &cuhk. edu.
Abstract
The efficiency of Bayesian optimization (BO) relies heavily the choice of the Gaussian process (GP) kernel, which plays a central role in balancing exploration and exploitation under limited evaluation budgets. Traditional BO methods often rely on fixed or heuristic kernel selection strategies, which can result in slow convergence suboptimal solutions when the chosen kernel is poorly suited the underlying objective function. To address this limitation, we propose a freshly- baked Context-Aware Kernel Evolution (CAKE) enhance BO with large language models (LLMs). Concretely, CAKE leverages LLMs as the crossover and mutation operators to adaptively generate and refine GP kernels based on the observed data throughout the optimization process. To maximize the power of CAKE, we further propose BIC-Acquisition Kernel Ranking (BAKER) to select the most effective kernel through balancing the model fit measured by the Bayesian information criterion (BIC) with the expected improvement each iteration of BO. Extensive experiments demonstrate that our fresh CAKE-based BO method consistently outperforms established baselines across range of real-world tasks, including hyperparameter optimization, controller tuning, and photonic chip design. Our code is publicly available at htttps /aattps://gitee /ttp//ggtttt.tt/// cwwmrichaadcsssaaciiiccccccaacce cake
Introduction
Many important scientific and engineering problems require optimizing objective functions that are noisy and expensive to evaluate. These objective functions often lack closed-form expressions, let alone gradient information, making optimization particularly difficult [50]. Nonetheless, Bayesian optimization (BO) has shown remarkable success optimizing such functions, due to its ability to operate on limited data and incorporate prior knowledge to guide the optimization process 17 In the past couple of decades, BO has been used for diverse tasks ranging from tuning hyperparameters in machine learning
to designing policies in robotics [ 33 and recommending new molecules The main idea behind BO is to first construct a surrogate model, typically using a Gaussian process (GP) (36, to represent the prior belief about the objective function. Then, by conditioning on the observations and the prior, the posterior is calculated using Bayes' rule to reflect the updated belief about the objective function. Based this posterior, an acquisition function further used to determine the next promising query positions while balancing exploration (i.e., moving to regions with high uncertainty) and exploitation (i.e., moving regions with high expected value).
43 29
in drug discovery 25
"Corresponding author.
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

Initial Population
Mating Selection
Mating
Crossover
Mutation
PER
LIN
M
LIN
4
SE
LIN SE
RQ
M3
M5
LLM
LLM
"Combine the parent kernels to improve the model fit
"Modify the kernel to
Final Population
End
Way LIN PER
improve the
model fit
LIN SB
Evaluation
Survivor Selection
www. wWw.
LIN
SE
LIN PER 1
LIN
LIN PER
LIN + PER
SE x PER
Fitness Calculator
1
0.53
0.71
Figure 1: Overview of CAKE. Starting with initial population of kernels, the LLM acts crossover and mutation operators, proposing new kernels based on the given prompts. The proposed kernels are then evaluated using fitness calculator, and the fittest ones advance to the next generation.
Although the past decades have witnessed rapid development of BO, much of the focus has been drawn on designing novel acquisition functions [ [. In contrast, the challenge of appropriately choosing the surrogate model has received comparatively less attention [40]. In the context of GPs, most off-the-shelf BO methods simply use general-purpose kernels, such as the squared exponential kernel or Matérn-5/2 kernel 43 While convenient, this one-size- sillssil--itt-ll approach may introduce bias that can negatively impact the sampling of potential solutions during optimization [15], especially when the kernel's s assumptions do not align with the statistical properties of the underlying objective function (37] It has also been studied that with a poor choice of the kernel, BO may converge very slowly, especially when optimizing complex functions in moderate-ter -o-hhhg dimensional spaces [16]. These considerations underscore the need for a more sophisticated kernel design in BO.
While the pursuit for automatic kernel design is not new 12 54], existing approaches might not be straightforward in the setting of BO. In BO, the objective function is typically expensive to evaluate, limiting the number of evaluations we can perform and leaving us with a small number of observations to tune the kernel. Moreover, since the functional form is generally unknown and the gradient information is unavailable, it becomes infeasible to apply traditional optimization-bent based kernel selection methods, such few-shot learning setting, where swift learning and generalization from limited data are required. Notably, these challenges align with the strengths of large language models (LLMs) [52], which excel generalizing from few-shot samples [8), thus enabling efficient exploration with limited data. The in-context learning capability of LLMs also acts as implicit Bayesian inference [56] 20), allowing them to encode prior knowledge about the optimization task, search space, and other relevant information. LLMs ability in performing complex reasoning further enhances their capacity to process contextual information and improve search performance [57]. Moreover, LLMs are also pre- trained on massive internet data, which potentially contains transferable domain knowledge applicable to various optimization tasks 29) Given these insights, we aim to investigate the following question: "Can LLMs, with their encoded knowledge and few-shot prompting, help to adaptively evolve kernel structures based on the observed data, thereby improving the BO performance?"
19. its core, these challenges can be framed within the
Contributions. In this paper, we introduce Context-Aware Kernel Evolution (CAKE) to enable adaptive BO using LLMs. Unlike conventional BO setups, which often rely on fixed kernels or heuristic selection strategies, CAKE leverages the in-context learning capabilities of L.MMs to iteratively generate and refine expressive kernel structures based the acquired data during the

---

optimization process (see Figure T. CAKE is guided by few-shot prompting and operates entirely in-context, without requiring fine-tuning or parameter updates to the LLM. To further complement CAKE, we propose BIC-Acquisition Kernel Ranking (BAKER) to select the most effective kernel through balancing the model fit measured by the Bayesian information criterion (BIC) with the expected improvement at each iteration of BO. We demonstrate through extensive experiments that our proposed method consistently outperforms established baselines across a range of real-world optimization tasks, including optimizing the hyperparameters of machine learning models, tuning controllers for dynamic environments, and designing photonic chips with optimal configurations.
Preliminaries
We first review some key concepts in BO and the so-called kernel grammar, which serve as the bedrock for this paper.
Bayesian optimization. Consider optimizing a "black-box" objective function f X C Rd + R. where the function evaluations are noisy, expensive, and the gradients are not available. Bayesian optimization (BO) addresses these challenges by employing a probabilistic surrogate model g. typically using a Gaussian processes (GP) (36,, to approximate I on the fly [7]. At each iteration t, the calibration of the posterior distribution p(g: (D:) given the previous observations 'D2 {(x,, yr b= informs where to explore and exploit in X'. Then, based on p(g: Dt), an acquisition function a X + R defines a policy to choose the next best point evaluate. After each evaluation, the surrogate model y is refined to reflect the updated belief about f. We refer the readers to Appendix
A. for a thorough treatment of BO with GPs.
Kernel grammar. The kernel grammar introduced by Duvenaud et al. [12 defines a comprehensive and flexible space of kernels. Such space exploits the closure properties of kernel functions under addition and multiplication, which ensures that: if k1 (x, x') and (2 (x,x') x') are valid kernels, then both ky (x, x') + kg (x, x') and h (x, x') x kg (x, x') are also valid kernels 42 Starting from set of base kernels, such as the squared exponential (SE), linear (LIN), and periodic (PER) kernels, one can construct more expressive kernels by combining such base kernels via addition and multiplication operators. For instance, LIN + PER kernel can capture periodic structure with a linear trend and SE x PER kernel can capture locally periodic components. Let denote a base kernel and S denote a subexpression. For example, in the expression LIN + (PER x SE), the term PER x SE a subexpression. The entire kernel space described by all kernels that can be constructed by adding base kernel to subexpression S + S + B, multiplying subexpression with base kernel + S x B, and replacing a base kernel with another base kernel B + B.
3 Context-Aware Kernel Evolution
The main motivation behind our method is to refine the kernel choice at each iteration before determining the next query point. To this end, we introduce Context-Aware Kernel Evolution (CAKE), which leverages LLMs genetic operators adaptively construct kernels based on the data obtained on the fly. We summarize the complete procedure of CAKE in Algorithm I
Conditioning the LLM. We begin by randomly sampling points from the input space X' to initialize the observations D {(x,, | These observations are then used as few-shot samples to prompt the LLM. The prompt is designed based on the concept of conditioning on high performance, as suggested by (0). Specifically, we start the prompt with a statement, "You are an expert in machine learning, specializing in Gaussian processes", ", to simulate the reasoning of human expert in the field. It has also been shown that chain-of -hhhught thought reasoning, or generating intermediate reasoning steps, can improve the performance of LLMs [53] 24). Motivated by this, we instruct the LLM to analyze the provided observations and identify patterns that can be represented by kernel functions, before proposing the kernels at each iteration. The designed system prompt is shown in Figure
Initializing the population. We draw some inspiration from the genetic algorithm 22 where we maintain population of candidates (kernels) throughout the optimization process. We define K as our population, and for each kernel E K, we measure its filness using the Bayesian Information Criterion (BIC) [39 The BIC is widely used metric for model selection that measures the trade-off between model fit and model complexity [46). It can also be viewed as an approximation of the Laplace method for estimating the marginal likelihood [35]. We provide more detailed discussions
3

---

System Prompt
You expert in machine leamning, specializing in Gaussian processes. Here the observations we have collected so far: {observations). Please analyze these observations to identify patterns in the data that can be captured [base_ kernels), and combine these kernels using the following operators: {operators). Your goal construct kernel expression that hest explains the observed data. The kemeel will be evaluated using a fitness score normalized between [0, II. where higher values indicate better to the data.
kernel function. You cam use any of the following base kernlls:
Figure 2: The designed system prompt. O indicate placeholders.
Crossover Prompt
Mutation Prompt
You are given parent kernels and their fitness scores: [kerneli> ((fitness1)), [kernel2) ((fitness2)). Please propose new kernel that has potentially higher fitness score. You may combine the parent kernels us- ing any of the operators from: {operatora). Briefly explain your reasoning behind pro- posed kerel..
You are given a kernel and its fitness score: [kernel} (ffitness)). Please propose new kernel that has potentially higher fit- ness score. You may replace base kernel in the current expression with another base ker- nel from the explain your reasoning behind the proposed kernel.
@base_kernel Briefly
(a) Crossover prompt.
(b) Mutation prompt.
Figure 3: Prompts for evolving the kernels via crossover and mutation.
on BIC and model selection for GPs in Appendix A.2 To ensure the fitness score is consistent across different tasks, normalize to the range [0, 1].
Proposing the kernels. We consider a generalized notion of the kernel grammar 7 which involves set of base kernels (k,.... . } and set of operators .Ti...., where r, E N. Each operator T; x K. K. for j = 1, .....,,, a closed operator (e.g., addition, multiplication, convolution, composition, affine transformation) on the space of kernels XC. Based this, we can define the kernel grammar space recursively as follows: 1Kg := (kI....,k, k and Kj == (k) K2) k k2 E Ki-1,j l.....(} K, for E N. By leveraging this kernel grammar space, we can leverage the LLM as genetic operators to propose kernels using the following operations:
Crossover: We perform ne crossover operations. For each crossover, we sample two parent kernels k1, k2 from K with probability proportional to their fitness. We then prompt the LLM to propose a new kernel Ke by applying an operator on the parent kernels.
2. Mutation: With probability Pm. perform mutation operation. We select the fittest kernel ky from K and prompt the LLM to suggest a new kernel km. by replacing one the base kernels in by with another base kernel.
The sample prompts for the crossover and mutation operations are shown in Figure 3 In the prompts, we also ask the LLM to report its reasoning behind the proposed kernels (see Appendix C.3 for a sample response). This serves by the LLM. The proposed kernels from the two operations are added to K, and their fitnesses are measured. Then, we select the top n. fittest kernels to form the next generation of K.
samity check, enabling to verify and interpret the choices made
Choosing the next query point. In our experiments, we observed that some kemels may promise good fit, but the actual improvement from the query points they propose is not substantial as expected, For this reason, we propose the BIC-Acquisition Kernel Ranking (BAKER) to jointly rank kernels based on both their model fit and their potential to yield high-utility query points. We first assign weights to each kernel K. based on its BIC: wy where BIC is the BIC value of the GP model associated with kernel k. We denote the acquisition
EveR exp(-BIC.)

---

Algorithm Context-Aware Kernel Evolution (CAKE)
Require: Budget T, number of crossovers The mutation probability Pm. population size np
1: Randomly sample n points form the initial observations D {(x,, )=)
2: Initialize the kernel population with the set of base kernels
3: for 1 to T do
4: Update system prompt with D (see Fig. 2
5: for
nee do 6: Sample two parent kernels k1, l2 from K.
7: Generate new kernel ke via crossover (see Fig. 3ap
8:
end for
9:
if r) < Pm then
10:
Select the fittest kernel in K Generate new kernel ki via mutation (see Fig. 3b)
11:
12:
end if
13:
Evaluate the fitnesses and keep the top-np kernels in K
14:
Choose the most effective kernel k via BAKER (see Eq.
15:
Obtain the next point x Xt, and evaluate Y: = f (xt)
16:
Update the observations D < D U ((x,, V:))
17:
end for
function as (x;; D, k), which quantifies the utility of evaluating a candidate point x under the model that kernel is is being used, given the current observations D. Based on this, BAKER computes weighted acquisition value for each kernel and selects the kernel k* that maximizes this value, i.e.,
h = arg max WI.o(Xt,13 (D, k), kEk
(1)
where X, denotes the candidate query point proposed by kernel à at iteration t. BAKER allows us to balance the kernel's 's ability to fit the data (as indicated by wx) with the expected improvement at the proposed query point (as measured by a). Once k' is selected, we use the corresponding kernel to obtain the next query point X, ,*, evaluate y/2 f(x),, and update the observations as D - D {(x,, }. This iterative process continues until a predefined budget T is exhausted.
4 Related Work
Expressive kernel design. Several methods have been developed to construct more expressive kernels beyond manual composition of base kernels. One such method involved multiple kernel learning techniques 4 19). which aim to identify the optimal kernel configuration by optimizing a linear nonlinear combination of base kernels. However, these methods restrict the kernel space and require prior specification of the kernel hyperparameters. Another approach involved searching for the optimal kernel structure across space of kernels [12], but since the space is infinite, efficiently navigating this space demands modeling expertise. Other works focus on designing flexible kernel or integrating GPs with deep neural networks (55). While powerful, these approaches either assume stationarity or require complex inference techniques. In contrast, our method is based on the kernel grammar and in-context learning via LLMs, which offers flexible yet computationally feasible approach.
families via spectral approximations 26 54
Surrogate modeling in BO. When using GPs the surrogate model in BO, the kernel is typically selected a priori based on an expert's knowledge concerning the problem at hand. Unfortunately. there is prior knowledge available, most BO methods simply default kernels such the SE kernel or the Matérn-5/2 kernel 43 While this seems reasonable, it has been reported that with poor overly general choices of the kernel, BO may converge very slowly [16]. For this reason, deep GPs have been proposed to help model non-stationary behaviors [21]. but at the cost of increased computational complexity. Other works explore adaptive kernel strategies, such as using discrete mixtures of GPs [18), maintaining parallel GPs with different kernels [37], or using ensembles of GP B1 Recent works have also showed great potential in using LLMs for surrogate modeling in
We use the expected improvement (EI) as our default acquisition function, normalized to (0, 1] to ensure comparability across different kernels.
5

---

BO 29 II 59.. We extend this line of research by using LLMs to automatically generate and refine kernels during the optimization process, enabling new-fashioned adaptive kernel design.
LLMs as genetic operators. As the model size and amount of training data increase, LLMs exhibit emergent abilities that significantly improve their performance across diverse tasks 52 Inspired by these abilities, recent works have explored using LLMs as genetic operators for generating code [4], assisting robot simulations [27], and designing neural network architectures [10]. To the best
of our knowledge, the current work is the first to use LLMs as genetic operators for constructing
adaptive and expressive GP kernel design for BO. Compared to the other transformer- based methods
41 our method can be applied entirely in- in-oonext and does not require any fine-tuning.
5 Experiments
To evaluate the performance of our proposed method, we test it against several baselines across a set of real-world optimization tasks with varying characteristics, including diverse optimization landscapes, dynamic environments, and multi-objective settings.
Setup. Our experiments were conducted using the software package BoTorch [6 and we used the OpenAl's
expected improvement (EI) as our default acquisition function. For the LLM,, gpt-4o-mini model as it offers an excellent balance between API cost affordability, fast inference speed, and intelligence for implementation. We define {SE, PER, LIN, RQ, M3, M5) our base kernels and {+, x) as our operators. Moreover, we set the number of crossovers ne = 5, mutation probability Pm 0.7, and population size n 10. To facilitate reproducibility, our code is available online at https: //tp://gttbbb com coom/richaaee chw.dsuuanddcdddii...... /ake, The shaded regions in all figures represent the standard error over independent trials. In the interest of space, more experimental details can be found in Appendix B, and additional results are provided in Appendix [C
Baselines. We compare our proposed method against the following established baselines:
Fixed: Default method BO, where we the kernel throughout the optimization process.
Adaptive: An adaptive kernel selection method proposed in [37] We employ three different selection criteria to adaptively change the kernel: Random, Utility, and BIC.
Deep GP: Uses deep GP (DGP) as the surrogate model, implemented through a functional
composition of stationary GPs (21]
Ensemble GP: Uses an ensemble of GPs (EGP) to adaptively select the surrogate model
[31] The kernel dictionary consists of the same six kernels used in CAKE.
Compositional Kernel Search (CKS): Uses greedy search to discover kernel structures that best explains the observed data 12]
Automated BO (ABO): Treats the kernel selection as "black-box" optimization problem and uses BO to solve it 32
5.1 Hyperparameter Optimization
Setup. We consider the hyperparameter optimization tasks available in the HPOBench package [4]. We included a total of 60 tasks, comprising 12 OpenML datasets and 5 machine learning (ML) models:
logistic regression (LR), support vector machine (SVM), random forest (RF), XGBoost (XGB), and multi-layer perceptron (MLP). Each model-dataset pair exhibits a unique optimization landscape, making these tasks an ideal testbed for evaluating the generalization performance of BO methods across diverse scenarios. Further details regarding the hyperparameter search spaces and dataset characteristics are provided in Appendix B Here, goal for each task is to maximize the accuracy of the ML model on the unseen test data. Each experiment was executed for T 100 trials and repeated using 20 different random seeds.
Results. Figure shows the average test accuracy for different ML models on all datasets. The results demonstrate that CAKE consistently achieves the highest accuracy compared to the other methods across all tasks. also worth noting that CAKE excels in the earlier stages of the optimization process, when fewer observations are available. This suggests that CAKE is able to effectively leverage fewer data samples to quickly converge to high-performing configurations. We provide guantitative analysis to support this finding in Appendix C.7 Our results also reveal significant

---

LR
SVM
RF
0.88
0.900
0.84
Accuracy (1)
0.87
0.76
0.80
0.84
0.72
0.76
0.81
0.68
0.78
0.72
0
40
80
40
80
40
80
XGB
MLP
0.92
0.87
SE
DGP
Accuracy (1)
0.84
M5
EGP
Random
CKS
Utility
ABO
0.84
BLC
CAKE
0.78
0.80
40
40
80
Trials
Triols
Figure Average test accuracy over 20 random seeds for different ML models.
variations in performance among fixed and adaptive kernel methods. For instance, M5 and Utility perform reasonably well in tuning SVM and RF models, but struggle with tuning LR and XGB. EGP and DGP exhibit moderate performance, often outperforming fixed kernels but falling short compared to more flexible approaches such as CKS and ABO. Overall, CAKE demonstrates superior performance consistently across all tasks. Another key advantage of CAKE is that the learned kernel expressions are also interpretable. In Appendix C.4 we amalyze one such expression and show how CAKE automatically translates it into natural language description.
5.2 Controller Tuning
Setup. We consider two real-world controller tuning tasks that simulate dynamic environments, where small changes in the environment condition may result in significantly different outcomes. For the first task, we consider the robot pushing problem [5I), which involves tuning a controller for two robotic hands to push two objects towards some specified target positions. The controller parameterized by d = 14 parameters that determine the position and orientation of the hands, the pushing speed, direction of movement, and duration of the push. The second task involves tuning controller for the lunar lander environment 48), which is defined by = 12 parameters that determine how to map the 8-dimensional state vector comprising of position, angle, velocity, and ground contact indicators, to one of four actions: firing the main engine, left or right orientation engines, or doing nothing. The goal is to achieve a cumulative reward of least 200 points, which corresponds to successful landing while minimizing penalties from crashes excessive engine For both tasks, we evaluate the performance using T = 1000 iterations, averaging the results over 10 different initial conditions (e.g., positions, terrains, velocities). We provide more details regarding the reward functions and environment implementations in Section B.3.2
The results shown in demmontttte that the controllers optimized by CAKE achieved the highest average rewards both tasks. In robot pushing task, CAKE converges to a high- performing solution much faster than the other methods, highlighting its ability to quickly learn effective control policies from limited data. On the other hand, we found that fixed kernels like SE and M5 tend to plateau earlier, while adaptive methods such Utility and BIC show only moderate
Results.

---

Robot Pushing
Lunar Lander
240
Target:
Reward (f)
180
120
200
400
600
800
1000
200
400
600
800
1000
Iteration
Iteration
SE
Random
BIC
EGP
ABO
M5
Utility
DGP
CAKE
Figure 5: Average reward for the controller tuning tasks over 10 different initial conditions.
improvements over the fixed kernels and still lag behind compositional approaches. Notably, ABO is the only method besides CAKE that is able to reach the target score of 200 points in the lunar landing task. Despite this, observed that ABO exhibits greater performance fluctuations compared to CAKE, especially in more challenging environments. Mcanwhile, the remaining methods struggle to maintain high scores and often incur greater penalties due crashes or inefficient landings. Overall, these results show that CAKE's adaptive nature enables it to effectively respond to changes in the underlying objective (e-g., environmental shifts), offering greater robustness than fixed-kernel methods that may fail under such conditions.
5.3 Photonic Chip Design
Setup. We apply our method to the design of photonic chips, which is a challenging problem in physics and engincering [28]. Due to the high cost of fabrication, it is infcasible try all, or even few, of the design parameters. As a result, one has to rely on extensive computer simulations to assess the chip's 's performance. This makes the task "black-box inverse design problem, where the goal is to optimize the chip parameters to meet some desired performance indicators. We consider five key indicators for assessing the chip's 's performance: Q-factor (f1), wavelength (/2), lasing area (/a), power (J4), and divergence angle (fs). Based on these performance indicators, we can calculate the overall score for given set parameters: a/1 + 3 + 7/3 + 6.4 e/s. where we set a = 3 = 1,
8 100, and 20 unify the scale between different objectives. Our goal is to find Pareto-optimal solution that balances the trade-offs among the five competing objectives, thereby achieving the best overall chip performance. We provide detailed descriptions of each objective and their physical interpretations in Appendix [B.3.]]
Baselines. We consider two videly-used multi-objective BO methods: Single-Task GP, where each objective is modeled separately using GP with an M5 kernel, and Additive GP, which models the overall objective as sum of independent GPs, each using an SE kernel. We also include CKS and ABO as compositional kernel baselines to provide direct comparison against our proposed CAKE method. For all methods, we set T = 250 with 10 different random initializations and employed the expected hypervolume improvement (EHVI) (88 as the acquisition function.
Results. Ssowss shows the score and hypervolume of the chip designed by CAKE and the competing baselines. Compared the basclines, CAKE achieved the highest values in both metrics, indicating superior optimization performance and better exploration of the Pareto front. This likely stems from its ability utilize different kernels tailored individual objectives, unlike the other baselines that rely on a single or additive kernel shared across all objectives. CAKE also outperforms CKS and ABO, demonstrating
advantage handling multi-objective settings. Notably, CAKE found solution with a significantly higher score in less than 40 trials, which is equivalent to tenfold speedup in the design cycle compared the baselines. From practical point-of-view, this level of acceleration translates to significant reductions in both the time and cost needed to design the chips.

---

320
1,00
240
Hypervolume (t)
0.75
Scone (1)
160
0.50
0.25
0.00
100
150
200
250
50
100
150
200
250
Trials
Trials
Single-Tast Task GP
Additive GP
CKS
ABO
CAKE
Figure 6: Average score and hypervolume of the designed chip over 250 trials.
Table I: Average rank (4) + standard error on HPOBench over 20 random seeds.
Method
LR
SVM
XGB
MLP
Average
Random Sampling
6.8 - 0.1
6.9 0.1
6.7 0.1
6.8 0.1
6.8 0.1
6.80
Genetic Algorithm
2.6 + 0.1
2.5 1.5+0.1 0.1
2.7
2.7 0.1
2.8 0.1
2.9 19+00
0.1
2.70
CAKE + BIC
3.0
3.1
2.1
0.0
3.1
3.02
CAKE Utility
2.3
2.2
2.4
2.5
2.6
2.40
Adaptive + BAKER
4.1
4.4
4.6
4.8
4.7
4.60
CKS + BAKER
3.1
3.2
3.1
3.1
3.2
3.12
CAKE + BAKER
1.1
1.0
1.0
1.1
1.1
1.04
5.4
Ablation Study
Setup. To evaluate the influence of each component of our method, we conduct an ablation study with the following configurations: (1) Random Sampling: randomly combines base kernels using addition and multiplication, (2) Genetic Algorithm: applies genetic operators
population of kernels guided by fitness, (3) CAKE + BIC CAKE Utility: CAKE selects kernels based on fitness or acquisition value only, (4) Adaptive + BAKER CKS + BAKER: adaptive or compositional
baselines with BAKER, and (5) CAKE + BAKER: full method with LLM and BAKER. We evaluate all ablation setups on the full set of 60 tasks in HPOBench.
Results. Table shows that CAKE + BAKER achieves the best overall performance, indicating that both the LLM and BAKER contribute to the effectiveness of our method. We found that removing either component leads to performance degradation. For instance, CAKE + BIC and CAKE Utility, which only use one selection criterion, perform reasonably well but underperform compared to CAKE + BAKER. This suggests that while the fitness-based or utility-based kernel selection individually effective, combining them with BAKER improves the performance across diverse tasks. As expected, random sampling performs the worst, suggesting that the LLM generates meaningful kernel expressions rather than just random combinations. We further support this by analyzing the evolution of the kernel population's fitness (see Scction[5.5] where we observe a distribution shift toward higher fitness values after each successive round of LLM edits. Although CKS BAKER outperforms Adaptive + BAKER, it still underperforms compared to CAKE + BAKER. This indicates that while compositional kernels capture more complex patterns than standard kernels, they still lack the contextual understanding provided by the LLM.
5.5 Evolution of Population Fitness
To demonstrate the benefits of using LLM for kernel generation, conduct experiment analyzing how the fitness of the kernel population evolves through successive edits by the LLM Starting with
In this context, an LLM edit refers performing both crossover and mutation operations.

---

LLM
Randonm
GA
0.0
0.2
0.4
Fitness
Intial
After edit
After 5 edits
After 10 odita
Figure Evolution the kernel population's fitness over successive edits by LLM, random recombi- nation, and genetic algorithm (GA).
initial population of base kernels, evaluate their fitness scores and then iteratively apply LLM edits to evolve the population over time. We further compare it with two baselines which replaces the edit step with random recombination and genetic algorithm (GA) operators. Figure 7 illustrates the progression the fitness distributions after 1, 5, and 10 successive edits. The initial population exhibits wide distribution of fitness values with concentration variability in the fitness scores among the base kernels. After just one LLM edit, the distribution immediately shifts toward higher fitness values, indicating that the LLM effectively enhances the population through both crossover and mutation. As more LLM edits are applied, the mean fitness continues increase, and the variance narrows, indicating consistent improvement and convergence toward high-pun perforreforing kernels. In contrast, Random and GA show slower convergence, with their fitness distributions remaining broader and less sharply peaked over the same number edits. in driving population fitness upward. Overall, these results demonstrate that not only L.LMs can serve as effective genetic operators, but also outperform those produced by random recombination and traditional GA operators.
lower scores, reflecting the
Conclusion
We introduced CAKE, novel kerne design method that leverages LLMSs crossover and tation operators to enable adaptive surrogate modeling in BO. To complement CAKE, we further propose BAKER to address the issue where kernels with good model fit may not lead to substan- tial improvement towards the optimization progress. Experimental results have shown that CAKE achieved superior accuracy across all the hyperparameter optimization tasks, particularly excelling in the early stages when the observations are scarce. In the dynamic controller tuning tasks, CAKE consistently obtained the highest average rewards and showed strong adaptability under varying environmental conditions. In the photonic chip design problem, CAKE demonstrated its ability to effectively balance multiple conflicting objectives while achieving significant speedup over baseline methods. While the current work focuses on BO, our broader goal is to develop universal adaptive kernel method that is applicable to various MI. tasks. We refer the readers Appendix D for further discussions the limitations and directions for future work.
10

---

Acknowledgements
This work was supported in part by the NSFC under Grant 62271433, in part by Shenzhen Science and Technology Program under Grant and part by the Guangdong Provincial Key Laboratory Big Data Computing, The Chinese University of Hong Kong, Shenzhen.
14 and ZDSSS200200222011000000000
References
[1] Aglietti, V., Ktena, I., Schrouff, J., Sgouritsa, E., Ruiz, J. R., Malek, A., Bellot, A., and Chi- appa. S. FumBO: Discovering acquisition functions for Bayesian optimization with FunSearch. arXiv preprint arXiv:2406.0444 048244 2024.
[2] Ament, S., Daulton, S., Eriksson, D., Balandat, M., and Bakshy, Unexpected improvements to expected improvement for Bayesian optimization. Advances in Neural Information Processing Systems, 36, 2023.
[3] Arango, S. P., Jomaa, H. S., Wistuba, M., and Grabocka, J. HPO-B: A large-scale reproducible benchmark for black-box HPO based OpenML. In Neural Information Processing Systems Datasets and Benchmarks Track, 2021.
[4] Bach, F. Exploring large feature spaces with hierarchical multiple kernel learning. arXiv preprint arXiv:0809, 1493, 2008
[5] Bach, F. Learning theory from first principles. MIT Press, 2024.
[6] Balandat, M., Karrer, B., Jiang, D., Daulton, S., Letham, B., Wilson, A. G., and Bakshy, E. BoTorch: A framework for efficient Monte-Carlo Bayesian optimization. Advances in Neural Information Processing Systems, 33, 2020.
[7] Bitzer, M., Meister, M., and Zimmer, C. Structural kernel search via Bayesian optimization and symbolical optimal transport. Advances in Neural Information Processing Systems, 35: 39047-39058, 2022.
[8] Brown, T., Mann, B., Ryder, N., Subbiah, M., Kaplan, J. D., Dhariwal, P., Neelakantan, A.. Shyam, P., Sastry, G., Askell, A., et al. Language models are few-shot learners. Advances in Neural Information Processing Systems, 33, 2020.
[9] Calandra, R., Seyfarth, A., Peters, J., and Deisenroth, M. Bayesian optimization for learning gaits under uncertainty. Annals of Mathematics and Artificial Intelligence, 76(1):5-23, 2016. ISSN 1573-7470.
[10] Chen, A., Dohan, D., and So, D. EvoPrompting: Language models for code-level neural architecture search. Advances in Neural Information Processing Systems, 36, 2024.
[11] Chen, G., Zhu, K., Kim, S., Zhu, H., Lai, Y., Yu, B., and Pan, D. Z. LLM-cnhanced Bayesian optimization for efficient analog layout constraint generation. arXiv preprint arX5v:2406.05250, 2024.
[12) Duvenaud, D., Lloyd, J., Grosse, R., Tenenbaum, J., and Zoubin, G. Structure discovery in nonparametric regression through compositional kernel search. In International Conference on Machine Learning, pp. 1166 1174. PMLR, 2013.
[13] Eggensperger, K.. Feurer, M., Hutter, F., Bergstra, J., Snock, J., Hoos, H. Leyton-Brown K.. et al. Towards an empirical foundation for assessing Bayesian optimization of hyperparameters. In NIPS Workshop on Bayesian Optimization in Theory and Practice, volume 10. pp. 1-5, 2013.
[14] Eggensperger, K., Müller, P., Mallik, N., Feurer, M., Sass, R., Klein, A., Awad, N., Lindauer, M., and Hutter, F. HPOBench: A collection of reproducible multi-fidelity benchmark problems for HPO. In Neural Information Processing Systems Datasets and Benchmarks Track, 2021.
[15] Frazier, P. I. A tutorial on Bayesian optimization. arXiv preprint 1rXivv1807.0225 2018.

---

[16] Gardner, J., Guo, C., Weinberger, K., Garnett, R., and Grosse, R. Discovering and exploiting additive structure for Bayesian optimization. In Artificial Intelligence and Statistics, pp. 1311- 1319. PMIR, 2017.
[17] Garnett, R. Bayesian optimization. Cambridge University Press, 2023.
[18] Ginsbourger, D., Helbert, C., and Carraro, L.. Discrete mixtures of kernels for kriging- -aagingg- optimization. Quality and Reliability Engineering International, 24(6):681 2008.
[19] Gönen, M. and Alpaydin, E. Multiple kernel learning algorithms. Journal of Machine Learning Research, 12(64):2211 1268,, 2011.
[20] Han, C., Wang. Z., Zhao, H., and Ji, H. Explaining emergent in-context learning as kernel regression. arXiv preprint arXiv:2305.12766 2023.
[21] Hebbal, A., Brevault, L.,, Balesdent, M., Talbi, E.-G., and Melab, N. Bayesian optimization using deep Gaussian processes with applications to aerospace system design. Optimization and Engineering. 22:321-361, 2021.
[22] Holland, J. H. Adaptation in natural and artificial systems: An introductory analysis with applications biology, control, and artificial intelligence. MIT press, 1992.
[23] Jones, D. R., Schonlau, M., and Welch, J. Efficient global optimization of expensive black-box box functions. Journal of Global Optimization, 13(4):455-455 492,, 1998.
[24] Kojima, T., Gu, S. S., Reid, M., Matsuo, Y., and Iwasawa, Y. Large language models are zero-shot reasoners. Advances in Neural Information Processing Systems, 35, 2022.
[25] Korovina, K., Xu, S., Kandasamy, K., Neiswanger, W., Poezos, B., Schneider, J., and Xing, P. ChemBO: Bayesian optimization of small organic molecules with synthesizable recommenda- tions. arXiv preprint arXiv: 01455, 2019.
[26] Lázaro-Gredilla M., Quinonero-Candele J., Rasmussen, C. E., and Figueiras- Vidal, A. R. Sparse spectrum Gaussian process regression. The Journal of Machine Learning Research, 11: 1865-1881, 2010.
[27] Lehman, J., Gordon, J., Jain, S., Ndousse, K., Yeh, C.,, and Stanley, O. Evolution through large models. In Handbook of Evolutionary Machine Learning, pp. 331-366. Springer, 2023.
[28] Li, R., Zhang, C., Xie, W., Gong, Y., Ding, F., Dai, H., Chen, Z., Yin, F., and Zhang, Z. Deep reinforcement learning empowers automated inverse design and optimization of photonic crystals for nanoscale laser cavities. Nanophotonics, 12(2):319-33 9344,33333220001 2023.
[29] Liu, T., Astorga, N., Seedat, N., and der Schaar, M. Large language models enhance Bayesian optimization. In International Conference on Learning Representations, 2024.
[30] Lloyd, J. R., Duvenaud, D., Grosse, R., Tenenbaum, J. B., and Ghahramani, Z. Automatic construction and language description of nonparametric regression models. arXiv preprint 1402 43044, 2014.
[31] Lu, Q. Polyzos, K. D., Li, B., and Giannakis, G. B. Surrogate modeling for Bayesian optimiza- tion beyond single Gaussian process. IEEE Transactions on Pattern Analysis and Machine Intelligence, 45(9): 1283-11296, 2023.
[32] Malkomes, G. and Garnett, R. Automating Bayesian optimization with Bayesian optimization. Advances in Neural Information Processing Systems, 31, 2018.
[33] Martinez-Cantin R. Bayesian optimization with adaptive kernels for robot control. In IEEE International Conference on Robotics and Automation, pp. 3350-380 3356, 2017.
[34] Meyerson, E., Nelson, M. J., Bradley, H., Gaier, A., Moradi, A., Hoover, A. K., and Lehman, Language model crossover: Variation through few-shot prompting. arXiv preprint 2:Xi:2230.112170 2024.
[35] Murphy, K. P. Probabilistic machine learning: An introduction. MIT Press, 2022.
12

---

[36] Rasmussen, C. E. and Williams, C. K. I. Gaussian processes for machine learning. MIT Press, Cambridge, Mass, 2006.
[37] Roman, I,, Santana, R., Mendiburu, A., and Lozano, A. An experimental study in adaptive
kernel selection for Bayesian optimization. IEEE Access, 7:184294-184302, 2019. doi: 10.1109/ACCESS 201999999....
[38] Salimbeni, H. and Deisenroth, M. Doubly stochastic variational inference for deep Gaussian processes, Advances in Neural Information Processing Systems, 30, 2017.
[39] Schwarz, G. Estimating the dimension of model. The Annals of Statistics, 6(2):461-464)
1978.
[40] Shahriari, B., Swersky, K., Wang, Z., Adams, R. P., and de Freitas, N. Taking the human out of the loop: A review of Bayesian optimization. IEEE, 104(I):148 175, 2016.
[41] Simpson, F., Davies, I., Lalchand, V., Vullo, A., Durrande, N., and Rasmussen, C. E. Kernel identification through transformers. Advances in Neural Information Processing Systems, 34: 10483-10495, 2021.
[42] Smola, A. J. and Schölkopf, B. Learning with kernels, volume 4. Citeseer, 1998.
[43] Snoek, J., Larochelle, H., and Adams, R. Practical Bayesian optimization of machine learning algorithms. arXiv preprint 1206:::20.29444 2012.
[44] Srinivas, N., Krause, A., Kakade, S. M., and Seeger, M. W. Information-theoreticc regret bounds for Gaussian process optimization in the bandit setting. IEEE Transactions on Information Theory, 58(5) 2012.
[45] Surjanovic, S. and Bingham, D. Virtual library of simulation experiments: Test functions and datasets. BRL.htp: //wp::////wwww// sfu. aaa.acaa/sss////// sssuraano 2013.
[46] Theodoridis, Machine learning: From the classics to deep networks, transformers, and diffusion models. Elsevier, 2024.
[47] Thompson, W. R. On the likelihood that one unknown probability exceeds another in view of the evidence of two samples. Biometrika, 25(3-4):285-299) 1933.
[48] Towers, M., Kwiatkowski, A., Terry, J., Balis, J. U., De Cola, G., Deleu, T., Goulao, M., Kallinteris, A., Krimmel, M., KG, A., et al. Gymnasium: A standard interface for reinforcement learning environments. arXiv preprint arXiv:2407.11033 2024.
[49] Tripp, A. and Hernández-Lobato, J. M. Diagnosing and fixing common problems Bayesian optimization for molecule design. arXiv preprint arXiv:2406.07779, 2024.
[50] Wang, X., Jin, Y., Schmitt, S., and Olhofer, M. Recent advances in Bayesian optimization. ACM Computing Surveys, 35(3ss), 2023.
[51] Wang, Z., Gehring, C., Kohli, P., and Jegelka, S. Batched large-scale Bayesian optimization in high-dimensional spaces. In International Conference on Artificial Intelligence and Statistics, pp. 745-754. PMLR, 2018.
[52] Wei, J., Tay, Y., Bommasani, R., Raffel, C., Zoph, B., Borgeaud, S., Yogatama, D., Bosma, M., Zhou, D., Metzler, D., et al. Emergent abilities of large language models, arXiv preprint arXiv:2206.07882 2022.
[53] Wei, J., Wang, X., Schuurmans, D., Bosma, M., Xia, F. Chi, E., Le, Q. V., Zhou, D., et al. Chain-of-thougle prompting elicits reasoning in large language models. Advances in Neural Information Processing Systems, 2244824-24837,
[54] Wilson, A. G. and Adams, R. P. Gaussian process kernels for pattern discovery and extrapolation. In International Conference on Machine Learning, pp. 1067-1075, 2013.
[55] Wilson, G., Hu, Z.., Salakhutdinov, R., and Xing, P. Deep kernel learning. International Conference Artificial Intelligence and Statistics, volume 51, pp. 370-378, 2016.
13

---

[56] Xie, S. M., Raghunathan, A., Liang, P., and Ma, T. An explanation of in in-context learning as implicit Bayesian inference. arXiv preprint arXiv:2111.02080 2022.
[57] Yang, C., Wang, X., Lu, Y., Liu, H., Le, Q. V., Zhou, D., and Chen, X. Large language models as optimizers. In International Conference on Learning Representations, 2024.
[58] Yang, K., Emmerich, M., Deutz, A., and Bäck, T. Multi-objective Bayesian global optimization using expected hypervolume improvement gradient. Swarm and Evolutionary Computation, 44: 945-956, 2019.
[59] Yang. Z., Ge, L., Han, D., Fu, T., and Li, Y. Reasoning BO: Enhancing Bayesian optimization with long-context reasoning power of LLMs. arXiv preprint vrXiv:2505.12833 2025.
[60] Zhou, Y., Muresanu, A. I., Han, Z., Paster, K., Pitis, S., Chan, H., and Ba, J. Large lan- guage models are human-level prompt engineers. International Conference on Learning Representations, 2023.
14

---

A Mathematical Details
A.1 Bayesian Optimization with Gaussian Processes
Gaussian process (GP) describes a collection of random variables, any finite number of which have a joint Gaussian distribution (3). Mathematically,, GP can be expressed as, GP (m(x), ke xx; 0)). where m(x) is the mean function and Ko (x, x'; is the covariance (kernel) function parameterized by some hyperparameters 0. Given any finite collection of inputs X = X1:1, the outputs are jointly Gaussian,
f(X) N (mx, (X, X)))
(2)
where mx (X) TR is the mean function vector evaluated X, often assumed to be 0 in practice, and ((X, X; O) E TR is the covariance matrix with entries [K(X,X; 0)kk, = ko [x, x,)
We assume that the evaluations of f any point x, corrupted by a o-sub-Gaussian noise
(xx) + et.
where & ~ (O, o?). Given the observed data D, = {X, where y = V1:ts the joint Gaussian distribution of the observed data and an arbitrary query point x is:
| N 0. Ki o.I kx,(xx)
ko x) 1):
x()
k (x)
where Ki:a- = K(X, X; 0) and (x) = ko(X, x). It follows that, the posterior distribution of any query point is marginally Gaussian,
f(x) (x ox:
(5)
where
Mx(x; 0) = ([f()) = (x)
(x) K + -.
(6a)
o, 0) [xf
ko (x, x) - (x) + - kx
(6b)
Based the above posterior distribution, the acquisition function the use its statistics to trade-off exploitation (where (x: 0) is high) and exploration (where of (x: 0) is high) effectively. Among the
various acquisition functions proposed, expected improvement (EI) (23 remains the default choice in
many BO applications [3). Let us define Má = maxxex d4 (x; 0) as the best mean value. The El
acquisition function can then be expressed in closed form as:
(x; D.) = [0, fxx - H or 0) [u + <du)].
(7)
where (x: 0) "/ (x: 0), and <(-) and +(.) are the standard normal density and cumulative distribution functions, respectively.
A.2 Model Selection
We conduct model selection over a discrete, infinite space of kernels = {ki, k.....). As each kernel comes with its own hyperparameters, when referring to a kernel ks we consider the whole family over its hyperparameters (he 0 E θ). Given some model selection criteria h IC + R. our goal is to identify the optimal kernel.
actually dealing with space of kernel families. Thus,
kM arg max h(k] [D).
(8)
keK
commonly-used criterion for probabilistic models, such as GPs, is the marginal log- likeliioodd [46].
(nk (D) = log X, k)) log ply X, (0)
Unfortunately, the above likelihood is generally intractable for GPs [36]. resort the Laplace approximation (35,
2 log det --1 dg log 2T
1
log ply X, k) RJ log X, 0, k) + log
(10)
2
where Ô denotes the maximum posteriori (MAP) estimate of the hyperparameters with de being its dimension. The term E-
V log (D k) represents the Hessian matrix evaluated at the MAP estimate. Note that Eq. TT) can be interpreted rewarding model fit while penalizing model complexity. In this work, we use the Bayesian Information Criterion (BIC) (39), which was also previously employed in [12] and can be seen as approximation of the Laplace method.
15

---

1.0
1.0
1.0
0.8
0.8
0.8
Fitneess (t)
Fitness (t)
0.6
Fitness (t)
0.6
0.4
12c
0.4
Ps 0.1
0.4
7/p
Rg
0.3
0.2
Rg
0.2
0.5
0.2
7/ 10
Rg
0.7
10
0.9
7y 20
0.0
0.0
0.0
20
10
15
30
20
Generation
Generation
Generation
(a) Number of crossovers.
(b) Mutation probability.
(c) Population size.
Figure 8: Average fitness (1) with respect to different CAKE setup. The error bars indicate the
standard errors.
B Experimental Details
In this section, we provide additional details on the implementation, baselines, and benchmarks employed in our experiments.
B.1 Implementation
As described in Algorithm I CAKE depends on three key parameters: the number crossovers ne- the mutation probability Pm. and the population size ny. To evaluate the impact of these parameters, we conduct a sensitivity analysis in Fig. 8 which shows the average fitness of the population over generations under different settings. Below, we provide some intuition for setting these parameters.
Number of crossovers. The number of crossovers determines how many are generated via crossover in each iteration. A higher value of ne can accelerate the exploration of diverse kernel combinations but increases the number of API calls to the LLM, which may be costly. In our experiments, we found that setting nt, 5 offers good balance between exploration and efficiency.
candidate kernels
Mutation probability. Each mutation operation introduces local variations to existing kernels in the population, which may help to refine solutions and escape local optima. The mutation probability controls the likelihood of applying a mutation operation during each iteration. A higher pm promotes greater diversity and prevents premature convergence, especially when the top-pert cop-performing kernels become similar. However, excessively high mutation rates risk disrupting promising kernel structures before they can be fully explored. Based our experiments, we Pm 0.7 maintain a balance between sufficient exploration and stable evolution.
Population size. The population size dictates how many candidate kernels evolve simultaneously. A larger ni enhances diversity in the search space and supports more thorough exploration, but also increases computational cost due repeated GP model fitting for each kernel. Conversely, smaller np may lead to premature convergence or insufficient sampling the kernel space. Empirically, we found that setting n. 10 maintains diverse yet computationally manageable population.
B.2 Baselines
To test our proposed method, we consider three categories of baseline methods: fixed, adaptive, and compositional approaches.
B.2.1 Fixed
For the fixed kernel-based baselines, we consider the following widely-used kernels in BO, along with their respective hyperpriors:
16

---

1. Squared Exponential (SE):
(- (x- xd)=
k(2, x') o² exp
(11)
212
where / the lengthscale parameter and a is the amplitude parameter. The hyperpriors are:
Gamma 2.0, 2.0), o² Camma 3.0).
(12a)
(12b)
2. Periodic (PER):
k( x/) = o² exp ( sin? (T
x' l/p)
(13)
where / is the lengthscale parameter, is the period length, and a is the amplitude parameter. The hyperpriors are:
Gamma (.00, 2.0),
(14a)
o² Gamma (2.0, 3.0),
(14b)
p Gamma (2.0, 2.0).
(14c)
3. Linear (LIN):
k(x, x²) = 0² xxx + oa.
(15)
where o² is the variance parameter, of is the noise variance. The hyperpriors
o² Gamma (2.0, 3.0),
(16a)
a Gamma Gamma(2.0,38
(16b)
4. Rational Quadratic (RQ):
k(x, x') = o² (1 (x x²)2
(17)
2ol2
where is the lengthscale parameter, is the relative weighting parameter, and a is the amplitude parameter. The hyperpriors are:
Gamma 2.00, 2.0),
(18a)
0² Gamma 2.0, 3.0),
(18b)
a Gamma (2.0, 2.0).
(18c)
5. Matérn:
k(xxx') 02 2'v
[(v) (V2rD) Kr (V2vD).
(19)
where
(x x/)2 x')2
(20)
is the distance between x and x' scaled by the lengthscale parameter, K, is the modified Bessel function, and a is the amplitude parameter. In our experiments, the smoothness parameter // is set 1/2, 3/2, 5/2, corresponding Matémn-1/2 (MI), Matérn-3, -32 (M3), or Matérn-5/2 (M5) respectively. The hyperpriors are:
Gamma 2.00 2.0).
(21a)
(2lb)
Note that all the above kernels are defined on R and are applied input dimension when indicated by the base kernel symbol, e.g., SE, denotes SE kernel is applied to the i-th dimension.
17

---

Table Details of the OpenML datasets used in the experiments. More information can be found https://wwys openml org
Dataset
Task ID
Number Instances
Number of Features
credit_
31
1000
21
vehicle
53
846
19
kcl
2109
2109
22
phoneme
9952
5404
blood transfusion
10101
748
5
australian
146818
690
146821
1728
7
segment
146822
2310
20
heart_h
50
294
14
tic_tac__ _a__ta__too _oe_
145804
958
kr kp
3196
37
qsar
9957
1055
42
B.2.2 Adaptive
For the adaptive kernel-based baselines, adopt the implementation from Roman et al. 37 and apply the following selection criteria:
1. Random: This criterion selects kernel randomly from the set of available kernels.
2. Utility: Based on the proposed query points from each kernel, this criterion selects the kernel with the highest utility (acquisition) value:
k = arg max (xx,, D, k).
(22)
kEk
3. BIC: This criterion selects the kernel with the lowest BIC value:
k arg min BIC(k:
(23)
B.2.3 Compositional
For the compositional kernel-based baselines, we consider the following methods:
1. Deep GP. For the deep GP baseline, we use the DeepGP implementation from GPyTorch, where training and inference are conducted using the doubly stochastic variational inference method [38]
2. Ensemble GP. For the ensemble GP baseline, we follow the implementation suggested by Lu et al. 31], using the six base kernels used in CAKE to form the kernel dictionary.
3. Compositional Kernel Search (CKS). For the CKS baseline, we start from the same base kernels used in CAKE and apply greedy search to search for the kernel structures [2]
4. Automated BO (ABO) For the ABO baseline, https: t/gtthu.. thub. com http//gustvvaaoooeeeoe abo and follow the setup suggested [32]
the code provided by the authors
B.3 Benchmarks
All experiments on the benchmarks were conducted locally consumer-grade lapto except for the photonic chip design experiment in Section 553 which was executed on high-peetormance computing (HPC) cluster due the computational demands of the physics-based simulation.
B.3.1 Hyperparameter Optimization
Datasets. We include 12 OpenML datasets available in the HPOBench package the selected datasets are given in Table
14
The details of
MacBook Air M2 (2022) with 8-core CPU, 8-core GPU, 8 GB unified memory, and 256 GB SSD storage.
18

---

Search space. We follow the search space designated in HPOBench, where we discretize the search space to facilitate efficient tabular lookup operations for various configurations [3]. Each hyperparameter is defined by its type (linear or log scale), along with lower and upper bounds. For example, [log, 0.001, 1.0) 0] indicates that the hyperparameter values are sampled on logarithmic scale between 0.001 and 1.0. In contrast, [linear, 0.O, 1. 1.0] implies uniform sampling over the interval [0.0, 1.0.. The search space for each ML model is summarized as follows:
Logistic Regression (d 2)
alpha: Regularization strength, [log, 0.001, 1.0]
-eta:: Initial learning rate, [log, 0.001, 1.0]
Support Vector Machine (SVM) (d = 2)
C: Inverse regularization strength, [log, 0.01, 10.0]
gamma: RBF kernel coefficient, [log, 0.001, 1.0]
Random Forest = 4)
max_depth: Maximum depth of each tree, [log, 1, 50]
max_ :ex_ureeuse:: Number of features to consider when looking for the best split, [linear, 0.0, 1.0]
sain_sampees___ leaf: Minimum number of samples required to be leaf node,
[linear, 1, 2]
mii_samples__pliss Minimum number of samples required to split an internal node, [log, 2, 128]
XGBoost (d = 4)
colsample_byter bytree: Fraction of features to use per tree, [linear, 0.1, 1, 1.0] eta: Learning rate that controls the contribution of each tree to the final prediction,
[log, 0.001, 1.0]
max_ max_depth: Maximum depth of a tree, [log, 1, 50]
reg_lambaa: L2 regularization term on weights, [log, 0.1, 10.0]
Multi-Layer Perceptron (MLP) (d=
alpha: L2 penalty (regularization term) coefficient, [log, 0. 001, 1.0]
batch_size: Number of training examples used one forward/backwwar pass, [log,
16, 128]
depth: Number of hidden layers in the neural network, [linear, 1, 3]
-leariing_rate patt_aateiint init: Initial learning rate for weight updates, [log, 0. 00001, 1.0]
- width: Number neurons in each hidden layer, [log, 16, 128]
B.3.2 Controller Tuning
Robot pushing. The reward function defined as: f(x = - Xx:
X
Xgi Xyal
where represents the starting positions of the objects, Xfs denotes their final positions, and Xgs indicates the goal. The objective to minimize the total distance from the initial and final positions of the objects to their respective goals, thereby maximizing the reward. We use the original code provided by Wang et al. [51], which is available online hattp:///iithu: com /z w/ Ensemble-Bane -ww.saayeeenniiiiaaaaaa-------ae
The reward system includes + +100 points for successful landing, -100 points for crash, +10 points per frame for each leg in contact with the ground, -0.3 points per frame for firing main engine, and -0.03 points per frame for firing side engines. implement the lunar lander environment using the code from
Lunar lander:
com Fontaaaraaaaaootttttoaaoo//ramma Gymnosiinnnnnnmmmmmmmmmm
B.3.3 Photonic Chip Design
Objective. The design parameters and indicators for optimizing the photonic chip are detailed in
Figure 9 We established target values based on our experiments and literature review to identify optimal standards for high-quality chip 28). For instance, a wavelength of 1310 nm is crucial for telecommunications and satellite applications, while high output power is essential for fields such as
19

---

Emission
Design Parameters
Length
Lattice constant
Width
Size of holes Filling factor
Layer thickness
Shape of holes
Material propertics
Indicator
Target
Q Q-factor (Q)
x 109
Wavelength (A)
1310
Lasing area (A)
A' 10-19
Power (P)
Pr 0.5
Divergence angle
1.0
Figure 9: Illustration of a photonic chip and its design parameters.
The inverse design problem focuses optimizing these parameters satisfy certain performance indicators.
Table 3: Details of the test functions used in the experiments.
Function
Domain
Ackley-d
[-5, 5]
2, 5
Beale
1, 1
Branin
5, 10j2
Dropwave
5.12, 5.12
Eggholder
-522, 512
Griewank-d
-600, 600
2,5
Hartmann
[0,
3
Levy
10
2,3
Rastringin-d
5.12, 5.12
2,4
Rosenbrock
1- 10
Six Six-Hump Camel
[- 2,
2
autonomous driving and medicine. Additionally, small divergence angle is vital for ensuring high beam quality and effective long-distance light propagation. Based on these performance indicators, we can define the objectives follows,
f1 1 Q' Q
(24a)
Q*
A X
f2 =
(24b)
X
A
f3 = 1
(24c)
A
p+
f1 =
(24d)
ps
(24e)
W
Q-factor (f1) is related to the loss and threshold of the laser, wavelength (f2) the operation wavelength of the laser, lasing area (fa) is the area of the laser beam at the laser's top surface, power (f4) the lasing power of the laser in watts, and divergence angle (fs) is the angle between outer boundary and centerline of the laser beam.
B.4 Prompt Design
Our prompts designed based three core components:
Problem Context: The optimization history, including the observed input-output pairs and corresponding fitness scores, provided in-context examples guide kernel evolution.
20

---

(a) Ackley-2
(b) Beale
(c) Branin
(d)
Dropwave
(e) Eggholder
(f) Griewank-2
(g) Levy-2
(h) Rosenbrock
Rastrigin-2
(j) Six Sa--Hump Camel
Figure 10: Visualization of the optimization landscapes of two-dimensional test functions.
0.9
0.8
Accuracy
0.7
0.6
0.5
Full
No Reasoning
No Context
No Instruct
Figure 11: Ablation of prompt components on HPOBench averaged over 20 different random seeds.
Task Instruction: The role assignment and explicit guidelines on how to generate valid kernels using the kernel grammar during crossover and mutation operations.
Reasoning: A phrase asking the LLM to provide a brief natural language explanation for each proposed kernel.
To evaluate the contribution of each component, conduct ablation study with the following configurations:
Full: This is the vanilla CAKE setup employed in our experiments, incorporating all three components.
No Context: This variant evaluates the importance of problem context by removing the optimization history (i.e., the observations and fitness values), while keeping the other components.
21

---

Table 4: Average normalized regret (4) over 20 random seeds for different test functions and methods. The values in parentheses indicate the standard errors. The best value is highlighted in bold and the second best value is underlined.
Fixed
Adaptive
Compositional
Function
SE
M5
Rundom
Utility
BIC
DGP
EGP
CKS
ABO
CAKE
0.1773
0.1220
0.1358
0.1062
0,1863
0,2510
0,1878
0.1012
0,1020
0.0783
Ackley-2
(0,0232)
(0.0262)
(0.0210)
(0.0165)
(0.0187)
(0.0357)
(0.0196)
(0.0177)
(0.0195)
(0.0203)
0,3185
0.2369
0.1722
0.2278
0.3590
0.3110
0.2285
0.1812
0.1910
0.1732
(0.0117)
(0.0184)
(0.0145)
(0.0171)
(0.0338)
(0.0215)
(0.0139)
(0,0856)
(0.0188)
(0.0250)
Beale
0.3554
0.3522
0.3855
0.4410
0.3571
0.4775
0,3088
0.4040
0,3118
0.2565
(0.0845)
(0.1006)
(0.0775)
(0.0989)
(0.0806)
(0.0958)
(0.0898)
(0.0712)
(0.0733)
(0.0786)
0.0183
0.0155
0.0227
0.0372
0.0371
0.4810
0.2045
0.0301
0.0101
0.0070
Branin
(0.0080)
(0.0037)
(0.0106)
(0.0107)
(0.0152)
(0.1017)
(0.0725)
(0.0122)
(0.0478)
(0,0534)
0.5110
0.5411
0.5460
0.5265
0.5461
0.5560
0.6290
0,5788
0,5529
0.4690
Dropwave
(0.0568)
(0.0523)
(0.0698)
(0.0622)
(0.0750)
(0.0531)
(0.0651)
(0.0669)
(0.0611)
(0.0538)
0.4941
0.3545
0.4015
0.4855
0.5485
0.4535
0,4345
0.4536
0.4210
0.1241
Eggholder
(0.0602)
(0.0452)
(0.0416)
(0.0527)
(0.0749)
(0.0615)
(0.0497)
(0.0533)
(0.0516)
(0.0541)
Griewank-2
0.1196
0.1282
0.1295
0.1310
0.1272
0.1156
0.0935
0,0589
0.0357
0.0267
(0.0692)
(0.0687)
(0.0686)
(0.0685)
(0.0686)
(0.0297)
(0.0248)
(0.0686)
(T11244)
(0,0256)
Griewank-5
0.0204
0.0223
0.0232
0.0178
0.0281
0.0815
0,0478
0.0258
0.0320
0.0185
(0.0032)
(0.0059)
(0.0051)
(0.0033)
(0.0096)
(0.0171)
(0:0110)
(0.0076)
(0.0125)
(0.0133)
0.0007
0.0019
0.0021
0.0358
0.6800
0.1305
0,1780
0.0001
0.0001
0.0001
Hartmann
(0.0001)
(0.0011)
(0.0009)
(0.0159)
(0.0734)
(0.0441)
(0.0402)
(0.0611)
(0.0544)
(0.0529)
Levy-2
0.1562
0.0418
0.0835
0.0555
0.1145
0.1965
0.0765
0,0668
0,0519
0.0353
(0.0684)
(0.0227)
(0.0460)
(0.0060)
(0.0666)
(0.0326)
(0.0198)
(0.0431)
(0.0187)
(0.0197)
0.1141
0.1422
0.1495
0,0880
0.1125
0.2266
0.0805
0,0580
0.0590
0.0505
Levy-3
(0.0209)
(0.0403)
(0.0290)
(0.0159)
(0.0230)
(0.0511) 1)
(0.0173)
(0.0194)
(0.0181)
(0.0190)
Rastringin-2
0.4325
0.4251
0.5310
0.3455
0,4490
0.5405
0.3869
0.3722
0,3420
0.3341
(0.0571)
(0.0765)
(0.0613)
(0,0442)
(0.0632)
(0.0751)
(0.0338)
(0.0588)
(0.0397)
(0,0468)
Rastringin-4
0.5765
0.5461
0.4815
0.5905
0.5200
0.5340
0.3270
0.3285
0.3179
0.3128
(0.0482)
(0.0671)
(0.0400)
(0,0509)
(0.0529)
(0.0475)
(0.0329)
(0.0511)
(0.0375)
(0.0499)
0.1025
0.0898
0.1015
0.1405
0.1475
0.5340
0.6040
0.0907
0.0901
0.0483
Rosenbrack
(0.0476)
(0.05 10)
(0.0573)
(0.0695)
(0.0688)
(0.1101)
(0.0971)
(0.0686)
(0.0907)
(0.0531)
0,2840
0.1507
0.3455
0.3310
0,3265
0,4940
0,5345
0,1071
0,1002
0,1015
Six Hump Camel
(0.0856)
(0.0459)
(0.1014)
(0.0760)
(0.0786)
(0.0854)
(0.0528)
(0.0771)
(0.0563)
(0.0669)
Mean regret
0.2454
0.2111
0.2341
0.2373
0.3026
0.3589
0,2881
0,1905
0.1745
0.1357
Median regret
0.1773
0,1422
0.1495
0.1405
0.3265
0.4535
0.2285
0.1012
0.1002
0.0783
No Instruct: This variant excludes task instructions and omits the explicit kernel generation guidelines (i.e., how to perform the crossover and mutation operators), while keeping the other components.
No Reasoning: Thus variant removes the requirement for the LLM to explain its reasoning. producing only the kernel expression.
We evaluate all variants on the HPOBench benchmark, as detailed in Section 5.1 The results, summarized in Figure [I] demonstrate that the full CAKE setup consistently outperforms all ablated variants, underscoring the importance of each prompt component in achieving superior optimization performance. The No Reasoning experience a slight degradation in performance, which suggests that the reasoning prompt not only serves to enhance interpretability, but also as form of self-reflection that helps the LLM evaluate and refine its own proposals. In contrast, the No Context variant performs significantly worse, demonstrating that access to optimization history (i.e., observed data points and kernel fitness values) is crucial and that the LLM effectively leverages this information. The No Instruct variant performed the worst compared to the other variants. To understand this drop, we examined the percentage of valid kernels, defined as the proportion of kernels that satisfy the kernel properties. We found that the No Instruct variant generated valid kernels only 68% of the time, while the other variants consistently produced valid kernels. This is due to the absence of the generation guidelines from the kernel grammar, which underscores its importance in the kernel generation process. Overall, these results strongly support the design choice of our prompts.
C Additional Results
In this section, we provide additional results and empirical analysis our proposed method.
22

---

Table Average computational time (4) in seconds per iteration for different methods.
Method
Time (s)
Main Bottleneck
Fixed
0.6
Single GP fitting
Adaptive
3.7
Multiple GP fitting & kernel selection
EGP
3.9
Multiple GP fitting & weight update
Deep GP
4.8
Variational inference
CKS
5.6
Multiple GP fitting & greedy search Multiple GP fitting & nested BO loop
ABO
7.4
CAKE
8.3
Multiple GP fitting & LLM inference
C.1 Benchmark Function Optimization
Setup. We consider optimizing a set of test functions commonly used as benchmark for optimization [45] We provide additional details on the selected test functions including the input domain and dimensionality, in Table 3 We visualize the optimization landscapes of the two-dimensional test functions in Figure TO From the figure, one can see the challenging nature of these functions, characterized by non non-convexity, many local minima, and steep ridges. The goal is to find the global minimum of each test function, where the maximum number of function evaluations is limited to 10 times the dimensionality of the function input, i.e., T = 10 x d. To evaluate the performance of each method, we consider the normalized regret (3)
(Xooe) f (Xbest)
(25)
f(xopt) - f (Xinit)
where [ximit) ) is the best function value among the initial points, (Xtees)) is the best value found by the method, and [(xopt)is )is the ground truth optimum. This metric is favorable it provides normalized and task-agnostic measure to compare the optimization performance across different tasks.
Results. Table shows the normalized regret averaged over 20 seeds for different functions and methods. Our results demonstrate that CAKE outperforms the baselines, achieving roughly 22.2% improvement in the mean regret and a 21.9% improvement in the median regret compared to the second-best method. Notably, CAKE ranks among the top two for all functions and achieves the best performance 12 times out of 15. Our results also reveal that fixed kernels such as SE and M5, which the default in BO, not universally effective and actually perform poorly in many test functions. We also found that adaptive methods exhibit inconsistent performance and generally underperform compared to the compositional approaches like ABO.
C.2 Computational Time
Table 5 shows the average clock time per iteration for different BO methods. Note that we only measure the time spent on surrogate model computation, not including the black-box function evaluations. In CAKE, we use OpenAl's gpt-4o-mini as the LLM. Based on our measurements, it processes about 104.5 tokens per second. Each LLM call, including both the input prompt and output response, uses around 180 tokens. This results in inference time of approximately 180/104.5- = 1.72 seconds per call. Each iteration of CAKE also involves fitting GP models for all kernels in the current population. Our experiments show that fitting one GP takes about 0.5 seconds. Therefore, fitting multiple GPs per iteration adds about 3-5 seconds to the total runtime. Other adaptive and compositional kernel methods that maintain multiple GP models also experience similar computational costs. However, methods like CKS and ABO involve additional steps, such as greedy search or nested BO loops, for constructing the kernels, which further increases their runtime. DGP requires performing doubly stochastic variational inference across deep layers, which adds even more computational overhead. In contrast, fixed and simple adaptive kernel methods, which either use a single GP or choose from a small set of kernels, are much faster per iteration. While CAKE has higher average runtime per iteration than the baselines, this increase largely due to external
'The analytic expression as well as the global optimum of these functions can be found at httpps://ww/
sfu. /ptpa///i//mmmiiiioooo
23

---

Crossover Response
Prompt:
You are given two parent kerelss and their fitness scores: LIN (0.53), SE (0.67). Please propose new kernel that has potentially higher fitness score. You may combine the parent kernels using any of the operators from { x). Briefly explain your reasoning behind the proposed kernel.
Response:
Kernel: LIN SE
Analysis: The L.IN kernel captures linear trends data, which may present given the varying function values across different inputs. The SE kernel, on the other hand, is effective in modeling smooth variations in the data, which could account for the fluctuations observed in the function values despite the overall linear trend. By combining these kernels with the addition operator (+), leverage linear trend from the LIN kernel while also allowing smooth deviations from this trend through the SE kernel.
Figure 12: Sample response for the crossover step.
Mutation Response
Prompt:
You are given a kernel its fitness score: LIN SE (0.72). Please propose a new kernel potentially higher fitness score. You may replace a base kernel in the current expression with another base kernel from the set {SE, PER, LIN, RQ, M3, M5). Briefly explain your reasoning behind the proposed kernel.
Response:
Kernel: LIN + RQ
Analysis: The LIN + SE kernel captures linear trend with smooth variations. However, the SE kernel assumes constant level of smoothness, which may not be appropriate for all data patterns. By replacing the SE kernel with the RQ kernel, introduce varying smoothness, allowing the model adapt different levels noise the data.
Figure 13: Sample response for the mutation step.
factors such as internet connectivity and API latency. Importantly, in most BO tasks, evaluating the black-box function is far more time-consuming than the BO process itself. Thus, reducing the number of function evaluations (i.e., improving sample efficiency) is often more important than minimizing the wall-clock time per iteration.
C.3 Case Study
We conducted case study examine the LLM's outputs and assess whether it provides useful responses. Specifically, we analyzed sample responses from the LLM during the crossover and mutation steps, which shown in Figure 122 and Figure respectively. In Figure [2] the L.M is given two parent kernels, LIN and SE, along with their fitness scores. It suggests combining them using the addition operator (++) improve the fitness. The explanation states that the LIN kernel captures linear trends, while the SE kernel models smoothness. Thus, combining them as LIN SE allows the new kernel represent both linear and smooth patterns in the data. In Figure [3 the LLM is given the kernel LIN + SE and its fitness score. It proposes replacing the SE kernel with the RQ kernel to further improve fitness. The reasoning is that while the SE kernel assumes constant smoothness, the RQ kernel allows for varying levels of smoothness. This makes the model more flexible, especially when dealing with data that changing noise patterns. These examples show that the L.M posesses solid understanding about the kernel properties and how they affect modeling (i.e., how different kernels can be combined or modified to better fit specific data characteristics), enabling it to make meaningful proposals during the kernel generation process.

---

C.4 Interpretability of CAKE
By design, the kernel grammar used in CAKE enables us to automatically generate interpretable descriptions of the data based on the proposed hypothesis (kernel expression). In this section, we analyze the kernel expression discovered by CAKE for one of the hyperparameter optimization tasks from HPOBench, which involves tuning an SVM model on the credit_ -g dataset. Recall that, for the SVM model, we have two hyperparameters to tune: the regularization parameter and the RBF kernel coefficient gamma. The kernel expression proposed by CAKE for this specific task is,
(SE PER) (LIN x RQ) + (M5 SE).
(26)
Existing kernel grammar- crasedr-bbased methods, such as Automatic Bayesian Covariance Discovery (ABCD) 300, typically interpret such expressions by first simplifying them into canonical uun-of-productt
form to enable modular interpretation. The natural language descriptions are then generated by applying predefined template rules that map each kernel component and its parameters descriptive phrases. contrast, our CAKE framework, the LLM itself acts as the interpreter of the kernel structures, producing natural language descriptions directly and automatically "for free" without the need for explicit simplification or handcrafted template rules (see Fig. [4 This level of interpretability highlights key advantage of our CAKE framework. The LLM's contextual understanding and access to broad domain knowledge enable it not only to evolve expressive kernels, but also to explain them through meaningful natural language descriptions. Unlike traditional methods that treat kernel learning as a purely numerical optimization task, CAKE generates interpretable hypotheses that reveal insights into the structure of the underlying "black-box" objective function.
CAKE's Interpretation
Response:
Kernel: (SE x PER) (LIN RQ) (SE M5)
Analysis: The observations can described using combination three distinct patterns. The first term, SE x PER, captures recurring fluctuations in the performance across the (C, gamma) space. High ganma values lead to overfitting, but increasing C mitigates this effect, resulting in overfitting-roooe recovvrry cycle. The second term, LIN x RQ, models general inerease in performance stronger regularization improves generalization. However, the RQ component allows the model adapt to varying levels smoothness which is useful near extreme gamma values. The third term, M5 x SE, combines the M5's 'ss robustness to non-smooth changes with SE's smoothness to capture complex, localized variations in performance that are not easily explained by global trends periodicity alone, such sharp drops in accuracy when both and gamma are too large.
increases, as
Figure 14:
Intepretation for the SVM hyperparameter optimization task on the credit__ dataset.
C.5 Choice of LLM
In this section, we analyze the impact of the LLM choice on the performance of our method. We evaluated three open-source models Qwen's quen-2.5-7b. DeepSeek's deepsee--r1-disttl---ees-
and Meta's 1lama-3. 1--aa-..--bb---- 1-8-----oggsiidd three closed-source models: Google's aamini-2.0-flase Anthropic's claude-3. 3laude-3.5-haiku,, 5-hakk,, and OpenAl's got-4o-mini, the HPOBench functions described in Section 5.1 The results in Table 6 that, despite some variations in performance across different HPOBench functions, there is a clear trend: our method performs better when using more recent and capable LLMs. We also observe performance boost with trepseek-ri-distille tawisiillieenn---- 7b, suggesting that reasoning- -bsed models can further improve results. However, this gain comes at the cost of longer inference time, a trade-off we plan to explore in future work. Overall, these results indicate that as L.LMs become more advanced, method naturally benefits from their improved capabilities.
C.6 Choice of Acquisition Function
We test the robustness of CAKE under different choice acquisition functions, including El 23 upper confidence bound (UCB) [44, and Thompson sampling (TS) (47). While each acquisition function embodies different exploration-exploittiie trade-off (i.e., TS exhibits slightly higher stan- dard errors due to its stochastic nature while CAKE-UCB occasionally underperforms or outperforms
25

---

Table Average accuracy (f) on HPOBencc over 20 random seeds for different LLMs. The values in parentheses indicate the standard errors.
LLM
LR
SVM
RF
XGB
MLP
0.7720
0.8520
0.8810
0.9030
0.8610
qwen-2.5-7b
(0.0310)
(0.0180)
(0.0260)
(0.0380)
(0.0330)
0.8122
0.8630
0.8964
0.9211
0.8692
deepseek-11-distill-qwwea
(0.0210)
(0.0141)
(0.0234)
(0.0320)
(0.0292)
0.7815
0.8581
0.8873
0.9110
0.8652
1lama-3.1-8b
(0.0290)
(0.0177)
(0.0252)
(0.0355)
(0.0312)
0.8177
0,8655
0,8982
0.9240
0.8712
claude-3.5-haike
(0.0231)
(0.0130)
(0.0242)
(0.0331)
(0.0301)
0.8253
0,8720
0,9056
0.9310
0.8780
gemini-2.0-flash
(0.0204)
(0.0121)
(0.0223)
(0.0303)
(0.0281)
0.8188
0.8663
0,8991
0.9257
0.8722
gpt-4o-mini
(0.0220)
(0.0130)
(0.0250)
(0.0340)
(0.0310)
Table 7: Average normalized regret (L) + standard error over 20 random seeds for CAKE with different acquisition functions. Value that are not significantly different from the lowest average regret for each function are bolded.
Function
CAKE-EI
CAKE-UCB
CAKE-TS
Ackley-2
0.0783 +: 0.0203
0.0812 + 0.0215
0.0921 + 0.0287
Ackley-5
0.1732 + 0.0250
0.1654 + 0.0262
0.1789 0.0310
Beale
0.2565 + 0.0786
0.2488 + 0.0810
0.2720 + 0.0920
Branin
0.0070 + 0.0534
0.0065 + 0.0510
0.0082 1 0.0601
Dropwave
0,4690 + 0.0538
0.4820 + 0.0560
0.5010 + 0.0650
Egg holder
0.1241 = 0.0541
0.1350 + 0.0570
0.1298 = 0.0620
Griewank-2
0.0267 = 0,0256
0.0275 0.0260
0.0310 + 0.0305
Griewank-5
0.0185 = 0.0133
0.0190 + 0.0140
0.0195 0.0180
Hartmann
0.0001 + 0.0529
0.0003 + 0.0515
0.0005 + 0.0580
Levy-2
0.0353 + 0.0197
0.0360 + 0.0205
0.0402 0.0240
Levy-3
0.0505 + 0.0190
0.0580 + 0.0200
0.0520 + 0.0235
Rastrigin-2
0.3341 + 0.0468
0.3520 + 0.0490
0.3650 =: 0.0580
Rastrigin-4
0.3128 + 0.0499
0.3150 + 0.0510
0.3300 + 0.0570
Rosenbrock
0.0483 + 0.0531
0.0490 + 0.0540
0.0510 + 0.0600
Six-Hump Camel
0.1015 = 0.0669
0.0920 0.0680
0.1030 0.0750
EI due to its optimism bias), CAKE consistently achieves comparable performance across all variants. As shown in table below, the performance of CAKE-UCB and CAKE-TS remains close to that of CAKE-EI across a diverse set of benchmark functions. To rigorously assess statistical equivalence, conducted one-sided paired Wilcoxon signed-rank test (at the 5% significance level) comparing CAKE-FI against CAKE-UCB and CAKE-TS across multiple random seeds. The results indicate that, on the majority of functions, differences are not statistically significant. Overall, these results confirm that CAKE's effectiveness is not sensitive to the specific choice of acquisition function.
C.7 Quantitative Analysis
To rigorously support our claim in Section 5.1 that CAKE excels in the early stages of optimization, we conducted a quantitative analysis based on normalized improvement, defined as:
f: fo
Normalized improvement at trial t
(27)
fo
where So the initial performance, ft the best performance at trial 6, and f" is the final performance at 100% budget. This measures how much of the total progress CAKE achieves up to the (-th iteration. As shown in Table CAKE achieves 67.5% of its total improvement within just 25% of the budget, on average. By 50%, reaches over 83% of its final improvement, and by 75%, it is nearly converged.
26

---

Table Normalized improvement on HPOBench at different budget level.
Budget
25%
50%
75%
LR
0.6183
0.8065
0.8401
SVM
0.7007
0.8244
0.9481
RF
0.6244
0.7685
0.8646
XGB
0.6914
0.8643
0.9767
ML.
0.7394
0.9155
0.9742
Average
0.6749
0.8358
0.9207
This demonstrates that CAKE rapidly identifies effective kernels and drives fast early progress, making it particularly effective in data-scarce regimes.
D Limitations and Future Work
Computational cost. While CAKE can be applied entirely in-context and does not require any fine-tuning, using LLMs for inference may result in a larger computational footprint compared to traditional BO methods (see Appendix C.2p. Despite this, our findings indicate that CAKE trades this off with improved sample efficiency, which is a particularly desirable property for black-box optimization tasks. This suggests the potential for integrating CAKE with more computationally efficient approaches, such as deploying it in the earlier stage of the optimization process.
Data Contamination. We acknowledge ledge the possibility that LLMs may have been exposed to scientific literature or code related to common optimization benchmarks during pre-training. However, we argue that data contamination is unlikely meaningfully affect our results. While the LLM may possess general knowledge about kernels or synthetic functions, our approach to adaptive kernel evolution in BO novel and there no evidence that the specific kernel expressions or the optimization trajectory, exist in any public dataset or text. Thus, we believe that the observed performance stems from in-context adaptation, not memorization. This is further supported by our ablation study Section B.4which shows that removing the observed data from the prompt leads to significant performance degradation, confirming that the LLM relies on in-context learning rather than prior knowledge alone.
Generalized kernel grammar: While we focus on addition and multiplication as initial proof-of- concept operators, these operators are in fact good enough form rich and expressive space of kernels. For example, by only using these operations, we can construct polynomial kernels to capture non-linear patterns as well
multi-dimensional kernels model interactions among input features [1]. However, we would like that to note that the kernel grammar can be extended using other operators that preserve the closure properties of kernel functions, such as convolution, composition, and affine transformations 42) We aim to explore these possibilities further in future work.
Extension to broader ML tasks. Our long-term goal is to develop universal adaptive kernel method that can be applied across a wide range of ML. tasks. While the current work focuses on BO, the underlying idea of using an LLM guide kernel evolution is not task-specifie. We believe that CAKE can be easily adapted for other kernel-based methods such as SVM-hased regression and classification, kernel principal component analysis, and metric learning with kernels. By leveraging task-specific performance signals, CAKE can automate and enhance kernel design across a variety of kernel-based methods, demonstrating its broader potential improve ML applications.
27

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification:
All claims are discussed in the main text.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results
be expected to generalize other settings.
is fine to include aspirational goals as motivation as long is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification:
The limitations of the proposed are clearly discussed in Appendix D
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification. asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low images are taken in low lighting. Or speech-to-text system might be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach address problems of privacy and faimess.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory assumptions and proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [NA]
28

---

Justification:
The current paper does not include any theoretical results.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced the statement of any theorems.
The proofs can either appear the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Question:
Answer: [Yes]
Justification: Details are provided in the main text and Appendix The code is also available online.
Guidelines:
The answer NA means that the paper does not include experiments. If the paper includes experiments, No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make it clear how reproduce that algorithm. (b) If the contribution is primarily
model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome deseribe the particular way they provide for reproducibility, In the case of closed-source models, it may be that access to the model is limited in some way (e.g., registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
29

---

Answer: [Yes]
Justification:
The code is available online at http::://ggthbb..c.//rriiiiieeee
cake
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://nips... cc/
for more details.
While we encourage the release of code and data, we understand that this might not be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to to reproduce the results. See the NeurlPS code and data submission guidelines (https
ssicnpolicy) for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: Details are provided the text and in Appendix B Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, or as supplemental material.
7. Experiment statistical significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [Yes]
Justification: The figures in Section incclude error bars based on the standard error across multiple independent runs with different random seeds.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.
The factors of variability that the error bars capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
30

---

It should be clear whether the error bar is the standard deviation or the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that out of range (e.g. negative error rates).
If error bars are reported in tables or plots, The authors should explain the text how they were calculated and reference the corresponding figures tables in the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed reproduce the experiments?
Question:
Answer: [Yes]
Justification:
The paper provides details regarding the computing resources in Appendix
Guidelines:
The answer NA means that the paper does include experiments.
The paper should indicate the type of compute workers CPU GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary failed experiments that didn't make into the paper).
9. Code of ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips. hwtpp///wb.ctttitcccciiiieeeeeee
Answer: [Yes]
Justification: The research was conducted in compliance with the NeurlPS Code of Ethics. Additionally, no deprecated datasets were used, and all external assets (e.g., OpenMI. datasets) were appropriately cited and used under permissible licenses.
Guidelines:
The answer NA means that the authors have reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., if there is special consid- eration due to laws or regulations their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [NA]
Justification: The proposed method does not directly lead to societal impacts that we feel must be highlighted beyond algorithmic improvements.
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No. they should explain why their work has no societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
31

---

The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification: The proposed method does not lend itself to this type of misuse.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Question: Are the creators or original owners of assets (e.g.. code, data, models), used in the paper. properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification: The paper uses publicly available datasets (e.g., OpenML) and benchmarks, all of which are properly cited with references their original sources.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (c.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the batasets.papersstt paperswi thcode. cww/ datasets package should be provided. For popular datasets,
has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
32

---

If this information is not available online, the authors are encouraged to reach out to the asset's 's creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [Yes]
Justification:
The code is well-documented and available at the anonymized repository https://github.com httpa///.....ccccacaaaaaaaaaaaaa
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/mooel as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create anonymized URL include anonymized zip file.
14. Crowdsourcing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given participants and screenshots, if applicable, as well as details about compensation (if any)?
Answer: [NA]
Justification:
The paper does not involve crowdsoureing or research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included the main paper.
According to the NeurIPS Code Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals or equivalent for research with human subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: The paper does not involve crowdsourcing or research with human subjects. Therefore, no IRB approval equivalent required.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurlPS Code of Ethics and the guidelines for their institution.
For initial submissions, not include any information that would break anonymity (if applicable), such the institution conducting the review.
33

---

16. Declaration of LLM usage
Question: Does the paper describe the usage of LLMs if it is an important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing, editing, or formatting purposes and does not impact the core methodology. scientific rigorousness, originality of the research, declaration is not required.
Answer: [Yes]
Justification: The use of LLMs is detailed in Section 3 and Appendix
B
Guidelines:
The answer NA means that the core method development in this research does not involve LLMs as any important, original, non-standard components.
Please refer to L.LM policy httpp///aaaapppc.coooooooerereee for what should or should not be described.
34


Reviews:
[{"review_id": "bAZbA8zzs5", "reviewer": "Reviewer_RoKJ", "summary": "- This paper proposes a principled framework using Large Language Models (LLMs) to tackle the limitations of heuristics-based selection of kernels from a standard bag of popular kernels used for Bayesian Optimisation (BO). The generalisation performance of a (GP) surrogate is crucial in BO, and having a heuristically designed surrogate often leads to poor performance, especially when the underlying structure of the objective function does not match the kernel assumptions such as continuity and smoothness.\n\n- Authors propose Context-Aware Kernel Evolution (CAKE), a novel method to use LLMs as genetic operators (from evolutionary algorithms) to adaptively generate and refine expressive kernels that are crucial for the success of the BO. Authors use the notion of kernel grammar space introduced in the prior literature to define a flexible space of kernels, which are then genetically evolved by LLMs. \n\n- To choose the most effective kernel, the authors introduced  BIC-Acquisition Kernel Ranking (BAKER), a Bayesian Information Criterion (BIC) kernel ranking scheme that jointly ranks the kernel based on the model fit and acquisition function utility.\n\n- The empirical results, along with the ablation studies, demonstrate its superiority against fixed, adaptive, and compositional kernel methods from the state-of-the-art literature across various tasks such as hyperparameter optimisation, robot controller tuning, and photonic chip design.", "questions": "- Authors do not discuss prior work - Kernel Identification Through Transformers (KITT) [3], which is another Transformer-based kernel selection scheme for GPs. Could the authors clarify why it wasn’t included as a baseline, and how CAKE differs? Can we empirically compare?\n\n- When LLMs genetically evolve kernel functions, are kernel hyperparameters assumed to be optimal? If not, how do we ensure the kernel fits the data properly? via fitness score? Because of the continuous nature of the hyperparameter space, could LLM pinpoint the exact values for the hyperparameters of the individual kernels?\n\n- The Baker formulation mentioned in Eq. (1) needs to be discussed in detail. If I understand correctly, lower BIC values will result in higher weights and higher BIC values will result in lower weights. Having that said, could there be a situation where a kernel with low weight and high acquisition value could be selected according to this optimisation? If so, then the selection will be sub-optimal. \n\n- In Figure 5, SE outperforms BIC in some cases. Does this suggest that BIC may not be the best fit criterion for certain kernels or tasks?\n\n- Would an adaptive or decaying mutation probability (something like a cooling schedule) improve performance compared to a fixed $p_m$, especially as optimisation progress?\n\n\n- **Minor Comments**\n  - Typo: In Line 242: \"In robot pushing task..\"\n  - Typo: In Line 308: \"..Observations are scarce.\"", "limitations": "Authors have discussed the limitations of CAKE and BAKER in the Appendix. \n\n- CAKE incurs high per iteration costs due to repeated GP fitting and LLM inference, and relies on commercial LLM APIs.\n\n- Currently, the method restricts kernel construction to additive/multiplicative grammar, omitting richer functional or operator-based kernel spaces. \n- The paper offers no convergence or regret guarantees, and does not analyse the robustness of the method to prompt design and kernel hyperparameter tuning.", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 4, "significance": 3, "originality": 3, "strengths_and_weaknesses": "**Strengths**\n\n- This paper presented a novel use of LLMs (few-shot prompting) for in-context kernel evolution for BO, addressing a key aspect in adaptive surrogate modelling. The use of LLMs in the BO community has gained attention in the recent past [1,2].\n\n- By Leveraging a grammar-based composition of base kernels, CAKE enables exploration of a rich and interpretable space of kernel structures that can adapt to diverse functional patterns, in contrast to a fixed selection of kernels from a popular list based on some heuristics and expertise. \n\n- Using LLMs to produce natural language explanations for kernel expressions suggested is compelling, as it creates a dialogue between the ML practitioner and LLMs, thereby providing insights into the learned surrogate models to promote trust and understanding.\n\n- The method is extensively evaluated across a wide range of tasks such as hyperparameter optimisation, controller tuning, and multi-objective photonic chip design. The empirical results demonstrate that CAKE outperforms competing approaches in the early phases of optimisation, reducing the number of evaluations needed, a much-needed aspect in the optimisation of expensive black-box functions.\n\n- The modularity of the CAKE framework enables it to be extended beyond BO to other kernel-based methods such as SVMs and kernel PCA.\n\n- The paper is very well written in most parts and clearly understandable.\n\n\n**Weaknesses**\n\n- The use of LLMs for adaptive kernel generation is an interesting idea; however, the core idea of leveraging Transformers to discover kernels is not entirely new. For instance, [3] explored Transformer-based kernel selection in Gaussian Processes. \n\n- This paper focuses primarily on kernel selection and composition strategies using discrete kernel grammar-based search, but does not discuss another line of work that optimises kernels directly in function space. For example, \"Function-space Distributions over Kernels\" [4] and \"Kernel Functional Optimisation\" [5] treat kernels as learnable objects via functional representations. The authors are encouraged to discuss this related literature and possibly compare it to demonstrate expressiveness, scalability, and sample efficiency.\n\n- Authors have only experimented with commercial OpenAI’s GPT-4o-mini via API calls, introducing concerns around reproducibility, cost, and accessibility. Any experiments with open-source LLMs would strengthen the empirical claims made.\n\n- This paper does not provide any formal guarantees or analysis of convergence, regret bounds, or expressivity of the generated kernels, which could help build confidence in its broader applicability. The use of transformer architecture in the kernel generation process could create this bottleneck.\n\n- The performance of CAKE heavily depends on prompt engineering and the LLM model version. Although the prompts are engineered well by the authors, the paper does not discuss the robustness regarding different prompt formats or LLM models/variants.\n\n- The authors have missed discussing a few key components of the proposed framework. Refer to the questions below.\n\n\n**References**\n\n[1] Yang, Zhuo, Lingli Ge, Dong Han, Tianfan Fu, and Yuqiang Li. \"Reasoning BO: Enhancing Bayesian Optimisation with Long-Context Reasoning Power of LLMs.\" arXiv preprint arXiv:2505.12833 (2025).\n\n[2] Chen, Guojin, Keren Zhu, Seunggeun Kim, Hanqing Zhu, Yao Lai, Bei Yu, and David Z. Pan. \"LLM-enhanced Bayesian optimisation for efficient analogue layout constraint generation.\" arXiv preprint arXiv:2406.05250 (2024).\n\n[3] Simpson, Fergus, Ian Davies, Vidhi Lalchand, Alessandro Vullo, Nicolas Durrande, and Carl Edward Rasmussen. \"Kernel identification through transformers.\" Advances in Neural Information Processing Systems 34 (2021).\n\n[4] Benton, Gregory, et al. \"Function-space distributions over kernels.\" Advances in Neural Information Processing Systems 32 (2019).\n\n[5] Anjanapura Venkatesh, Arun Kumar, et al. \"Kernel functional optimisation.\" Advances in Neural Information Processing Systems 34 (2021): 4725-4737.", "was_revised": true, "final_justification": "I have read the rebuttal, and the authors have satisfactorily addressed my concerns. Accordingly, I am increasing my scores.", "rebuttal": "*Remark:* \n- *(W1) means the 1st weakness, (Q1) means the 1st question.*\n- *Due to character limits, we kindly invite you to refer to our responses for other reviewers' similar questions. We will use* **ref. \"Response Title\"@Reviewer**. *Thank you for your understanding!*\n---\n# Comparison with KITT [3]\n\n> (W1) The use of LLMs for...kernel selection in Gaussian Processes. \n> \n> (Q1) Authors do not discuss prior...Can we empirically compare?\n\nWe summarize the main differences between CAKE and KITT as follows:\n\n- **Different objectives**: CAKE is designed for adaptive kernel learning in BO, which requires balancing exploration and exploitation. Meanwhile, KITT is developed for regression, which fouced on fitting and prediction accuracy.\n- **Usability**: KITT uses a fixed kernel vocabulary and requires fine-tuning. CAKE leverages LLMs as plug-and-play tools, usable without fine-tuning and accessible to non-experts.\n \nWe did consider including KITT during the rebuttal phase, but ultimately could not due to practical and methodological constraints. KITT requires large-scale labeled data (e.g., 200K samples) and significant computational resources (e.g., hours on Tesla V100 GPUs). More critically, it is tailored to regression and has not been adapted or evaluated for BO. Adapting it to BO would require substantial retraining on BO tasks and optimization trajectories. We instead focused on established, directly applicable BO baselines to ensure a fair and reproducible comparison.\n\n---\n# Comparison with FKL [4] and KFO [5]\n\n> (W2) This paper focuses primarily on...scalability, and sample efficiency.\n\nWe first clarify the fundamental differences between CAKE and functional kernel methods (FKL/KFO):\n\n- CAKE performs **discrete** search over **parametric** kernel compositions, which preserves symbolic kernel structures for interpretability. \n- FKL/KFO operate in **continuous**, **non-parametric** function spaces, which sacrifice interpretability for flexibility.\n\nHence, their kernel search spaces and optimization methods differ significantly from CAKE, making direct empirical comparison challenging and somewhat orthogonal in objectives.\n\nNonetheless, we appreciate the reviewer's suggestion and conducted additional experiments comparing CAKE with KFO and both FKL variants on UCI datasets. Using the same benchmark and negative log-likelihood as in [5], the results show that CAKE is highly competitive with these state-of-the-art functional kernel methods and outperforms KFO on several datasets.\n\n| Dataset   | SE                           | KFO                           | FKL-Shared         | FKL-Separate       | CAKE                           |\n| --------- | ---------------------------- | ----------------------------- | ------------------ | ------------------ | ------------------------------ |\n| Fertility | $\\underline{-3.90 \\pm 1.76}$ | $5.15 \\pm 2.95$               | $-2.70 \\pm 1.25$   | $-1.83 \\pm 3.30$   | $\\mathbf{-4.60 \\pm 1.50}$      |\n| Yacht     | $1.65 \\pm 76.47$             | $\\mathbf{-34.60 \\pm 1.60}$    | $-14.52 \\pm 4.70$  | $-15.70 \\pm 8.20$  | $\\underline{-32.30 \\pm 2.33}$  |\n| Slump     | $36.30 \\pm 7.93$             | $\\mathbf{-3.01 \\pm 1.70}$     | $4.38 \\pm 1.33$    | $59.40 \\pm 12.87$  | $\\underline{-2.50 \\pm 1.46}$   |\n| Boston    | $139.61 \\pm 11.50$           | $\\underline{-24.70 \\pm 4.20}$ | $122.60 \\pm 3.91$  | $143.70 \\pm 5.71$  | $\\mathbf{-25.00 \\pm 3.50}$     |\n| Auto      | $96.18 \\pm 8.02$             | $\\underline{-8.78 \\pm 1.20}$  | $101.70 \\pm 19.30$ | $98.94 \\pm 6.10$   | $\\mathbf{-9.00 \\pm 1.51}$      |\n| Airfoil   | $358.93 \\pm 8.93$            | $\\mathbf{-204.10 \\pm 4.50}$   | $270.00 \\pm 28.40$ | $284.80 \\pm 48.70$ | $\\underline{-195.00 \\pm 5.10}$ |\n\n---\n# Try other LLMs\n\n> (W3) Authors have only experimented...the empirical claims made.\n\nWe have expanded our experiments to include 3 open-source LLMs (**ref. \"Try other LLMs\"@esv8**). Results show that open-source models can achieve competitive performance, with `deepseek-r1-distill-qwen-7b` even outperforming several commercial models. This demonstrates that our approach does not rely on proprietary APIs to work well.\n\n---\n# Any theoretical analysis?\n\n> (W4) This paper does not provide...create this bottleneck.\n\nWe have actually conducted some analysis through the Bayesian regret framework. By assuming that the underlying objective function is a sample path of a zero-mean GP or a function in the reproducing kernel Hilbert space (RKHS) induced by the obtained kernel function, we can leverage existing results to show that the cumulative Bayesian regret can be bounded as the kernel is refined over iterations using the LLM. However, due to the adaptive nature and complex form of the kernel function across iterations, we are still unable to quantify the convergence speed, which our experiments nevertheless show to be significantly faster than that of the state-of-the-art methods. We can include our current analysis results in the Appendix of the updated draft.\n\nNevertheless, our primary goal is to highlight the novel kernel search method and its superior performance in BO. To this end, we placed a greater emphasis on conducting extensive empirical analyses and ablation studies to rigorously evaluate the contributions of various components, including prompt design and parameter choices. In the Appendix, we present an empirical study of the fitness distribution across iterations, which provides a practical justification for why the LLM can effectively serve as a genetic operator that evolves kernels toward higher fitness. Additionally, our case study on the expressivity and interpretability of the generated kernels offers qualitative evidence of our method's effectiveness.\n\n---\n# Robustness with different prompts or LLM variants\n\n> (W5) The performance of CAKE...LLM models/variants.\n\nWe have added an ablation study (**ref. \"Does LLM reasoning matter?\"@esv8**) to assess the contribution of each prompt component:\n\n- **Removing context** harms kernel adaptation, showing the LLM relies on observed data to generate meaningful structures.\n- **Omitting task instructions** leads to hallucinated or invalid kernels, degrading performance.\n- **Removing the reasoning step** slightly reduces performance, suggesting explanations improve self-consistency and generation quality\n\nThese results demonstrate that our prompt design is purposeful, with each element playing a critical role in ensuring correctness and coherence. While CAKE involves some prompt engineering, this is standard in LLM-based methods, and our ablation clarifies which components matter and why.\n\nRegarding the robustness with different LLM variants, CAKE actually performs better with stronger LLMs (**ref. \"Try other LLMs\"@esv8**), indicating it naturally benefits from advances in LLM capabilities and remains forward-compatible with future models.\n\n---\n# Kernel hyperparameters\n\n> (Q2) When LLMs genetically evolve...hyperparameters of the individual kernels?\n\nWe did not expect the LLM to pinpoint exact hyperparameter values, as we did not equip it with any external tools to do so. Its role is to evolve the symbolic kernel expression. Once a kernel expression is proposed, its hyperparameters are optimized via standard maximum likelihood estimation using GPyTorch, and fitness is then evaluated under the properly tuned hyperparameters.\n\n---\n# Explanation to Eq. (1)\n\n> (Q3) The Baker formulation mentioned...will be sub-optimal.\n\nIt is indeed possible for a poorly fitting kernel to propose an overly optimistic expected improvement due to overestimated uncertainty or a misspecified mean function. If the kernel selection were based solely on acquisition value, such overconfident but poorly calibrated models could be repeatedly favored, thus leading to suboptimal querying behavior. To mitigate this risk, BAKER computes a weighted score by taking the product of each kernel's weight and its corresponding acquisition value. By maximizing this product across the kernel population, BAKER inherently balances the model fit (as measured by the weight) and query utility (as measured by the acquisition value). Consequently, even if a poorly fitting kernel proposes an high acquisition value, its low weight acts as a natural regularizer to suppress its influence. In this way, kernels with both high weights and high acquisition values are more likely to be selected.\n\n---\n# Explanation to Fig.5\n\n> (Q4) In Figure 5, SE outperforms...certain kernels or tasks?\n\nThe fact that the fixed SE kernel sometimes outperforms the BIC-based adaptive method highlights a key limitation: selecting kernels solely based on model fit does not ensure better optimization performance. This motivates our design of BAKER, which jointly considers fit (via BIC) and utility (via acquisition value). As shown in our ablation study (Section 5.4), using either criterion alone leads to suboptimal results, while their combination in BAKER yields more robust performance across tasks.\n\n---\n# Can adaptive mutation probability help?\n\n> (Q5) Would an adaptive or decaying...as optimisation progress?\n\nAn adaptive or decaying mutation probability could enhance performance by promoting exploration early and exploitation later. Our sensitivity analysis (**ref. \"Hyperparameter sensitivity analysis\"@fLo9**) shows that while higher $p_m$ increases diversity and prevents premature convergence, overly high values can disrupt promising kernels. This trade-off suggests a decaying schedule may offer better control. In this work, we use a fixed $p_m = 0.7$ for the sake of reproducibility as it provides a robust and balanced performance across tasks. Although we did not implement adaptive $p_m$ due to time constraints, we see it as a promising direction for future improvements.\n\n---\n\n> **Minor Comments**\n>    - Typo: In Line 242: \"In robot pushing task..\"\n>    - Typo: In Line 308: \"..Observations are scarce.\"\n\nThanks! We will correct them in the camera-ready version."}, {"review_id": "OPk5ud29fX", "reviewer": "Reviewer_2phH", "summary": "The authors present CAKE, a method leveraging LLMs to efficiently explore the space of kernels for GPs in BO. The LLM is tasked to combine and mutate a population of kernels with each other in order to discover more suited kernels to the blackbox objective.\n\nAlong with this, they also introduce a selection criterion called BAKER which relies on the Bayesian Information Criterion to select the most suited kernels in the population to continue to the next generation of mutation.\n\nFinally they conduct a series of experiments comparing their method CAKE+BAKER with a wide range of baselines, showing improvements on all objective functions.", "questions": "1. Why use BIC and not directly marginal log-likelihood to measure fitness? It is a common metric used in GP literature to measure how good the fit is. Also you mention that you use `BoTorch` and I think there is a restart mechanism for GP training in which the optimization can be restarted mutliple times to avoid local optima (common in mll optimization), do you use that mechanism? Because any fitness metric would be sensitive to that as well.\n\n2. I am not sure I understood exactly how the fitness computation is done, is it simply that after each new observation collected during the BO you fit again all GPs in the curernt population and keep the best ones? Or do you actually \"replay\" the trajectory with each GP to check which one would have discovered the current best point with the highest probability?\n\n3. Have you tried adding more context to the LLM, for example some domain knowledge of the task? Or is the LLM purely reasoning about kernels and fitness metrics?\n\n4. What is the shaded area in the Figures 4 to 6? is it the standard deviation or the standard error of the mean? It would be best have some statistical tests of significance between CAKE+BAKER and the top few baselines (pairwise). It is easy to do so at regular intervals and it is informative to see at what step are the methods significantly different.", "limitations": "yes", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 4, "significance": 2, "originality": 2, "strengths_and_weaknesses": "### Strengths\nIn general the paper is very easy to read and quite clear. The objectives are well-motivated and framed in the context of BO using LLMs.\n\nThe interface between the GP mechanism and the LLM is also clearly explained and relatively simple to grasp. The authors restrict the search space of the kernels to a certain grammar, which is very reasonable and grounded in previous literature.\n\nThe next point selection mechanism, i.e. expected weighted by fitness, also makes perfect sense in this context.\n\nFinally, the experiments are very well conducted and the results are strong.\n\n### Weaknesses\nI think the kernel space is a bit restrictive. Although it does not seem to impact the results too much, more complex tasks could benefit from a wider range of kernel transformations. This would make the search space richer, even though it might be harder to explore too.\n\nAnother point is the information given to the LLM, I find the prompts almost too simple. Assuming the LLM has knowledge of kernels for GPs and is able to mutate/combine them already, could it not benefit from more in-context examples? \n\nFor the baselines, the authors could add another LLM-BO baseline. Even though I understand that this is not exactly the point of the paper and other LLM-BO related work do not explore the kernel space for the GP but leverage the LLMs in different ways, it is still somewhat unfair in my opinion to compare LLM-based methods to more classical baselines. I appreciate that the authors have made an extensive effort to run many baselines, but I think it would make the paper stronger if there was a baseline such as LLAMBO, BOPRO, ...\n\n**Lines 216-218**: you mention that CAKE excels in the earlier stages of the optimization, I think there needs to be more analysis on that claim. Although it might be visualy clear in Figures 4-6, it would make sense to analyse the early optimization stages more carefully.\n\nThe biggest weakness of this work in my opinion is the cost of such a method. Given that the authors have discussed it, I think they are genuine in their analysis of the computational cost of maintaining a population of kernels/GPs at each step. This slows down the whole process and could be a bottleneck in practice.\n\nFinally, it would be best to use open-source LLMs for reproducibility.", "was_revised": true, "final_justification": "After fruitful rebuttal from the authors I am confident this paper is good for acceptance. I think other reviewers share my verdict.\nThe authors also promised to include clarifications and extra experiments in the final version of the paper, which will contribute to the quality of their work.", "rebuttal": "*Remark:* \n- *(W1) means the 1st weakness, (Q1) means the 1st question.*\n- *Due to character limits, we kindly invite you to refer to our responses for other reviewers' similar questions. We will use* **ref. \"Response Title\"@Reviewer**. *Thank you for your understanding!*\n---\n# Kernel space\n\n> (W1) I think the kernel ... to explore too.\n\nWe agree that incorporating more base kernels and operators can enrich the search space. However, in our work, we focus on addition and multiplication as initial proof-of-concept operators as they already yield a rich, expressive kernel space—e.g., enabling polynomial kernels that capture complex non-linear patterns and multi-dimensional kernels that model interactions among input features. Moreover, our method supports a generalized kernel grammar, allowing easy integration of other valid operators (e.g., convolution, composition). We consider this an important and promising direction for future work. We also appreciate your observation that a richer kernel space inherently increases the complexity of exploration. Balancing expressiveness and tractability in kernel search is indeed a challenging problem and an interesting avenue for future research.\n\n---\n# Prompt design\n\n> (W2) Another point is ... in-context examples?\n\nIn designing our prompts, we intentionally adopted a simple yet principled approach to ensure reproducibility and generalization across diverse tasks. Enriching the prompts with more in-context examples (such as successful kernels from similar tasks or synthetic patterns) could potentially improve performance and is a promising direction for future work, particularly in meta-learning or multi-task settings. However, in the current few-shot, single-task BO setting, our current prompt design strikes an effective balance between simplicity and performance (**ref. \"Does LLM reasoning matter?\"@esv8**), avoiding the risks of prompt overload or distraction from the task in hand.\n\n---\n# LLM-BO Baseline\n\n> (W3) For the baselines ... such as LLAMBO, BOPRO, ...\n\nWe clarify that methods like LLAMBO and BOPRO operate in a fundamentally different paradigm from CAKE. These methods use LLMs to augment the BO pipeline (e.g., warm-starting or generating candidates) without learning or evolving kernel structures. In contrast, CAKE focuses on improving the GP surrogate by evolving its kernel via an LLM-guided process. Our goal is to enhance the kernel itself, not to replace BO components with LLM heuristics. Thus, comparing CAKE to LLM-augmented BO methods would conflate distinct objectives. For a fair evaluation, we compare CAKE against GP-based BO baselines that share the same underlying framework and goals.\n\n---\n# Early stage analysis\n\n> (W4) **Lines 216-218**: you ... stages more carefully.\n\nTo rigorously support our claim that CAKE excels in the early stages of optimization, we conducted a quantitative analysis based on *normalized improvement*:\n\n$$\n\\text{Normalized improvement at trial } t = \\frac{f_t - f_0}{f^* - f_0},\n$$\n\nwhere $f_0$ is the initial performance, $f_t$ is the best performance at trial $t$, and $f^*$ is the final performance at 100\\% budget. This measures how much of the total progress CAKE achieves early on. In the HPOBench tasks, CAKE achieves 67.5\\% of its total improvement within just 25\\% of the budget, on average. By 50\\%, it reaches over 83\\% of its final improvement, and by 75\\%, it is nearly converged.\n\n| Budget  |  25%   |  50%   |  75%   |\n| :-----: | :----: | :----: | :----: |\n|   LR    | 0.6183 | 0.8065 | 0.8401 |\n|   SVM   | 0.7007 | 0.8244 | 0.9481 |\n|   RF    | 0.6244 | 0.7685 | 0.8646 |\n|   XGB   | 0.6914 | 0.8643 | 0.9767 |\n|   MLP   | 0.7394 | 0.9155 | 0.9742 |\n| Average | 0.6749 | 0.8358 | 0.9207 |\n\n---\n# Computational cost\n\n> (W5) The biggest weakness ... a bottleneck in practice.\n\nFirst, we note that the computational overhead of maintaining a population of GPs is **not unique to CAKE**. Compositional and ensemble-based methods also require fitting and managing multiple GP models per iteration.\n\nSecond, while CAKE evaluates multiple kernels at each iteration, the **training of these GPs are independent and parallelizable**. In practice, the kernel population can be trained in parallel across multiple cores or GPUs to significantly reduce the wall-clock time.\n\nMost importantly, as highlighted in Appendix C.2, in most real-world black-box optimization tasks, the cost of evaluating the objective function far exceeds the cost of surrogate model fitting. For example, running a physics-based simulation can take hours, while fitting a population of GPs takes seconds. In such settings, sample efficiency, or reducing the number of function evaluations, is far more critical. We note that our method excels in this regard, as our experiments showed that it could converge to high-performing solutions in fewer iterations.\n\n---\n# Try other LLMs\n\n> (W6) Finally, it would ... for reproducibility.\n\nWe have expanded our experiments to include 3 open-source LLMs (**ref. \"Try other LLMs\"@esv8**). Results show that open-source models can achieve competitive performance, with `deepseek-r1-distill-qwen-7b` even outperforming several commercial models. This shows that strong results are also achievable with open-source models.\n\n---\n# Why BIC?\n\n> (Q1) Why use BIC ... good the fit is.\n\nBoth the MLL and BIC are valid metrics for model evaluation. We deliberately choose BIC because it imposes an **explicit penalty** on model complexity proportional to the number of hyperparameters, which provides a more direct control of overfitting in our context. While MLL also includes an **implicit complexity penalty** via the log determinant term, it can sometimes favor overly complex kernels especially when the hyperparameter optimization converges to a local minima. In contrast, BIC’s penalty increases with sample size and hyperparameter count, which discourages unnecessary complexity as more data becomes available.\n \n---\n# BoTorch multi-start\n\n> (Q1) ... Also you mention that ... sensitive to that as well.\n\nWe did employ BoTorch's multi-start optimization procedure to mitigate convergence to poor local optima. We found this improves robustness in estimating the hyperparameters, which in turn helps stabilize the fitness evaluation.\n\n---\n# GP retraining\n\n> (Q2) I am not sure ... the highest probability?\n\nWe do not replay past decisions or evaluate counterfactual trajectories (e.g., \"which kernel would have found the best point earlier?\"). Instead, we assess each kernel on its current ability to model the entire observed data. After each new observation is collected during the BO, we retrain all the GP models in the current kernel population using the updated dataset. The fitness of each kernel is then computed according to the BIC, which is later used to select the best kernels for the next generation and to guide the LLM in proposing new ones.\n\n---\n#  Adding domain knowledge\n\n> (Q3) Have you tried ... and fitness metrics?\n\nWe designed our prompts to be generic to avoid manual prompt engineering and ensure applicability across diverse tasks without requiring domain expertise. However, we agree that incorporating domain knowledge, when available, has strong potential to improve performance. For example, we see this potential in our photonic chip design experiment (see Section 5.3), where physical meaning of the chip's parameters and objectives are well-understood. However,  this involves careful prompt engineering and iterative refinement to determine the optimal level/type of domain information to include. Since the photonic chip design experiments rely on computationally intensive simulations (i.e., each evaluation taking hours to complete), we were unable to finish the experiments within the rebuttal timeline. Nevertheless, we appreciate the suggestion and plan to include this in an extended version of the paper.\n \n---\n# Shaded regions in figures\n\n> (Q4) What is the shaded ... standard error of the mean? \n\n The shaded regions in Figures 4 to 6 represent the standard error of the mean computed over independent runs/trials. We will add a remark in the figure captions to clarify this.\n\n---\n# Significance tests\n\n> (Q4) ... It would be best ... methods significantly different.\n\nWe have performed a two-sided Wilcoxon signed-rank test to evaluate the statistical significance of performance differences between CAKE+BAKER and the top baselines. The test assesses whether the median performance difference across tasks is non-zero, with the null hypothesis being no difference between methods. We focus on ABO as the primary comparison because it is the second-best performing method overall on HPOBench. We test significance at key budget levels: 10\\%, 25\\%, 50\\%, 75\\%, and 100\\% of the total evaluation budget, to identify when CAKE+BAKER begins to significantly outperform ABO. As shown in the table below, CAKE+BAKER significantly outperforms ABO as early as 10\\% of the budget across all models, with $p < 0.05$ or better. Notably, for LR and XGB, the improvement is highly significant ($p < 0.001$) even at this early stage. This early and consistent separation demonstrates that CAKE+BAKER identifies high-quality configurations more efficiently and maintains a statistically significant advantage throughout the optimization process.\n\n| Budget | 10% | 25% | 50% | 75% | 100% |\n| ------ | --- | --- | --- | --- | ---- |\n| LR     | *** | *** | *** | *** | ***  |\n| SVM    | **  | **  | **  | *** | ***  |\n| RF     | *   | *   | *   | **  | **   |\n| XGB    | *** | *** | **  | **  | **   |\n| MLP    | *   | **  | *   | *   | **   |\n\n`*` denotes significant difference ($p < 0.05$),  \n`**` denotes very significant difference ($p < 0.01$),  \n`***` denotes highly significant difference ($p < 0.001$)."}, {"review_id": "Sz1qQkaEyM", "reviewer": "Reviewer_fLo9", "summary": "The authors propose Context-Aware Kernel Evolution (CAKE), a method for designing kernels for Gaussian Process-based Bayesian Optimization using large language models (LLMs). Additionally, they introduce BIC-Acquisition Kernel Ranking (BAKER), a strategy for selecting the most effective kernel. The proposed approach achieves state-of-the-art performance across various real-world tasks.", "questions": "- Could the authors provide a hyperparameter sensitivity analysis ($\\textit{i.e.,}$ number of crossovers, mutation probability, and population size)?\n\n- Is the proposed method robust to the acquisition function? ($\\textit{i.e.,}$ UCB, Thompson sampling, LogEI, etc)\n\n- How does the size and composition of the kernel candidate pool affect the optimization performance? \n\n- Regarding BAKER (Equation 1), the role of the weight parameter $w$ is relatively intuitive, but the influence of the $\\alpha$ is less clear. The EI values computed from GP models fitted with different kernels are not directly comparable, as each EI reflects the model’s own uncertainty and predictions; thus, using their magnitudes to judge kernel quality can be misleading. I would appreciate a more detailed explanation or justification for this design choice.\n\nIf the above concerns are adequately addressed, I would be inclined to raise my score.", "limitations": "yes", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "**Strengths**\n\n- Consistently strong performance across diverse tasks and multiple random seeds.\n\n- While most of the focus in Bayesian Optimization has been on acquisition function design, the authors propose an approach for automatic kernel design.\n\n**Weaknesses**\n\n- Lack of thorough analysis (see Questions for suggested experiments).\n\n- The intuition behind BAKER is insufficient (see Questions for clarification requests).", "was_revised": true, "final_justification": "The authors provided a satisfactory rebuttal to my concerns during the discussion period. In particular, my main concern regarding the BAKER formulation—specifically, the issue that EI values could not be directly compared across different kernel functions—has been resolved. Accordingly, I have increased my rating to a 4.", "rebuttal": "*Remark:* \n- *(W1) means the 1st weakness, (Q1) means the 1st question.*\n- *Due to character limits, we kindly invite you to refer to our responses for other reviewers' similar questions. We will use* **ref. \"Response Title\"@Reviewer**. *Thank you for your understanding!*\n---\n# Hyperparameter sensitivity analysis\n\n> (Q1) Could the authors provide a ... and population size)?\n\nWe have added a sensitivity analysis that shows the average fitness of the population for different settings. We found that:\n\n- Increasing the number of crossovers ($n_c$) enhances exploration initially, but beyond $n_c = 5$ , the gains in average fitness diminish while the computational cost (in terms of LLM API calls) increases significantly. Thus, $n_c = 5$ offers an optimal trade-off between effective exploration and efficiency.\n\n| Generation | 1                   | 5                   | 10                  | 15                  | 20                  | 25                  | 30                  |\n| ---------- | ------------------- | ------------------- | ------------------- | ------------------- | ------------------- | ------------------- | ------------------- |\n| $n_c = 1$  | $0.2163 \\pm 0.0095$ | $0.3006 \\pm 0.0087$ | $0.3779 \\pm 0.0107$ | $0.4379 \\pm 0.0113$ | $0.4918 \\pm 0.0110$ | $0.5261 \\pm 0.0106$ | $0.5465 \\pm 0.0106$ |\n| $n_c = 3$  | $0.2235 \\pm 0.0093$ | $0.3407 \\pm 0.0097$ | $0.4479 \\pm 0.0136$ | $0.5308 \\pm 0.0141$ | $0.6070 \\pm 0.0138$ | $0.6558 \\pm 0.0136$ | $0.6858 \\pm 0.0142$ |\n| $n_c = 5$  | $0.2405 \\pm 0.0093$ | $0.4252 \\pm 0.0110$ | $0.5962 \\pm 0.0168$ | $0.7292 \\pm 0.0164$ | $0.8203 \\pm 0.0166$ | $0.8778 \\pm 0.0161$ | $0.9054 \\pm 0.0150$ |\n| $n_c = 7$  | $0.2487 \\pm 0.0093$ | $0.4686 \\pm 0.0127$ | $0.6720 \\pm 0.0200$ | $0.7998 \\pm 0.0196$ | $0.8925 \\pm 0.0158$ | $0.9313 \\pm 0.0079$ | $0.9419 \\pm 0.0052$ |\n| $n_c = 10$ | $0.2728 \\pm 0.0094$ | $0.5787 \\pm 0.0140$ | $0.7902 \\pm 0.0212$ | $0.8545 \\pm 0.0169$ | $0.9266 \\pm 0.0089$ | $0.9473 \\pm 0.0014$ | $0.9500 \\pm 0.0008$ |\n\n- Increasing the mutation probability ($p_m$) enhances population diversity and helps avoid premature convergence, but when $p_m > 0.7$ , excessive mutations disrupt promising kernel structures and destabilize the evolutionary process. Thus, $p_m = 0.7$ provides a balanced level of mutation that supports refinement without sacrificing stability.\n\n| Generation  | 1                   | 5                   | 10                  | 15                  | 20                  | 25                  | 30                  |\n| ----------- | ------------------- | ------------------- | ------------------- | ------------------- | ------------------- | ------------------- | ------------------- |\n| $p_m = 0.1$ | $0.2315 \\pm 0.0093$ | $0.3714 \\pm 0.0097$ | $0.4855 \\pm 0.0136$ | $0.5539 \\pm 0.0142$ | $0.6000 \\pm 0.0132$ | $0.6333 \\pm 0.0137$ | $0.6590 \\pm 0.0137$ |\n| $p_m = 0.3$ | $0.2311 \\pm 0.0093$ | $0.3813 \\pm 0.0110$ | $0.5188 \\pm 0.0163$ | $0.6252 \\pm 0.0166$ | $0.7163 \\pm 0.0164$ | $0.7703 \\pm 0.0164$ | $0.8056 \\pm 0.0175$ |\n| $p_m = 0.5$ | $0.2414 \\pm 0.0093$ | $0.4310 \\pm 0.0116$ | $0.6061 \\pm 0.0178$ | $0.7422 \\pm 0.0173$ | $0.8397 \\pm 0.0176$ | $0.8952 \\pm 0.0145$ | $0.9169 \\pm 0.0119$ |\n| $p_m = 0.7$ | $0.2475 \\pm 0.0093$ | $0.4630 \\pm 0.0127$ | $0.6621 \\pm 0.0199$ | $0.8169 \\pm 0.0192$ | $0.9184 \\pm 0.0110$ | $0.9472 \\pm 0.0026$ | $0.9500 \\pm 0.0014$ |\n| $p_m = 0.9$ | $0.2409 \\pm 0.0094$ | $0.4387 \\pm 0.0153$ | $0.6155 \\pm 0.0231$ | $0.7254 \\pm 0.0233$ | $0.8675 \\pm 0.0198$ | $0.9305 \\pm 0.0083$ | $0.9392 \\pm 0.0060$ |\n\n- Increasing the population size ($n_p$) improves diversity and allows more exploration, but for $n_p > 10$, the marginal improvement in average fitness diminishes relative to the increased computational burden from repeated GP model fitting. Thus, $n_p = 10$ ensures sufficient diversity while maintaining computational tractability.\n\n| Generation | 1                   | 5                   | 10                  | 15                  | 20                  | 25                  | 30                  |\n| ---------- | ------------------- | ------------------- | ------------------- | ------------------- | ------------------- | ------------------- | ------------------- |\n| $n_p = 5$  | $0.2251 \\pm 0.0093$ | $0.3417 \\pm 0.0110$ | $0.4239 \\pm 0.0152$ | $0.4546 \\pm 0.0155$ | $0.4984 \\pm 0.0141$ | $0.5331 \\pm 0.0151$ | $0.5634 \\pm 0.0146$ |\n| $n_p = 8$  | $0.2310 \\pm 0.0093$ | $0.3709 \\pm 0.0101$ | $0.4846 \\pm 0.0144$ | $0.5530 \\pm 0.0150$ | $0.6009 \\pm 0.0138$ | $0.6360 \\pm 0.0143$ | $0.6634 \\pm 0.0143$ |\n| $n_p = 10$ | $0.2432 \\pm 0.0093$ | $0.4329 \\pm 0.0097$ | $0.6127 \\pm 0.0138$ | $0.7527 \\pm 0.0134$ | $0.8600 \\pm 0.0137$ | $0.9101 \\pm 0.0103$ | $0.9278 \\pm 0.0082$ |\n| $n_p = 15$ | $0.2474 \\pm 0.0094$ | $0.4507 \\pm 0.0093$ | $0.6463 \\pm 0.0130$ | $0.7986 \\pm 0.0127$ | $0.9152 \\pm 0.0102$ | $0.9470 \\pm 0.0013$ | $0.9499 \\pm 0.0004$ |\n| $n_p = 20$ | $0.2504 \\pm 0.0094$ | $0.4611 \\pm 0.0090$ | $0.6665 \\pm 0.0121$ | $0.8265 \\pm 0.0120$ | $0.9342 \\pm 0.0054$ | $0.9500 \\pm 0.0011$ | $0.9500 \\pm 0.0007$ |\n\n\n---\n# Try other acquisition functions\n\n> (Q2) Is the proposed method ... Thompson sampling, LogEI, etc)\n\nWe found that CAKE is robust across different acquisition functions (**ref. \"Try other acquisition functions\"@8KrH**), including EI, UCB, and Thompson sampling (TS). While each acquisition function embodies a different exploration-exploitation trade-off (i.e., TS exhibits slightly higher standard errors due to its stochastic nature while CAKE-UCB occasionally underperforms or outperforms EI due to its optimism bias), CAKE consistently achieves comparable performance across all variants. To rigorously assess the statistical equivalence, we conducted a one-sided paired Wilcoxon signed-rank test (at 5\\% significance level). We found in most cases, the differences are not statistically significant.\n\n---\n# Kernel pool\n\n> (Q3) How does the size and ... optimization performance?\n\nWe initialize the kernel pool with the standard set of 6 base kernels: {SE, LIN, PER, RQ, M3, M5}. These kernels were selected because they encode a diverse set of functional priors, enabling CAKE to start with diverse representation of potential function structures. It is worth noting that CAKE is designed to be flexible with respect to the initial kernel set. When domain knowledge on specific structures are available, we can customize the initial kernel pool to include relevant kernels for a more informed and efficient search. Otherwise, we use a standard diverse set to avoid biasing the search towards any specific assumptions.\n\nFor the subsequent generations, the pool size is controlled by the population size hyperparameter $n_p$. Larger $n_p$ increases diversity but imposes higher computational overhead (due to GP fitting), while smaller $n_p$ risks premature convergence. Empirically, we found that $n_p = 10$ offers a favorable trade-off: it supports sufficient diversity for effective exploration of the kernel space while keeping computational costs manageable.\n\n---\n# BAKER formulation\n\n> (Q4) Regarding BAKER ... for this design choice.\n\nOur initial motivation for incorporating $\\alpha$ into the kernel selection process was to assess the practical utility of each kernel in guiding the search toward high-value regions of the input space. While metrics like BIC measure how well a kernel fits the observed data, they do not directly reflect the kernel's ability to propose high-utility query points, which is ultimately the goal in BO. We acknowledge that raw EI values across different kernels might not be directly comparable. This concern motivated us to conduct an experiment using a normalized variant of BAKER (denoted as BAKER-N). Encouragingly, our experiment shows that BAKER-N improves the performance of our original BAKER. \n\n| Configuration  | LR                  | SVM                 | RF                  | XGB                 | MLP                 |\n| -------------- | ------------------- | ------------------- | ------------------- | ------------------- | ------------------- |\n| CAKE + BIC     | $0.7632 \\pm 0.0314$ | $0.8105 \\pm 0.0201$ | $0.8427 \\pm 0.0337$ | $0.8693 \\pm 0.0421$ | $0.8015 \\pm 0.0392$ |\n| CAKE + Utility | $0.7951 \\pm 0.0243$ | $0.8476 \\pm 0.0167$ | $0.8703 \\pm 0.0275$ | $0.9081 \\pm 0.0363$ | $0.8594 \\pm 0.0341$ |\n| CAKE + BAKER   | $0.8188 \\pm 0.0220$ | $0.8663 \\pm 0.0130$ | $0.8991 \\pm 0.0250$ | $0.9257 \\pm 0.0340$ | $0.8722 \\pm 0.0310$ |\n| CAKE + BAKER-N | $0.8364 \\pm 0.0205$ | $0.8831 \\pm 0.0118$ | $0.9178 \\pm 0.0221$ | $0.9513 \\pm 0.0302$ | $0.8917 \\pm 0.0289$ |\n\nWe believe this is because, after normalization, both $w_k$ and $\\alpha$ lie in comparable ranges, leading to a more balanced trade-off between model fit and utility. Given these results, we will update the paper to include BAKER-N in the camera-ready version to reflect this improved formulation. We are grateful for the reviewer's constructive feedback which has helped us improve the performance of our method."}, {"review_id": "rAHg8bXcvx", "reviewer": "Reviewer_8KrH", "summary": "This paper proposes CAKE (Context-Aware Kernel Evolution) to adaptively select appropriate Gaussian Process (GP) kernels during Bayesian Optimization (BO).  An appropriate GP kernel for a particular BO can improve the associated surrogate model, and hence improve overall BO performance.  Between each expensive function evaluation, CAKE is applied to determine the next kernel and associated query point that (approximately) optimizes the acquisition function for that kernel, where Expected Improvement (EI) was used for single objective tasks and Expected Hypervolume Improvement (EHVI) for multi-objective tasks.  CAKE operates by performing an evolutionary search across kernels using mutation and crossover driven by a large language model (LLM) that is queried for kernel suggestions, leveraging strong prior knowledge combined with in-context learning.  Each kernel's parameters in the population are then optimized via maximum likelihood and the acquisition function is weighted by the exponentiated Bayesian Information Criteria (BIC) to avoid overfitting.  BO using CAKE is shown to outperform many baselines across many tasks.", "questions": "1.  Did the authors consider allowing the LLM access to the data directly?  Or to the optimized kernel hyperparameters?  Both perhaps for selecting the kernel or for even suggesting the next query point to run directly.  Alternatively, could the LLM perform the population update directly (i.e. given the list of kernels and fitness, propose a new list of kernels to try?)  More insight into what alternatives the authors tried in landing upon this choice of prompts would be helpful and strengthen the paper.  \n\n2.  One concern using LLM's could be *contamination*.  For example, the LLM training data could include information about HPOBench, and certainly the synthetic functions described in Appendix C.1.  Could the authors comment on whether contamination could be an issue for their results?\n\n3.  Do the results depend on the choice of acquisition function?  Would another choice besides EI result in performance differences?\n\n4.  The BAKER BIC weighting procedure could use more theoretical justification.  Could the authors comment on why this particular combination of acquisition function with BIC?", "limitations": "yes", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 4, "significance": 3, "originality": 3, "strengths_and_weaknesses": "In my view, this paper is a strong contribution providing a practical solution to an important challenge of Bayesian Optimization with Gaussian Processes (i.e. data-adaptive kernel selection).\n\nStrengths\n- The paper is well-written and organized with clear informative figures, associated code, and appendices including substantive details.    The new method is compared against appropriate strong BO baselines across a wide range of tasks and the results indicate that the proposed LLM-driven kernel evolutionary search is successful with non-trivial performance gains.  \n- Ablations of the different components offer convincing evidence that the different choices are relevant to the method's success.\n\nWeaknesses\n- A minor weakness is that CAKE does rely upon having access to a sufficiently strong LLM (in this case gpt-4-o-mini) which has some associated costs to query, including increased surrogate optimization time in Table 5.  However, for most BO applications this surrogate optimization time is irrelevant compared to objective evaluation.", "was_revised": true, "final_justification": "The rebuttal successfully addressed my questions around contamination, prompt choice, and acquisition functions with substantive responses that should improve the paper further.  I remain convinced it should be accepted as a good contribution tackling an important problem in Bayesian Optimization.", "rebuttal": "*Remark:* \n- *(W1) means the 1st weakness, (Q1) means the 1st question.*\n- *Due to character limits, we kindly invite you to refer to our responses for other reviewers' similar questions. We will use* **ref. \"Response Title\"@Reviewer**. *Thank you for your understanding!*\n---\n#  Query and surrogate fitting cost\n\n> (W1) A minor weakness ... to objective evaluation.\n\nWe agree that CAKE relies on a sufficiently capable LLM and that each LLM query introduces additional computational cost, as reflected in Table 5. However, as the reviewer rightly pointed out, in most practical BO scenarios, the time spent on surrogate modeling (including LLM queries and GP population updates) is **negligible compared to the cost of evaluating the black-box function**. For example, training a machine learning model or running a physics-based simulation can take minutes to hours, while the average wall-clock time per CAKE iteration (including LLM inference and GP fitting) is on the order of seconds. Moreover, since the training of these GPs is independent and parallelizable, the **kernel population can be trained in parallel across multiple cores or GPUs** to significantly reduce the wall-clock time for surrogate model updates.\n\nOur experiments also showed that CAKE achieves **significantly better sample efficiency** than baseline methods (see Figures 4-6). It can find high-performing solutions in fewer iterations (**ref. \"Early stage analysis\"@2phH**), which directly translates to **reduced total optimization cost** in expensive black-box settings. In this context, the slight increase in per-iteration time is a worthwhile trade-off for drastically reducing the number of costly function evaluations. \n\nFinally, we note that LLM inference can depend on external factors such as internet connectivity and API latency, which can be mitigated through strategies like batching multiple queries or deploying smaller open-source models locally (**ref. \"Try other LLMs\"@esv8**). This makes the overhead of using an LLM acceptable, especially when weighed against the benefits.\n\n---\n# Prompt alternatives\n\n> (Q1) Did the authors consider...helpful and strengthen the paper. \n\nWe did not give the LLM direct numerical access to the raw data or kernel hyperparameters because we did not equip the LLM with any external tools or code interpreters (e.g., through MCP). Instead, we provide structured summaries of the optimization state through the designed prompts, where we update the prompt with the collected observations as well as the fitness scores from the previous trials.\n\nWe actually experimented with having the LLM suggest the next query point directly, but we observed the LLM struggled with precise numerical sampling in continuous, high-dimensional spaces, especially without external tools. Similarly, we tested a \"population update\" approach where the LLM proposes a new list of kernels at once. However, this led to unstable or overly aggressive changes. In contrast, we found the evolutionary approach, using LLM-guided crossover and mutation, provides a more controlled and stable kernel evolution. \n\nThese experiments led to the current prompt design where the LLM focuses on symbolic reasoning and hypothesis generation, while the conventional BO components handle the numerical optimization parts. We will include a discussion of these alternatives in the camera-ready version to clarify the design rationale.\n\n---\n# Data contamination\n\n> (Q2) One concern using LLM's...issue for their results?\n\nWhile we acknowledge that LLMs are trained on vast corpora that may include scientific literature or code related to common optimization benchmarks, we believe that data contamination is unlikely to meaningfully influence our results, for several key reasons.\n\nFirst, while the LLM may have encountered general descriptions of synthetic functions or high-level details about HPOBench during pre-training, it is highly improbable that the **specific kernel expressions** or **optimization trajectories** used in our experiments exist in any public dataset or text. These are dynamically generated based on noisy observations and fitness feedback unique to each BO run. Thus, even if the LLM has prior knowledge of benchmark functions, it cannot memorize the exact solutions or search paths we evaluate.\n\nSecond, a supporting evidence is our photonic chip design experiment (see Section 6.3), which is based on proprietary simulations developed in our own work. The data were generated using our own simulator and have never been published or released. This makes it **effectively impossible** for this task to be present in the LLM’s training data. Yet, CAKE performs strongly on this task, further confirming that its success relies on reasoning over real-time feedback, not data leakage.\n\n**Finally**, our prompt ablation study (**ref. \"Does LLM reasoning matter?\"@esv8**) directly demonstrates that removing the observed BO history (i.e., observations, scores, and past kernels) from the prompt leads to a significant drop in performance. This shows that the LLM critically depends on in-context learning—it cannot achieve strong results using prior knowledge alone.\n\nIn summary, while pre-training may provide foundational knowledge, the specific inputs, outputs, and evolutionary paths in our experiments are unique. We will include a brief discussion of these points in the camera-ready version.\n\n---\n# Try other acquisition functions\n\n> (Q3) Do the results depend ... in performance differences?\n\nWe thank the reviewer for raising this question. Our results demonstrate that CAKE is robust across different acquisition functions, including EI, UCB, and Thompson sampling (TS). While each acquisition function embodies a different exploration-exploitation trade-off (i.e., TS exhibits slightly higher standard errors due to its stochastic nature while CAKE-UCB occasionally underperforms or outperforms EI due to its optimism bias), CAKE consistently achieves comparable performance across all variants. As shown in table below, the performance of CAKE-UCB and CAKE-TS remains close to that of CAKE-EI across a diverse set of benchmark functions. To rigorously assess statistical equivalence, we conducted a one-sided paired Wilcoxon signed-rank test (at the 5\\% significance level) comparing CAKE-EI against CAKE-UCB and CAKE-TS across multiple random seeds. The results indicate that, on the majority of functions, differences are not statistically significant. Overall, these results confirm that CAKE's effectiveness is not sensitive to the specific choice of acquisition function.\n\n| Function       | CAKE-EI                 | CAKE-UCB                | CAKE-TS                 |\n| -------------- | ----------------------- | ----------------------- | ----------------------- |\n| Ackley-2       | **0.0783 $\\pm$ 0.0203** | **0.0812 $\\pm$ 0.0215** | 0.0921 $\\pm$ 0.0287     |\n| Ackley-5       | **0.1732 $\\pm$ 0.0250** | 0.1654 $\\pm$ 0.0262     | **0.1789 $\\pm$ 0.0310** |\n| Beale          | **0.2565 $\\pm$ 0.0786** | **0.2488 $\\pm$ 0.0810** | 0.2720 $\\pm$ 0.0920     |\n| Branin         | **0.0070 $\\pm$ 0.0534** | **0.0065 $\\pm$ 0.0510** | 0.0082 $\\pm$ 0.0601     |\n| Dropwave       | 0.4690 $\\pm$ 0.0538     | 0.4820 $\\pm$ 0.0560     | 0.5010 $\\pm$ 0.0650     |\n| Eggholder      | **0.1241 $\\pm$ 0.0541** | 0.1350 $\\pm$ 0.0570     | **0.1298 $\\pm$ 0.0620** |\n| Griewank-2     | **0.0267 $\\pm$ 0.0256** | **0.0275 $\\pm$ 0.0260** | 0.0310 $\\pm$ 0.0305     |\n| Griewank-5     | **0.0185 $\\pm$ 0.0133** | **0.0190 $\\pm$ 0.0140** | **0.0195 $\\pm$ 0.0180** |\n| Hartmann       | **0.0001 $\\pm$ 0.0529** | **0.0003 $\\pm$ 0.0515** | **0.0005 $\\pm$ 0.0580** |\n| Levy-2         | **0.0353 $\\pm$ 0.0197** | **0.0360 $\\pm$ 0.0205** | 0.0402 $\\pm$ 0.0240     |\n| Levy-3         | **0.0505 $\\pm$ 0.0190** | **0.0580 $\\pm$ 0.0200** | **0.0520 $\\pm$ 0.0235** |\n| Rastringin-2   | 0.3341 $\\pm$ 0.0468     | 0.3520 $\\pm$ 0.0490     | 0.3650 $\\pm$ 0.0580     |\n| Rastringin-4   | **0.3128 $\\pm$ 0.0499** | **0.3150 $\\pm$ 0.0510** | 0.3300 $\\pm$ 0.0570     |\n| Rosenbrock     | **0.0483 $\\pm$ 0.0531** | **0.0490 $\\pm$ 0.0540** | **0.0510 $\\pm$ 0.0600** |\n| Six-Hump Camel | **0.1015 $\\pm$ 0.0669** | **0.0920 $\\pm$ 0.0680** | **0.1030 $\\pm$ 0.0750** |\n\n*Note: Values that are not significantly different from the lowest average regret for each function are **bolded**.* \n\n---\n# BAKER justification\n\n> (Q4) The BAKER BIC weighting ... function with BIC?\n\nWe designed BAKER to balance model fit (via BIC) and the potential to yield high-utility query points (via acquisition value). As shown in our ablation study (see Section 5.4), relying solely on BIC may lead to kernels that fit the data well but propose low utility query points. On the other hand, using only acquisition value can favor optimistic kernels, those that suggest high improvement but are poorly calibrated, leading to repeated sampling in suboptimal regions. By combining both, BAKER balances the trade-off: it promotes kernels that are both well-fitted to the data and has the potential to yield high-utility query points."}, {"review_id": "9celoNsXYI", "reviewer": "Reviewer_esv8", "summary": "The paper proposes and evaluates a method for iteratively designing kernel functions for Bayesian Optimization. The method is similar to a genetic algorithm, but where an LLM is queried to propose new offspring based on a fixed set of recombination operators.", "questions": "Referring to Strengths and Weaknesses, I would like to see the following experiments:\n- A standard, non-LLM genetic algorithm baseline or ablation study.\n- Results for different LLMs in CAKE, such as Claude, Llama, Qwen and Deepseek\n- What happens if the reasoning instruction is removed from the prompt? Does interpretability even matter given that the prompt responses are generic (as seen in the appendix)?\n\nGiven that the CAKE algorithm is clearly not original, the paper would need substantial quantitative and qualitative studies demonstrating why LLMs are superior to random genetic algorithms.", "limitations": "Yes", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 4, "significance": 2, "originality": 1, "strengths_and_weaknesses": "In terms of clarity, the paper is well-written, well-motivated, and easy to understand.\n\nIn terms of quality, significance and originality, the proposed method CAKE merely substitutes an LLM for the genetic algorithm recombination step. Although the paper reports competitive results over baselines, it provides little evidence to justify why CAKE is sufficiently original (over the cited LLM-genetic algorithms), or why, in the context of BO kernels, that using an LLM should result in better offspring than random recombination. In particular, the paper is missing a genetic algorithm baseline/ablation with random recombination, which should have been provided to demonstrate that the LLM-designed offspring are superior to a random genetic algorithm. Furthermore, what happens if a different LLM, such as Claude, Llama, Qwen, Deepseek, is used? Does the reasoning provided by the LLM even matter, given that the LLM responses (as seen in the appendix) are generic and not taking the actual BO experiment/trial results into context? What happens if the reasoning prompt is removed; would there even be an adverse effect on CAKE's performance? Ultimately the paper does not provide convincing evidence why CAKE should outperform a standard genetic algorithm, while at the same time CAKE is not original compared to the cited works on LLM-genetic algorithms.", "was_revised": true, "final_justification": "The authors have fully addressed my concerns regarding baselines (specifically, how CAKE improves over a baseline genetic algorithm), as well as partially addressed my concerns regarding originality (how does CAKE differ from a baseline genetic algorithm). While CAKE's originality is still a bit limited for a NeurIPS paper, I think the insights that the new experiments provide regarding BO kernel design are interesting enough to recommend acceptance.", "rebuttal": "*Remark:* \n- *(W1) means the 1st weakness, (Q1) means the 1st question.*\n- *Due to character limits, we kindly invite you to refer to our responses for other reviewers' similar questions. We will use* **ref. \"Response Title\"@Reviewer**. *Thank you for your understanding!*\n---\n# Genetic algorithm baseline \n\n> (W1, Q1) In particular, the paper is missing a genetic algorithm baseline/ablation with random recombination, which should have been provided to demonstrate that the LLM-designed offspring are superior to a random genetic algorithm.\n\nTo directly address your concerns, **we have now added a genetic algorithm (GA) baseline** that performs random crossover and mutation guided by the fitness scores, with the same population size and number of generations as CAKE. This baseline allows us to isolate the impact of replacing random recombination with LLM-guided recombination. We also would like to note that our original ablation already included a _Random Sampling_ baseline, which randomly combines kernels via addition and multiplication without any evolutionary process.\n\nAs shown in the table below, **Random Sampling** performs the worst (average rank: 6.80), confirming that blind recombination is ineffective. The **GA baseline improves over Random Sampling significantly** (average rank: 2.70), demonstrating the value of evolutionary selection. Most importantly, **CAKE + BAKER achieves the best average rank (1.04)**, substantially surpassing GA. This suggests that the LLM-enhanced recombination in CAKE goes far beyond random recombination by leveraging **semantic understanding and domain knowledge** that conventional GA operators lack. \n\n| Method            | LR            | SVM           | RF            | XGB           | MLP           | Average  |\n| ----------------- | ------------- | ------------- | ------------- | ------------- | ------------- | -------- |\n| Random Sampling   | 6.8 $\\pm$ 0.1 | 6.9 $\\pm$ 0.1 | 6.7 $\\pm$ 0.1 | 6.8 $\\pm$ 0.1 | 6.8 $\\pm$ 0.1 | 6.80     |\n| Genetic Algorithm | 2.6 $\\pm$ 0.3 | 2.5 $\\pm$ 0.3 | 2.7 $\\pm$ 0.4 | 2.8 $\\pm$ 0.3 | 2.9 $\\pm$ 0.4 | 2.70     |\n| CAKE + BIC        | 3.0 $\\pm$ 0.4 | 3.1 $\\pm$ 0.3 | 2.9 $\\pm$ 0.3 | 3.0 $\\pm$ 0.4 | 3.1 $\\pm$ 0.3 | 3.02     |\n| CAKE + Utility    | 2.3 $\\pm$ 0.3 | 2.2 $\\pm$ 0.3 | 2.4 $\\pm$ 0.2 | 2.5 $\\pm$ 0.2 | 2.6 $\\pm$ 0.3 | 2.40     |\n| Adaptive + BAKER  | 4.5 $\\pm$ 0.4 | 4.4 $\\pm$ 0.4 | 4.6 $\\pm$ 0.3 | 4.8 $\\pm$ 0.4 | 4.7 $\\pm$ 0.4 | 4.60     |\n| CKS + BAKER       | 3.1 $\\pm$ 0.3 | 3.2 $\\pm$ 0.2 | 3.0 $\\pm$ 0.2 | 3.1 $\\pm$ 0.3 | 3.2 $\\pm$ 0.2 | 3.12     |\n| CAKE + BAKER      | 1.1 $\\pm$ 0.1 | 1.0 $\\pm$ 0.1 | 1.0 $\\pm$ 0.1 | 1.1 $\\pm$ 0.1 | 1.1 $\\pm$ 0.1 | **1.04** |\n\nTo answer _why_ the LLM can generate better offsprings, we analyzed the evolution of the fitness distributions after successive edits by LLM, Random, and GA. We found that CAKE achieves faster and more consistent improvement in fitness. After just 1 edit, CAKE's distribution shifts significantly to the right, indicating rapid progress. After 10 edits, we can observe that CAKE's distribution is sharply peaked at very high fitness values, demonstrating consistent generation of high-quality offspring. In contrast, Random and GA show slower convergence and broader distributions, suggesting less effective exploration of the solution space. These results provide strong evidence that using an LLM results in better offsprings than random recombination or GA. \n\n| Method | Initial           | After 1 edit               | After 5 edits              | After 10 edits             |\n| ------ | ----------------- | -------------------------- | -------------------------- | -------------------------- |\n| LLM    | $0.179 \\pm 0.039$ | $\\mathbf{0.254 \\pm 0.023}$ | $\\mathbf{0.513 \\pm 0.026}$ | $\\mathbf{0.707 \\pm 0.030}$ |\n| Random | $0.179 \\pm 0.039$ | $0.208 \\pm 0.044$          | $0.403 \\pm 0.036$          | $0.569 \\pm 0.044$          |\n| GA     | $0.179 \\pm 0.039$ | $0.230 \\pm 0.036$          | $0.475 \\pm 0.039$          | $0.610 \\pm 0.034$          |\n\n---\n# Try other LLMs\n\n> (W2 & Q2) Furthermore, what happens if a different LLM, such as Claude, Llama, Qwen, Deepseek, is used?\n\nWe have conducted additional experiments using 3 open-source LLMs:`qwen-2.5-7b`, `deepseek-r1-distill-qwen-7b`, and `llama-3.1-8b`, and 2 commercial ones: `gemini-2.0-flash` and  `claude-3.5-haiku`. Results show that CAKE consistently works well across different LLMs, with performance generally improving with model capability. Notably, we observe a performance boost with the open-source `deepseek-r1-distill-qwen-7b`, suggesting that reasoning-based models can further enhance results.  Overall, these results indicate that as LLMs become more advanced, our method naturally benefits from their improved capabilities. We hope this comprehensive evaluation of diverse LLMs strengthens our empirical claims and provides more baselines for future research.\n\n| LLM                         | LR                  | SVM                 | RF                  | XGB                 | MLP                 |\n| --------------------------- | ------------------- | ------------------- | ------------------- | ------------------- | ------------------- |\n| qwen-2.5-7b                 | $0.7720 \\pm 0.0310$ | $0.8520 \\pm 0.0180$ | $0.8810 \\pm 0.0260$ | $0.9030 \\pm 0.0380$ | $0.8610 \\pm 0.0330$ |\n| deepseek-r1-distill-qwen-7b | $0.8122 \\pm 0.0210$ | $0.8630 \\pm 0.0141$ | $0.8964 \\pm 0.0234$ | $0.9211 \\pm 0.0320$ | $0.8692 \\pm 0.0292$ |\n| llama-3.1-8b                | $0.7815 \\pm 0.0290$ | $0.8581 \\pm 0.0177$ | $0.8873 \\pm 0.0252$ | $0.9110 \\pm 0.0355$ | $0.8652 \\pm 0.0312$ |\n| claude-3.5-haiku            | $0.8177 \\pm 0.0231$ | $0.8655 \\pm 0.0130$ | $0.8982 \\pm 0.0242$ | $0.9240 \\pm 0.0331$ | $0.8712 \\pm 0.0301$ |\n| gemini-2.0-flash            | $0.8253 \\pm 0.0204$ | $0.8720 \\pm 0.0121$ | $0.9056 \\pm 0.0223$ | $0.9310 \\pm 0.0303$ | $0.8780 \\pm 0.0281$ |\n| gpt-4o-mini                 | $0.8188 \\pm 0.0220$ | $0.8663 \\pm 0.0130$ | $0.8991 \\pm 0.0250$ | $0.9257 \\pm 0.0340$ | $0.8722 \\pm 0.0310$ |\n\n---\n# Does LLM reasoning matter? \n> (W3) Does the reasoning provided by the LLM even matter, given that the LLM responses (as seen in the appendix) are generic and not taking the actual BO experiment/trial results into context?\n\nYes, LLM reasoning _does_ matter, both for performance and interpretability, and it is meaningfully grounded in the actual BO trial results. \n\nWe would like to first clarify that **the reasoning is not generic**. Our prompt design explicitly incorporates the full history of BO trials—including observations, fitness scores, and prior kernel performance—to ensure the LLM’s reasoning is grounded in actual experimental outcomes. As shown in Appendix C.5, the LLM generates context-specific justifications, such as noting that _\"the observations can be described using a combination of three distinct patterns...\"_ or identifying increasing uncertainty in specific input regions. These are not generic statements but direct responses to the observed BO trajectory. To further clarify this point, we will include additional representative examples of the LLM’s reasoning in the camera-ready version.\n\n> (Q3) What happens if the reasoning prompt is removed; would there even be an adverse effect on CAKE's performance?\n\n**Removing the reasoning prompt _does_ harm performance**, as demonstrated in our ablation study below. The variant without reasoning, where the LLM outputs only a kernel expression, exhibits a drop in performance compared to the full prompt. This proves that the reasoning step is not merely decorative: it actively improves kernel generation. We hypothesize this is because articulating reasoning forces the LLM into a self-reflective process, enabling it to evaluate, refine, and validate its proposals against past results (i.e., functioning as an internal critique mechanism)\n\n| Configuration |      Accuracy       |\n| :------------ | :-----------------: |\n| Full          | 0.8736 $\\pm$ 0.0244 |\n| No Reasoning  | 0.8209 $\\pm$ 0.0380 |\n| No Context    | 0.7439 $\\pm$ 0.0647 |\n| No Instruct   | 0.6685 $\\pm$ 0.0704 |\n\n---\n# Originality of CAKE\n\n> (W4 & Q4) Ultimately the paper does not provide convincing evidence why CAKE should outperform a standard genetic algorithm, while at the same time CAKE is not original compared to the cited works on LLM-genetic algorithms.\n\nWhile we acknowledge that LLM-genetic algorithms has been explored in other domains, CAKE is, to our knowledge, **the first to apply LLM for automating kernel search in BO**. Unlike generic code or string-based evolution, we believe kernel search introduces unique challenges, as every kernel proposal must be (1) mathematically valid, (2) preserve compositional semantics, and (3) meaningfully explain observed data patterns. Hence, a key novelty of CAKE lies in our use of a **generalized kernel grammar** as a domain-specific language (DSL) that formally defines valid kernel expressions and operators. By embedding this grammar into the prompt, we enable the LLM to perform structured search while ensuring all outputs are semantically meaningful and mathematically valid. We believe this differentiates CAKE from both traditional GAs and existing LLM-genetic algorithms in two key ways:\n1. Unlike standard GA operators, which combine kernels through random crossover and mutation, CAKE leverages the LLM’s learned understanding of function properties and data patterns to _reason_ about plausible and effective kernel structures.\n2. By combining the LLM’s generative capabilities with the formal grammar, CAKE can not only produce mathematically valid kernel expressions but also natural language descriptions for these kernels without human expert intervention.\n\nWe believe this combination of domain-specific language, LLM-based reasoning, and interpretability represents a significant step forward beyond existing LLM-genetic frameworks, particularly in the context of kernel design for BO."}]

Author rebuttal:

Meta-review: This work proposes leveraging LLMs to perform automatic kernel design for Gaussian Process surrogate models in the context of Bayesian optimization (with a modified fitness function the basic approach applies to more general problems as well). The approach is based on the existing notion of a kernel grammar space to define a flexible set of kernels. The novelty is to leverage LLMs as generic operators that evolve kernels to higher fitness, as measured by a Bayesian Information Criterion (BIC) that ranks kernels based on surrogate model fit and acquisition function utility. Benchmarks demonstrate that the proposed approach outperforms a number of baselines. 

Strengths: 
- Well motivated
- Novel use of LLMs for kernel selection; meaningful fitness criterion
- Comprehensive evaluation against relevant baselines
- Strong performance of the proposed method
- Clarity of writing

Weaknesses:
- Technical depth of contributions is limited; no theoretical analysis
- Limited ablations in terms of the LLM and prompts used
- Cost of the method (only relevant in a high-throughput setting)

Reviewers generally agreed that the paper presents a valuable contribution to the important problem of kernel selection for GP surrogate models in Bayesian optimization. The questions and concerns raised during the reviews and discussion were relatively minor (e.g. additional baselines, ablations) and authors provided comprehensive responses (including additional experimental results) addressing these points. After the discussion reviewers had no major outstanding concerns regarding the work.

Overall, this submission presents a strong contribution addressing an important practical problem. I believe that part of the strength of the proposed approach lies in its relative simplicity while still achieving substantial performance gains compared to baselines. 

I recommend to the authors that they incorporate the many good points raised by the reviewers during the discussion in their final revision.
Decision: Accept (poster)