Improving LLM-based Global Optimization with Search Space Partitioning
Authors: Andrej Schwanke
Lyubomir Ivanov
David Salinas
Fabio Ferreira
Aaron Klein
Frank Hutter
Arber Zela
Venue: ICLR 2026 Poster (2026)
Keywords: global optimization
LLMs
blackbox optimization
multi-armed bandits
Primary area: optimization
TL;DR: HOLLM is a global optimization algorithm that adaptively partitions the search space and generates promising candidate solutions in them with LLMs.

Abstract: Large Language Models (LLMs) have recently emerged as effective surrogate models and candidate generators within global optimization frameworks for expensive blackbox functions. Despite promising results, LLM-based methods often struggle in high-dimensional search spaces or when lacking domain-specific priors, leading to sparse or uninformative suggestions. To overcome these limitations, we propose HOLLM, a novel global optimization algorithm that enhances LLM-driven sampling by partitioning the search space into promising subregions. Each subregion acts as a ``meta-arm'' selected via a bandit-inspired scoring mechanism that effectively balances exploration and exploitation. Within each selected subregion, an LLM then proposes high-quality candidate points, without any explicit domain knowledge. Empirical evaluation on standard optimization benchmarks shows that HOLLM consistently matches or surpasses leading global optimization methods, while substantially outperforming global LLM-based sampling strategies.

Published as a conference paper at ICLR 2026
IMPROVING LLM-BASED GLOBAL OPTIMIZATION WITH SEARCH SPACE PARTITIONING
Andrej Schwanke's
Lyubomir Ivanov': David Salinas?
Fabio Ferreira211 Aaron Klein?, Frank Hutter & Arber !.
University of Freiburg
2ELLIS Institute Tübingen Prior Labs
ABSTRACT
Large Language Models (LLMs) have recently emerged as effective surrogate models and candidate generators within global optimization frameworks for ex- pensive blackbox functions. Despite promising results, LLM-based methods often struggle in high-dimensional search spaces when lacking domain-specific priors, leading to sparse or uninformative suggestions. To overcome these limitations, we propose HOLLM,, novel global optimization algorithm that enhances L.-M-driven sampling by partitioning the search space into promising subregions. Each subre- gion acts as a "meta-arm" selected via bandit-inspired scoring mechanism that effectively balances exploration and exploitation. Within each selected subregion, an LLM then proposes high-quality candidate points, without any explicit domain knowledge. Empirical evaluation standard optimization benchmarks shows that HOLLM consistently matches or surpasses leading global optimization methods, while substantially outperforming global LLM--based sampling strategies.
INTRODUCTION AND MOTIVATION
Global optimization Jones et al. 1998 Rios & Sahinidis, 2013) (also known as gradient-free or zeroth-order optimization) of blackbox functions, where the only information provided to the optimizer is the function value, is fundamental challenge across numerous domains including hyperparameter tuning (Snoek et al. 2012 Turner et al., (021), policy search (Calandra et al. 2016), molecular design and chemical engineering (Langer & Tirrell, 2004 Hernnndez-Looo
et al.
2017). just to name a few. Methods such as Bayesian optimization (Shahriari et 2016 Garnett
2023) and evolutionary algorithms (Hansen 2016) have been standard and effective choice across various applications. However, they typically require assumptions regarding the underlying objective function's nature, which consecutively affect algorithmic design choices.
At the same time, recent advances in Large Language Models (LLMs) have demonstrated remarkable capabilities in generative modelling and reasoning (Brown al., 2020 OpenAl| 2023 Touvron
ct al., 2023), suggesting their potential usage for optimization tasks as well (Song et al. 2024). Efforts in integrating LLMs within blackbox optimization algorithms
surrogate models or as
candidate samplers have already shown encouraging results (Yang et al. 2024 Liu et al. 2024
Zhang et al., 2023 Aglietti et al., 2025) Agarwal et al.) [225 Kristiadi et al. 2024). Yet these methods typically rely on carefully engineered, domain-specitt prompts, and in higher dimensions and complex search spaces the I.L.'s suggestions tend to scatter sparsely, covering only a fraction
of the domain (Krishnamurthy al. 2024).
As a motivating example, we investigated the capabilities of LLMs to simulate uniform sampling from a unit hypercube. In Figure Ia we show 80 samples drawn from the unit square [0, 1j², comparing uniform sampling (blue) with Gemini- 1.5's (Reid et al. 2024) attempt at simulating uniform sampling using the prompt provided in Listing [ in Appendix (green points), and Gemini-1.5 performing uniform sampling with 5 samples per smaller subregion, using the same prompt (red points). We can clearly notice that even in 2D the LLM demonstrates high bias when sampling, therefore failing to appropriately fill the space as it was tasked to, whilst partitioning the space and prompting the LLM 16 times yields a more faithful simulation. Another illustrative example is shown in Figure Ib] where
"Equal contribution. Email to: andre undrejschwan jccwanke 90gma com, a arber. zel aeelaette aêtue.e" el1 s.ee

---

Published as a conference paper at ICLR 2026
mpliin
LLM Partition Sampling
Housdeff Dalence Bimeriioral Spece
(a)
(b)
(c)
Figure 1: (a) 80 samples in [0. 1]2: Gemini-1.5 simulating uniform sampling (green), and with region-wise partitioning (red) using the prompt in Listing (b) Gemini-1.5 prompted (see Listing 2 to generate 80 samples around the 2 minima (red crosses) globally (triangles) and withing the two
bounding boxes (circles). (c) Hausdorff distance du (P, [0, 11$) for uniform vs. -immlidl sampling in the 8-D hypercube.
we prompt Gemini-1.5 1.5 (using the prompt shown in to sample close to the two global minima (red stars) of quadratic function, given the input space boundaries. We can clearly notice the higher sampling bias when the input space is [o, 1]² instead of the smaller regions denoted via the dashed bounding boxes. Finally, in Figure Ic. compute the Hausdorff distance, di (P, [0, 1J²). between the set of E (10, 20, 50, 70, 100) sampled points P and the 8-dimensional unit hypercube [0, 1 The blue curve indicates the values for standard uniform sampling and the other ones performed by various non-agentic LLMs. Similarly as in the 2D case, partitioning the hypercube into 32 regions and sampling within each (Gemini 1.5 partitioning) notably improves the spatial coverage, enabling the LLM to more closely approximate uniform sampling.
In this paper, we introduce Hierarchical Optimization with Large Language Models (HOLLM), novel blackbox optimization method that leverages adaptive spatial partitioning guide LLMMbbasdd sampling. HOLLM iteratively builds a KD-tree on existing evaluation data, creating adaptive local partitions whose granularity evolves with sampling density. Each subregion is assigned a bandit- inspired utility score, balancing exploitation (regions with promising observed values) and exploration (geometrically large statistically uncertain regions). Subregions selected stochastically ac- cording to these scores, and LLMs then generate localized candidate proposals within the chosen regions. As LLMs trained optimization literature and scientific data encode valuable meta meta-prior about typical function behavior (e.g., local unimodality). we effectively harness this prior without assuming fixed parametric surrogate (e.g.. Gaussian Process). Furthermore, restricting candidate generation to smaller, power-dimension compared global high- nigh-dimensional sampling. We note that spatial partitioning heuristics have already proven effective in continuum armed bandits (Munos) 2011; Bubeck et al. [01 Valko et al. 2013] Grill et al. 2015), Trust Region Bayesian optimization (Eriksson et al. [219]] Daulton et al. 2022), and Monte Carlo Tree Search (Kim et al.) 2020; Wang et al.) 2020 Yang et 2021). The key contribution of this work the integration these partitioning ideas to substantially improve LLM-driven global optimization performance.
subregions significantly reduces LLM sampling difficulty
Empirical evaluations on continuous and discrete benchmark functions, including hyperparameter optimization and real world tasks, demonstrate that HOLLM effectively balances exploration and exploitation, matching or outperforming state-of-the-art methods, including established Bayesian optimization variants and Trust Region algorithms, particularly in scenarios requiring efficient navigation of complex landscapes. Furthermore, compared to approaches that prompt the LLM to propose candidates globally, HOLLM achieves considerable gains by focusing LLM suggestions locally. We provide the implementation of our algorithm in the following repository: https git
7/githubs
BACKGROUND AND RELATED WORK
We consider the problem of maximizing blackbox function f R where X' is compact domain. The objective is to find x" arg max,ex f(x through a sequence of function evaluations. In this blackbox setting, we do not have access gradients other properties of f, and can only
2

---

Published as a conference paper at ICLR 2026
observe function values at queried points. The performance of optimization algorithms in this context can also be measured using simple regret or cumulative regret. For a sequence of evaluated points I1 .2..... 2t the simple regret after t iterations is defined as: Te fx=) (a fzz))
while the cumulative regret is: R = Ei= (f(')( - fxx))).
Bayesian Optimization. Bayesian Optimization (BO) (Garnett 2023 Frazier 2018
Shahriari et al.,
2016) is a well-established framework for optimizing expensive black
box box functions by maintaining probabilistic surrogate (typically a Gaussian Process (Rasmussen & Williams 2006)) to guide
evaluations and optimizing an acquisition function (e.g. Expected Improvement (Zhan & Xing 2020)
in order to balance exploration and exploitation and efficiently search the space. Extensions like
TuRBO (Eriksson et al. 2019 Daulton et al. 2022) address high-dimensional settings by maintaining
multiple trust regions, which are dynamically resized based on optimization progress, enabling scalable and focused exploration around promising evaluations via local GPs.
Multi-Armed Bandits and Hierarchical Optimization Algorithms. Multi-Armed Bandits (MABs) (Slivkins, 2019) deal with the problem of sequential decision-making under the exploration- exploitation dilemma. In the basic setting, a MAB algorithm repeatedly selects among fixed (also infinite) number of arms or actions, each with an unknown reward distribution, aiming to minimize the cumulative regret. In the global optimization setting, the arms are the points that lie in the input space X' and at each iteration 1, an arm 2g E X' is pulled and the regret is computed by evaluating the function (2,) Grill et al. 2015). Several MAB algorithms leverage hierarchical space parti- tioning (Kleinberg al. 2008). Most notably, HOO (Bubeek et al. 2011 constructs hierarchical partitioning of the search space using n-ary trees and at each step, an unexplored region (tree leaves)
is selected based on upper confidence bounds (UCB) (Auer al. 2002 Kocsis & Szzpesvarr] 2006)
and f is evaluated at a point uniformly sampled inside the selected region. Building on HOO, extensions include parallel versions (Grill al. 20155) optimization without explicit smoothness
knowledge (Munos 20IT) or under noisy observations (Valko et al., (2033), and adaptive trees (Bull 2013) including Monte Carlo bandits (Wang et al., 2020 Yang et al., (202T). Most of these methods come with theoretical guarantees on regret bounds that depend on the dimensionality and smoothness properties of the objective function.
Large Language Models for Blackbox Optimization. Recent work has increasingly explored integrating LLMS into blackbox optimization workflows. Some approaches prompt L.MSs directly
to generate candidate solutions in natural language (Liu et al.) 2024; Yang et al., 2024 Agarwal ctt al., 2025; Zhang et al.) 2023). use them to estimate uncertainty (Ramos et al. 2023 extract features (Kristiadi al." 2024), even design novel acquisition functions (Aglietti
al. 2025).
Others replace traditional surrogate models with LLMs to predict function values and guide search in applications such hyperparameter tuning (Liu et al. 2024) and molecular design (Ramos et al.) 2023). However, these methods often rely on carefully engineered prompts containing domain- specific information (e.g., dataset statistics or problem descriptions), raising concerns about their robustness in domains where this information is not available. Recent work by KKishnamurthy et al. (2024) shows that, in simple MAB settings, LLMs struggle to explore effectively without significant prompt intervention, highlighting their limitations in decision-making
Our algorithm builds upon these foundations in order to improve by integrating tree- rre-based space partitioning, UCB -nspiirdd score function for balancing exploration and exploitation, and LLM--based candidate generation within locally promising regions.
blackbox optimization
3 HOLLM: HIERARCHICAL OPTIMIZATION WITH LLMS
In this section, we present the HOLLM algorithm for optimizing potentially noisy blackbox functions f X' -> R, which consists of main steps: Pa Partition, Score, Select, Sample and Evaluate. Given an initial set of no evaluations D.., {x (2:)) the algorithm iteratively calls each of these steps. It starts by adaptively partitioning the search space in data-driven way, scores each of these regions to balance exploration-exploittiii
selects the M most promising regions based on their score, leverages LLMs to sample candidates within these regions, and finally evaluates the best candidates according to their predicted function value from the LLM. We provide an illustrative depiction of these steps in Figure 2 This approach allows the LLM to focus on promising smaller regions of the space while benefiting from the global partitioning strategy. We
3

---

Published as a conference paper at ICLR 2026
provide the algorithm pseudocode in Algorithm a more detailed version in the Appendix A In the following, we explain each step in detail.
3.1 PARTITION: ADAPTIVE DISCRETIZATION
Based on the motivating examples we presented in Section I we hypothesize that firstly identifying promising smaller regions in the input space X' makes the LLM-based sampling more reliable compared to prompting the LLM to sample globally. To this end, we propose using an adaptive input space partitioning method based on the evaluated data at each iteration of the algorithm. In order to obtain disjoint space partitions that cover the entire space, we use k-dimensional trees (KD-trees), a space-pale panttppatiiioning data structure that recursively divides the space into half-spaces, so that we can efficiently compute the partitions in high dimensions (O log(()) for balanced tree where / is the number of iterations), whereas for other methods, such as a Delaunay triangulation (Gramacy) et al. 2022) and Voronoi diagram (Kim et al. 2020; WWyooff et al., 2024), this would become quickly impractical as the dimension increases. Each non-leaf node a KD-tree represents a splitting hyperplane perpendieular
one of the coordinate axes, dividing the space into two parts. Points on the "left" side of this hyperplane are represented by the left subtree, and points on the "right" side are represented by the right subtree. Starting from the root node X' X' every internal node chooses a split dimension (the one with the largest variance among points in the node) and a split value 8 (the mean across the selected dimension). This produces two child nodes
Xict = {a E X vg S 6), Xight (a E xx 6).
whose union equals their parent and whose interiors are disjoint. After inserting n sample points, the K leaves } form partition of X' into axis-aligned hyperrectangles and contain information about the points evaluated within it, including their coordinates and function values. We denote the
set of indices each leaf X holds as: I {i S 24 ),), with sample size n, |Tel S my- me is the maximum number of points a leaf' in the KD-tree can keep before splitting, parameterized by the number of iterations. At the start of round t, we optionally set m, mg + A log(1 + {) where mo ([d/2] by default) is the initial leaf size and A (0 by default) is the growth parameter. The logarithmic growth of m, ensures that the partitions do not become too fine-grained quickly.
An infinite-armed bandit view. Conceptually, the KD-tree can be interpreted as data-driven discretizer in infinite-armed bandits (Wang et al., 2008) CCrpentier & Valko 2015): its leaves form a coarse partition at the beginning of learning and refine only where information accumulates, mirroring
the "zooming" phenomenon in continuum-armen bandits (Kleinberg et al. 2008; Wang et al. 2008
Munos 2011 Bubeck et al. 2011: Slivkins, 2011; Valko et al. [013] Bull 2013] Carpentier &
Valko
2015 Grill et al., 2015). A key distinction, however, is that the entire tree is re-fitted at
every round, adapting the partitions' boundaries based on the current data to avoid potential early convergence to local minima. This strategy is similar to Wang et al. (2020); Yang et al. (2021), where they observed that recomputing the partitioning every few iterations resulted in better empirical performance. Although partition boundaries may merge or shift, one can frame the procedure as operating on a fixed, infinite KD-tree whose internal nodes are activated and deactivated on the fly, as abstracted in adaptive- treed bandits (Bull 2013) Slivkins 2011). Following this paradigm, every axis axis-aligned hyper-rectangular partition, X C. can be seen as a "meta-arm" (Slivkins) 2011). I). The reservoir of all such boxes uncountable, hence discovering arms, rather than merely pulling them, becomes part of the learning problem. In this language, our algorithm may be viewed as an infinite-armed bandit strategy that (i) repeatedly draws batch of candidate active input space partitions by re-fitting KD-tree to the ever-growing set of evaluations, and (ii) allocates pulls among those boxes according to a score function (as described below).
cccan
3.2 SCORE: SYNTHESIS OF EXPLOITATION, GEOMETRY AND UNCERTAINTY
At every iteration t the KD-tree yields a finite collection of leaves (partitions) { In order to decide where to spend limited evaluation budget, we need to rank these leaves based scoring function (also called utility or acquisition function) that balances exploitation of good leaves and exploration of large regions that may hold good points and that are under-sampp
(i) Exploitation via the empirical maximum. The exploitation term should optimistically reflect the best empirical evidence available for each region. Classical HOO algorithms use the sample mean as
4

---

Published as a conference paper at ICLR 2026
Pantition
Spate
Se leat
Sample
Fiza Tust
Figure 2: Overview of the HOLLM algorithm: starting from initial data D, it iteratively performs Partition, Score, Select, Sample (via LLM), and Evaluate steps to balance exploration and exploitation. For the partitioning here, we utilized KD-Tree where each axis is split based on the mean values. Each rectangle represents partition defined by the tree leaves. The red stars represent the new sampled points from the LLM.
low-variance proxy for the local reward (Kleinberg et al. 2008 Wang al. 2008 Bubeck et al. 2011] Valko et al. 2013). In global optimization, however, the objective may be highly heteroscedasticc, where one exceptionally good point inside an otherwise mediocre box can be more informative than the entire distribution. We therefore let our exploitation statistic be the largest improvement ever observed in a region Ne.::
Smin (t) = min ((x), Y = f(()) Smin + E, M4t RS!
max Y,- :E
We subtract the current empirical minimum fmin (f) (since we are maximizing f) so the values become strictly non-negative and comparable across rounds]] Choosing a max rather than an average emphasizes regions that contain good function value, behavior also found in acquisition functions in Bayesian optimization (Garnett 2023) and MCTS (Wang et al. 2020: Yang et al. 2021).
(ii) Geometric exploration through hypervolume. Let [ee ue,] x ... x [le uea] be the axis-aligned hyperrectangle corresponding to leaf Xe,,, where le, and ue, are the low and upper axis values across dimension d determined by the points in Xe, {xr E X' exploration score to regions in the input space that are underexplored, we use the d-th root of the ("e, - -1,)) 1/4 which is equivalent to the geometric mean of the side lengths of the hyperrectangle and is less sensitive to side lengths across single dimensions compared to the cell diameter. The d-th root scales Vol(X,,,) so it has the same units as length. Because axis- axis-aligned boxes shrink anisotropically as the KD-tree refines, the d-th root removes the strong dependence on dimension and yields comparable numbers across d.
Ie,.}. In order assign high
leaves' Euclidean volume (ol()).):: Vz,
(iii) Statistical exploration via UCB-V term. Even tiny region may deserve further sampling if contains few samples with high variance. Let of,- be the empirical unbiased variance of the observed function values {Yi within region Xe, at iteration f, and let ng, = |Teeel be the number of samples in that cell. We adopt an exploration factor inspired by UCB-V V (Upper Confidence Bound with Variance estimates) type algorithms (Audibert et al. 2009 Audibert & Bubeck 2009;; Wang ett al. 2008) Mukherjee et al. 2018), and apply it to our dynamic KD-tree partitioning, reminiscing UCB-AII for infinite-armer armed bandits where the number of arms increases at each iteration (Wang ct al. 2008). More specifically, we score the region Xe, with:
E1,t 20 max(0, ln(t/ (n(t/()))ee))) e --max (0, In(t In(t/((mme.))
(2)
nigt
Tig
Here, K; = [[ } is the current number of active leaves (partitions) in the KD-tree at iteration t, and is positive constant P The argument of the logarithm, t/(K,nu,r), compares the average number of samples per region (T/ K2) ) to the samples n4g, in the specific region Xz. This is a concentration term that focuses exploration on regions sampled less frequently than the current average. The max(O, In(-)) ensures the logarithmic term contributes non -egatiivel,,. effectively diminishing direct exploration incentive from this term for regions sampled more than average relative to K,. Since the effective noise or function variability can vary significantly across regions, we scale this concentration
'hee additive constant E prevents zero scores during the startup phase. 2c is often related to the range of function values or is tuning parameter. We set it to since in the total score we weight the total exploration factor.
5

---

Published as a conference paper at ICLR 2026
Algorithm 1: HIERARCHICAL OPTIMIZATION WITH LLMS (HOLLM)
Data: Initial data D. budget T, batch size b, regions to sample from M, proposals per region k
1. while S
2. Update temperature at (and optionally maximum leaf size ma)
3. Partition space by building KD-tree on D, obtaining K2 leaves (Ke,)e= K
11 Partition
4.
for each leaf Xe. do Compute Me, (Eq. [. Vt, Vol(Zr,,)) /4 and Ei., (Eq. 2 Normalize and compute total score B = pe,, + ae (31 Ve, + 82 Ee,,) 7. end
5.
// Score
Select M leaves by sampling with probabilities Pt, Вr,
// Select
9.
Generate candidates for each chosen leaf via LLM_LLL___BBBBBR_ GLLLM__GNRRATEDD... Ne,,,
// Sample
10.
Pick the top b proposals by their LLM predicted scores Evaluate f on them, add to D, and set t <- t +
n.
1/ Evaluate
end
13. return best (z., y) E
term inside the first summand with the empirical variance o2, of the corresponding region. The second summand is correction term characteristic of Bernstein-style concentration bounds (Audibert al. 2009 Maurer & Pontil 2009). It helps to ensure that the exploration bonus is sufficiently large, particularly when Ry,, is small or when the empirical variance o², happens to be small or zero This makes the exploration strategy more robust for leaves with limited observations.
Final composite score. All components must live on a shared numeric scale; otherwise, whichever component happens to have the largest dynamic range would dominate the others and nullify the intended trade-off. After each rebuild, we normalize the scores to [0, 1], preserving the intended relative weights even when the set of leaves changes drastically. The total score of each partition determined by the KD-tree partitioning is:
B Fe, + a (31 Ve, + B2 Ee,),
(3)
where ir,t, Ve,.& Er.t are the min-max normalized scores, and BI, 32 are hyperparametens (3i+ = 1 by default) weighting the geometric versus statistical exploration. The o, multiplier is total exploration weight following an annealing schedule (cosine in our experiments). In the early phase (a,, RC mmax) the В reduces to near- cear-uniform mixture of exploitation and the two exploratory terms. Assuming 31 32 and non-drastically changing regions, as t grows, the influence of Ve, faster faster than that of εg,7 because the latter itself shrinks with ny. Hence, geometric exploration is front-loaded, while statistical calibration persists more throughout the optimization. When t is close to T the rule essentially becomes greedy maximizer of pe,t which is optimal once e-accurate maximizer has already been isolated. Thus, this composite score represents the classical trade-off: go where I have seen something good, go where I have not looked at all, and go where my estimate is still uncertain".
3.3 SELECT: STOCHASTIC SELECTION OF PARTITIONS
Once the score Be.t equation has been computed for every leaf, the algorithm must decide where to spend the next evaluation budget of size b. The Select step stochastically selects partitions by sampling from a categorical distribution over leaves. At round f, we draw without replacement a batch of M distinct leaves, denoted as B,, from this categorical distribution where the sampling probability
is: Pr,t = Be,s/ E,=,, Br,t, where is the leaf index and 1 < S K,. Sampling stochastically instead of selecting the top-M leaves means that sub-optimal leaves sampled infinitely often (Audibert) eet al. 2009), potentially helping to mitigate premature convergence especially in highly non-convex and multimodal functions. Each leaf has always positive probability due to the small constant e <> 0 we add to the exploitation term in Equation] | and the min-max normalization in Equation 3 As grows, those exploratory components shrink and B become increasingly peaked around the empirical best leaves, pushing Pe. toward a near-greedy regime. Moreover, a smooth annealing
Whee neg < 2, the empirical variance of, is undefined or zero. To prevent a misleadingly small exploration bonus in such highly uncertain cases, of, might he initialized to small positive default value.

---

Published as a conference paper at ICLR 2026
Hutmiand
Eutmaans
Lety
Roerabrock
Ristrigh
B cralantios
23
20
20 1
3)
13
25
Nunser
Nimber eraluttoos
PORE
CgR
GP-R1
RR
Ri
BS KT-Tow
11M
HOLEM
Figure 3:
Best function value across 50 iterations on the synthetic problems. HOLLM outperforms or matches the performance of baselines.
of in Equation 3 avoids abrupt sovitch-to-greedyy policy, which may ignore late-appearing, high-value regions if they happen be discovered just after the switch. Finally, sampling M leaves without replacement diversifies evaluations by always sampling distinct regions.
3.4 SAMPLE:
CANDIDATE GENERATION
After the Select step has identified a batch of leaves B, = their corresponding hyperrectangular partition boundaries, HOLLM suggests new candidate points inside each chosen partition by prompting an LLM with the following logic: "Given the history of evaluations Do propose new points that are likely to reveal high values of f inside X.,." We construct a structured prompt (see Appendix [ containing: (i) points in DD as in-context examples, (ii) the numeric partition bounds (l,,, Us )a_ for cell Ks,,, and (iii) task instructions return new proposals and their estimated function values. Feeding this prompt to the LLM yields
J., which also contain
(se, f.) LLM_ GENERATE (Di, (le,, U4, (4,,44 ),=,
where x; = (er.1, } Ns,, and f. = (f........
E IR* are, respectively, the LLM's candidate locations and their predicted function values. We prompt the LLM to generate candidates and predict their performance with two prompts, one for generation and one for prediction. Across the M selected leaves we thus obtain .M suggestions. The parameter trades off the breadth of local exploration against prompt complexity and LLM inference cost. Finally, HOLLM keeps the globally best according to f and evaluates them true function.
4 EMPIRICAL EVALUATION
In this section, we evaluate HOLLM on a variety of search spaces and tasks. These span continuous
synthetic functions, hyperparameter optimization (Feurer & Hutter 2019: Yu & Zhu 2020 over discrete spaces, and real world tasks.
Baselines. We implement HOLLM Syne (Salinas et al., 2022) and compare against several
different state-of offie--ff-hhearrr the-art algorithms found in the same framework, such multi- culti-fidelity methods Bayesian optimization (BO) (CQR (Salinas et al., (203)). Gaussian Process BO with Expected Im-
provement (GP-EI (Snoek et al. 2012 Balandat et al. (0220)) density estimator methods (BORE Tiao
et al. 202T)), evolutionary strategies (RE RReal et 2019)) and random search (RS (Bergstra &
Bengio 2012)). We also ran space partitioning methods such as TurBO (Eriksson al. 2019) and LA-MCTS (Wang et al. 20200 Yang et al.) 1022] using their original implementations. We show baselines Appendix C.T In all benchmarks, we also compare to the global LLM- based optimizer baseline (see Algorithm 2 in the appendix) that uses the exact same prompt structure HOLLM (we provide the prompt templates in Appendix [ with the only difference being the region boundaries. To isolate the effect of M-beed sampling, we add an RS + KD-Tree baseline that draws candidates uniformly random from each subregion.
the comparison to the last
Setup. Starting from no 5 initial random evaluations, we run each method 10 times for a total of T = 50 iterations with different random seeds and report their mean and standard error. If not stated otherwise, for HOLLM we always decay the a, exploration coefficient from 1.0 to 0.01 using a cosine annealing schedule (Loshchilov & Hutter (017), a batch size b = 4, M 5 selected partitions, & = proposals per selected partition. and a fixed maximum leaf size m, ma [d/2]. In Appendix C.2 we provide robustness experiments when varying these hyperparameter choices in algorithm. Unless stated otherwise, we use Gemiini-2.0-Flash
default L.LM choice
7

---

Published as a conference paper at ICLR 2026
Ieraten
tendban17
Tendloo
CalSenationnee Probald bit
Seldttationnnn bis
Figure 4: Illustrative example of HOLLM optimizing a ID multimodal problem. The rectangles represent the space partitions (top figure) and are highlighted in orange whenever they are selected based on their respective probabilities (bottom figure). We used batch size 3. All new points (red stars) are LLM suggestions. Notice how partitions become fine-grained around the global maximum.
in Sample due its fast inference speed, low cost, and large context window. Importantly, the LLM is provided with only minimal task information: the input dimensionality, variable names whenever applicable (e-g., hyperparameter names), partition boundaries, and in-context examples. No task-specific descriptions dataset statistics are included. While prior work (Liu et al.) (024) shows that performance can improve by enriching prompts with such information, we avoid this to prevent potential contamination and reported performance overly engineered prompts. We provide the full experimental details Appendix B
4.1 SYNTHETIC FUNCTIONS
We benchmark on six synthetic functions of varying nature and dimensionality: Hartmann-3D and Hartmann-6D (smooth but sharply multimodal), Rosenbrock-8D (unimodal with narrow valley and ill-conditioning), Rastrigin-1OD (regular multimodality with 10 local minima) and Lévy-10D (plateaus, cliffs, and funnels). These functions pose challenges ranging from fine local search to broad exploration. See Table in Appendix(B.2 B.2 more details on these functions. Results presented in Figure show that HOLLM consistently outperforms the global LLM baseline, especially the multimodal functions, also exhibiting less variance between runs. It also matches or surpasses all other baselines, except Rastrigin, most likely due to the high number of local minima. In Appendix C. we provide additional results using Beminni-155-Flan as well as a comparison to TurBO and LA-MCTS, which were implemented outside SyneTTne.
Visualizing the Optimization Process. In Figure we show a visualization of HOLLM's mechanics on ID multimodal function. The rectangles represent the KD-tree space partitions and they highlighted in orange whenever they get selected. We can see that during the first iterations the partitions are larger and HOLLM is more exploratory, also confirmed by the regions' respective probabilities (bottom bar plot). Later on, the regions become smaller, high modes are identified and by the end the score probability mass concentrates more around the global maximum. We provide similar visualizations for Lévy-1D and Rosenbrock-ID in Appendix C
4.2 HYPERPARAMETEL OPTIMIZATION
We assess the effectiveness of HOLLM on hyperparameter optimization by optimizing the 9D cate- gorical space from FCNet (Klein & Hutter 2019), where the task is to minimize the validation MSE
Naval
Parkinoms
Prosein
Slice
Penction saluue
0000
-300
3084
-0000
00000
-3,301
-303
10
31
30
10
10
30
80
10
10
51
30
10
30
20
10
Numburr of eve Inations
Number of musiattion
Nambor of csnluatione
Number of coniraations
BORE
CQR
GP-EI
RS
RS KD-Tree
LLM
HOLLM
Figure 5: Hyperparameter optimization on 4 datasets from the FCNet search space. All basclines from Synetune evaluated asynchronously using 4 workers.
8

---

Published as a conference paper at ICLR 2026
Pusicilii
CarSa chupact
Veliceessetty
Piartta pellie
$1.-
Sumber rolinations
Nambee eralintionss
Number of rcuiisatoon
B2
BORE
Eggi
CIFRI
RE
RS 4 KD-Tbwe
HOLLM
Figure 6: Results on 3 real-world tasks comparing HOLLM to base- Figure 7: Critical difference lines from SyneTune. We run each method 10 times and report the diagram. Lower ranks indi- mean and standard error.
cate better performance.
of fully connected network 4 distinct datasets: PROTEIN, NAVAL, PARK PARKINSONS and SLICE. See Appendix [B.3] for more details on this search space. Results shown in Figure 5 demonstrate that both HOLLM and the global LLM optimizers outperform methods such BORE and CQR, which typically are the off-the-shelf best choices on these benchmarks. A single run with 50 trials of the HOLLM method has an averaage runtime of approximately 330 seconds, with a total cost of $0.14 for the Geminii2.0-PP 2.0-Flash API calls. Compared to the global LLM baseline, the benefit of using space partitioning in this task is less pronounced compared to the other tasks, probably due to the discrete nature of the search space. However, HOLLM shows clear benefit of LLM-based subregion sampling compared to the much lower performance of RS + KD-Tree
4.3 REAL WORLD TASKS
To demonstrate further the practical applicability of HOLLM, we evaluate it on 3 real-world tasks from Tanabe Ishibuchi (2020) with continuous input space: Penicillin Production (7D), where the aim is to maximize penicillin yield given seven continuous variables related to fermentation; Vehicle Safety (5D), where the goal is to optimize vehicle crash safety by adjusting five structural design parameters; and Car Side Impact (7D), where the task is to improve side-impact performance through seven design variables. We refer the reader to Appendix [ for a detailed description each of these tasks. Finally, in Figure / we compute the Critical Difference Diagram and show that HOLLM demonstrates the highest average rank across all tasks.
4.4 ABLATIONS AND ANALYSIS
Performance with non-proprietary LLMs. To assess the impact of the L.M model choice, we compared HOLLM against the global L.M optimizer on the Vehicle Safety task using 1-8ma-3.1-8B (Dubey et al.) 2024), Llama-3.3-70B. and Diven3-30B-A3.
(Team 2025) as the surrogate and candidate generator LLM model. Results in Figure Sa show that HOLLM consistently found better function values and demonstrated superior robustness the Vehicle Safety benchmark. While the standard optimizer failed by stalling immediately when paired with the Llama models, HOLLM successfully used these same models to achieve performance approaching that of (emini-220-FFlas In Appendix H
we present the results on the other real-world benchmarks.
Ablation Study:
Scoring, Selection, and Sampling. To justify the HOLLM scoring function, conduct an ablation study on Vehicle Safety
isolating key components: (i) HOLLM removing the exploration bonus; (iii) HOLLM (Exp EExploration ion
Only), removing the exploitation term; and (iv) HOLLM (Unife (Uniform) form). selecting regions uniformly. Results (Figure 3b highlight the importance of sariance-awar exploration: UCBI 1 over-explores stable but suboptimal regions, while UCB-V adapts to empirical variance, enabling faster identification of high-potentic subregions; replacing UCB-V with UCB1 reduces convergence speed and final performance. The full scoring function outperforms both Exploitation-on/e and Exploration-only variants, confirming the need for balanced tradcoff, while the Uniform baseline performs worst, indicating that HOLLM's 's gains stem from intelligent budget allocation via the scoring function rather than from partitioning or LLM capability alone. Additional results NAS-Bench-20 (Dong & Yang 2020 (Gemini-1 1.5-Fiah provided in Appendix
(UCB1), repeaiiceplacig replacing UCB-V with UCB1 (Auer et al. 2002); (ii) HOLLM (Exploitat
Robustness to Observation Noise. Another key advantage of integrating sariance-sensitive scoring function (UCB-V) V) with localized partitioning is the ability to handle noisy objective functions. To evaluate this robustness aspect, ran HOLLM and the baselines on the Vehicle Safety benchmark
9

---

Published as a conference paper at ICLR 2026
Vehicle Safety
Vehicle Safety
Vehicle Safety Noise
-1060
-1080
5P%
-1665
-1864
-3470
1666
1666
3675
1865
1668
Lite
1670
-167
-1405
-1674
-174
Number of ovaluations
Namber ioaluaatoons
Number eecalaation
(a) Various LLM models
(b) Scoring ablation
(c) Noisy benchmarks
Figure 8: Analysis & Ablations on Vehicle Safety. (a) Performance when using various LLM models. (b) Ablation of the scoring function: we compare the HOLLM scoring function (black) against variants replacing UCB-V with UCBI, using only exploration exploitation terms, and selecting regions uniformly at random. (c) Performance on noisy benchmarks: HOLLM demonstrates superior robustness and sample efficiency compared the global LLM baseline and traditional methods when the function is corrupted by N(0, o²) noise (a is 1% of the global function range).
under observation noise. The noise was modeled as zero-mean homoscedastic Gaussian noise (O( where the standard deviation (α) was fixed at 1% of the global objective function range. As illustrated in Figure 8c] HOLLM exhibits clear and significant performance advantage over all baselines. While the global LLM baseline and standard methods like TPE (Bergstra et al. 2011) and Random Search (Bergstra & Bengio 2012) struggle to filter the noise and plateau prematurely, HOLLM consistently reaches higher objective value. By restricting the context locally, HOLLM obtains more reliable mean estimates even under heavy observation noise, ensuring that exploitation remains focused and robust.
Runtime Complexity, Surrogate Performance & Sample Diversity. In Appendix G we discuss the runtime complexity and sample efficiency of HOLLM. Moreover, in Appendix and Appendix D we study the surrogate performance of the LLM in comparison to Gaussian Processes (Rasmussen & Williams 2006) and the recent TabPFN-v2 (Hollmann et al. 2025), as well as the sample diversity of HOLLM the global L.M.
CONCLUSION, LIMITATIONS AND SOCIETAL IMPACT
We propose HOLLM, novel L.LM-based global optimization method for expensive blackbox functions, that combines adaptive KD-tree partitioning. a bandit-inspired score function, and LLM capabilities for generating new candidate points on locally promising regions. HOLLM excels especially on multimodal functions with many local optima that pose a risk for premature convergence, hyperparameter optimization and real-world tasks, consistently outperforming LLMs with a global sampling policy and other non-LLM state-of the-art methods.
Limitations. While HOLLM combines nonparametric bandit methods with LLM-based sampling and shows strong empirical performance, it has several limitations. Firstly, its effectiveness depends heavily on the quality of I.L. generated proposals; biased or miscalibrated models can misguide the search or waste evaluations. Secondly, the inference and monetary cost of LLMs, especially proprietary ones, can limit scalability high-dimensional settings. Third, although default parameter values perform well in our experiments, real-world deployment may require tuning them to avoid premature convergence or excessive exploration. Finally, our approach currently lacks formal theoretical guarantees, especially regarding regret bounds, which we leave for the future.
Impact. The use of LLMs in global optimization has several societal implications. HOLLM has the potential to accelerate progress in areas such as drug discovery, materials design, and energy systems by reducing experimental costs and enabling personalized solutions. On the other hand, reliance on LLMs trained on biased data risks perpetuating social injustices when guiding sensitive decisions (e.g., hiring). Additionally, repeated LLM queries incur considerable energy costs, and the opacity of LLM decisions may limit transparency and reproducibility. Therefore, responsible deployment requires bias assessment, usage controls, and transparency in both computational and ethical impacts.
10

---

Published as a conference paper at ICLR 2026
ACKNOWLEDGMENTS
We acknowledge funding by the European Union (via ERC Consolidator Grant DeepLeerning 2.0, grant no. 101045765). Views and opinions expressed are however those of the author(s) only and do not necessarily reflect those of the European Union or the European Research Council. Neither the European Union nor the granting authority can be held responsible for them. Frank Hutter acknowledges the financial support of the Hector Foundation. We also thank Google Cloud for their free trial program, that enabled to use the Google Gemini models throughout this project.
Funded by the European Union
REFERENCES
Dhruv Agarwal, Manoj Ghuhan Arivazhagan, Rajarshi Das, Sandesh Swamy, Sopan Khosla, and Rashmi Gangadharaiah. Searching for optimal solutions with LLMs via bayesian optimization. In The Thirleenth International Conference Learning Representations, 2025.
Virginia Aglictti, Ira Ktena, Jessica Schrouff, Eleni Sgouritsa, Francisco J. R. Ruiz, Alan Malek, Alexis Bellot, and Silvia Chiappa. FunBO: Discovering acquisition functions forbayesian opti- mization with funsearch, 2025.
Jean- Yves Audibert and Séhastien Bubeck. Minimax policies for adversarial and stochastic bandits. In Proceedings of the 22th annual conference on learning theory, pp. 217-226, Montreal, Canada, June 2009.
Jean-Yves Audibert, Rémi Munos, and Csaba Szepesvári. Exploration-exploitttinn tradeoff using variance estimates in multi-armed bandits. Theor. Comput. Sci., (11((19)1176611199 April 2009. ISSN 0304-3975.
Peter Auer, Nicolò Cesa-Bianchi and Paul Fischer. Finite-timg analysis of the multiarmed bandit problem. Machine Learning, 47(2):235-22 256, May 2002.
Maximilian Balandat, Brian Karrer, Daniel R. Jiang, Samuel Daulton, Benjamin Letham, An- drew Gordon Wilson, and Eytan Bakshy. BoTorch: A Framework for Efficient Monte-Carlo Bayesian Optimization. In Advances in Neural Information Processing Systems 33, 2020.
James Bergstra and Yoshua Bengio. Random search for hyper-parameter optimization. Journal of Machine Learning Research, 13(10):281-300 2012.
James Bergstra, Rémi Bardenet, Yoshua Bengio, and Balázs Kégl. Algorithms for parrmer optimization. In Advances Neural Information Processing Systems, volume 24, 2011.
Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, et Language models are few-shot learners. In Advances in Neural Information Processing Systems, 34, Red Hook, NY, USA, 2020.
Sébastien Bubeck, Rémi Munos, Gilles Stoltz, and Csaba Szepesvári. X-armed bandits. J. Mach. Learn. Res., 12(mul)::6555-1695 July 2011. ISSN 1532-4435.
Adam Bull. Adaptive-treed bandits. Bernoulli, 21, 02 2013. doi: 10.3150/14-BEJ644
Roberto Calandra, André Seyfarth, Jan Peters, and Mare Peter Deisenroth. Bayesian optimization for learning gaits under uncertainty. Annals of Mathematics and Artificial Intelligence, 76(1-2):5-22 February 2016. ISSN 1012-2443.
Alexandra Carpentier and Michal Valko. Simple regret for infinitely many armed bandits. In Proceedings of the 32nd International Conference International Conference on Machine Learning, 37, pp. 1133-1141, 2015.
Patryk Chrabaszez, Ilya Loshchilov, and Frank Hutter. A downsampled variant of imagenet as an alternative to the cifar datasets. CoRR, abs/1 707. 707.088199 2017.

---

Published as a conference paper at ICLR 2026
Alexander
Wenlong Lyu, Rasul Tutunov, Zhi Wang, Antoine Grosnit, Ryan- Griffiths, Alexandre Maravel, Jianye Hao, Jun Wang. Jan Peters, and Haitham Bou Ammar. Hebo: Pushing the limits of sample-efficient hyperparameter optimisation. Journal of Artificial Intelligence Research, 74, 07 2022.
Samuel Daulton, David Eriksson, Maximilian Balandat, and Eytan Bakshy. Multi-objective bayesian optimization over high- -isheninnnsoo search spaces. In Proceedings the Thirty-Eightn Conference on Uncertainty in Artificial Intelligence, volume 180, pp. 507-517. PMLR, 2022.
Xuanyi Dong and Yi Yang. Nas-bench-200 Extending the scope of reproducible neural architecture search. In International Conference Learning Representations, 2020.
Abhimanyu Dubey et al. The llama 3 herd of models, 2024.
David Eriksson, Michael Pearce, Jacob Gardner, Ryan D Turner, and Matthias Poloczek. Scalable global optimization via local bayesian optimization. In Advances in Neural Information Processing Systems, volume 32. Curran Associates, Inc., 2019.
Matthias Feurer and Frank Hutter. Hyperparameter Optimization, pp. 3-33. Springer International Publishing, Cham, 2019. ISBN 978-3-030-05318-5 doi: 10. 0000://9788-00000511-------
Frazier. tutorial on bayesian optimization. ArXiv,
2018.
Roman Garnett. Bayesian Optimization. Cambridge University Press, 2023.
Robert Gramacy, Annie Sauer, and Nathan Wycoff. Triangulation candidates for bayesian opti- mization. In Advances in Neural Information Processing Systems, volume 35, pp. 35933-35945. Curran Associates, Inc., 2022.
Jean-Bastien Grill, Michal Valko, Remi Munos, and Remi Munos. Black-box optimization noisy functions with unknown smoothness. In Advances in Neural Information Processing Systems, volume 28. Curran Associates, Inc., 2015.
Nikolaus Hansen. The ema evolution strategy: tutorial, 2016.
Nikolaus Hansen and Anne Auger. Cma-es: evolution strategies and covariance matrix adaptation. In Proceedings of the 13th Annual Conference Companion on Genetic and Evolutionary Computation, GECCO 11, pp. 991-1010, New York, NY, USA, 2011. Association for Computing Machinery.
José Miguel Hernández- Lonnadd,z--Lobto,, James Requeima, Edward O. Pyzer-Knaph and Alán Aspuru- Guzik. Parallel and distributed thompson sampling for large-scale accelerated exploration of chemical space. Proceedings of the 34th International Conference on Machine Learning Volume 70, ICML'17, pp. 1470-1479. JMLR.org, 2017.
Noah Hollmann, Samuel Müller, Lennart Purucker. Arjun Krishnakumar, Max Körfer, Shi Bin Hoo, Robin Tibor Schirmeister, and Frank Hutter. Accurate predictions on small data with a tabular foundation model. Nature, 01 2025. doi: 10. 10388644158--0244882222
Donald R. Jones, Matthias Schonlau, and William J. Welch. Efficient global optimization of expensive black-box functions. J. Global Optimization, 13(4):455-44 492, 1998.
Beomjoon Kim, Kyungjae Lee, Sungbin Lim, Leslie Kaelbling, and Tomas Lozano-Perez, Monte carlo tree search in continuous spaces using voronoi optimistic optimization with regret bounds. Proceedings of the AAAI Conference on Artificial Intelligence, 34(06))9916-992) Apr. 2020. doi:
10.1609/aaii 1..1609/aai.v344406.66 v34i06.65444 6656.
Aaron Klein and Frank Hutter. Tabular benchmarks for joint architecture and hyperparameter optimization, 2019. URL https:// httttpp://axxiy.cco///////..... wttpooggg/aaa..ooo/////b/ 00tp./////1990090909009
Robert Kleinberg, Aleksandrs Slivkins, and Eli Upfal. Multi-armed bandits in metric spaces. In Pro- ceedings of the Fortieth Annual ACM Symposium Theory of Computing, STOC '08, pp. 681-690, New York, NY, USA, 2008. Association for Computing Machinery. ISBN 9781605580470.
Levente Koesis and Csaba Szepesvari. Bandit based Monte-Carlo planning. In European Conference Machine Learning, pp. 282-203 203. Springer, 2006.
12

---

Published as a conference paper at ICLR 2026
Akshay Krishnamurthy, Keegan Harris, Dylan J. Foster, Cyril Zhang, and Aleksandrs Slivkins. Can large language models explore in-context? In Advances in Neural Information Processing Systems, volume 37. pp. 120124- 1201188- Curran Associates, Inc., 2024.
pp.
Agustinus Kristiadi, Felix Strieth-Kalthoff Marta Skreta, Pascal Poupart, Alán Aspuru-Guzik, and Geoff Pleiss. sober look at Ilms for material discovery: are they actually good for bayesian optimization over molecules? In Proceedings of the 41 International Conference Machine Learning, ICML'24, 24. JMLR, 2024.
Robert Langer and David A. Tirrell. Designing materials for biology and medicine. Nature, 428 (6982))4877-) 492, April 2004.
Marius Lindauer, Katharina Eggensperger, Matthias Feurer, André Biedenkapp, Difan Deng, Carolin Benjamins, Tim Ruhkopf, René Sass, and Frank Hutter. SMAC3: versatile bayesian optimization package for hyperparameter optimization. Journal of Machine Learning Research, 23(54):1-9, 2022.
Tennison Liu, Nicolás Astorga, Nabeel Seedat, and Mihaela van der Schaar. Large language models enhance bayesian optimization. In The Twelfth International Conference on Learning Representations, 2024.
Ilya Loshchilov and Frank Hutter. SGDR: Stochastic gradient descent with warm restarts. In International Conference Learning Representations, 2017.
Andreas Maurer and Massimiliano Pontil. Empirical bernstein bounds and sample-variance penaliza- tion., 2009.
Subhojyoti Mukherjee, K. P. Naveen, Nandan Sudarsanam, and Balaraman Ravindran. Efficient-ucbv: An almost optimal algorithm using variance estimates. Proceedings of the AAAI Conference on Artificial Intelligence, 32(1), Apr. 2018. doi: 10.1609/aaai. v32ia..i.32i..... 332i1.12110.
Rémi Munos. Optimistic optimization of a deterministic function without the knowledge of its smoothness. In Advances Neural Information Processing Systems, volume 24, 2011.
OpenAl. GPT-4 technical report, 2023.
Mayk Ramos, Shane Michtavy, Marc Porosoff, and Andrew White. Bayesian optimization of catalysts with in-context learning, 04 2023.
Carl Edward Rasmussen and Christopher K. Williams. Gaussian Processes for Machine Learning. The MIT Press, 2006.
Esteban Real, Alok Aggarwal, Yanping Huang, and Quoe V. Le. Regularized evolution for image classifier architecture search. In Proceedings of the Thirty-Thibe AAAI Conference on Artificial Intelligence. AAAI Press, 2019.
Machel Reid al. Gemini 1.5: Unlocking multimodal understanding across millions of tokens of context. CoRR, abs/2403.05530, 2024.
Luis Miguel Rios and Nikolaos V. Sahinidis. Derivative-free optimization: a review of algorithms
and comparison of software implementations. J. Glob. Optim., 1247-11277-299 2013.
David Salinas, Matthias Seeger, Aaron Klein, Valerio Perrone, Martin Wistuba, and Cedric Archam- beau. Syne tune: A library for large scale hyperparameter tuning and reproducible research. In Proceedings of the First International Conference on Aulomated Machine Learning, 2022.
David Salinas, Jacek Golebiowski, Aaron Klein, Matthias Seeger, and Cedric Archambeau. Optimiz- ing hyperparameters with conformal quantile regression. In Proceedings of the 40th International Conference on Machine Learning, volume 202, 2023.
Andrej Schwanke, Lyubomir Ivanov, David Salinas, Frank Hutter, and Arber Zela. Multi-objective hierarchical optimization with large language models, 2026.
Bobak Shahriari, Kevin Swersky, Ziyun Wang, Ryan P. Adams, and Nando de Freitas. Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE, 104, 2016.
13

---

Published as a conference paper at ICLR 2026
Aleksandrs Slivkins. Multi-armed bandits on implicit metric spaces. In Advances in Neural Informa- tion Processing Systems, volume 24. Curran Associates, Inc., 2011.
Aleksandrs Slivkins. Introduction multi-armed bandits. Found. Trends Mach. Learn., 12(1-2): 1-286, November 2019. ISSN 1935-8237. doi: 10.1561/2000000068.
Jasper Snoek, Hugo Larochelle, and Ryan P. Adams. Practical bayesian optimization of machine learning algorithms. In Proceedings of the 26th International Conference on Neural Information Processing Systems, pp. 2951-2959, Red Hook, NY. USA, 2012. Curran Associates Inc.
Xingyou Song, Yingtao Tian, Robert Tjarko Lange, Chansoo Lee, Yujin Tang, and Yutian Chen. Position: leverage foundational models for black-box optimization. In Proceedings of the 41st International Conference on Machine Learning, ICML'22. JMLR.org. 2024.
Ryoji Tanabe and Hisao Ishibuchi. An easy-to-use real-world multi-objectivv optimization problem suite. Appl. Soft Comput., 89(C), April 2020. ISSN 1568-4946. doi: 10.1016fj.sscc22000.00000
Qwen Team. Qwen3 technical report, 2025. URL httpt:ttt/t//ttttte org/ abs 2505.
Louis Tiao, Aaron Klein, Cédric Archambeau, Edwin V Bonilla, Matthias Seeger, and Fabio Ramos. Bayesian Optimization by Density- Estimation. In Proceedings of the 38th International Conference on Machine Learning (ICML2021), Virtual (Online), July 2021.
Hugo Touvron et al. Llama 2: Open foundation and fine-tuned chat models, 7 2023.
Ryan Turner, David Eriksson, Michael McCourt, Juha Killi, Eero Laaksonen, Zhen Xu, and Isabelle Guyon. Bayesian optimization is superior to random search for machine learning hyperparameter tuning: Analysis of the black-box optimization challenge 2020. In Proceedings of the NeurIPS 2020 Competition and Demonstration Truck, pp. 3-26, 2021.
Michal Valko, Alexandra Carpentier, and Rémi Munos. Stochastic simultaneous optimistic opti- mization. In Sanjoy Dasgupta and David McAllester (eds.). Proceedings of the 30th International Conference on Machine Learning. Proceedings of Machine Learning Research, pp. 19-27, Atlanta, Georgia, USA, 17-19 Jun 2013. PMLR.
Linnan Wang, Rodrigo Fonseca, and Yuandong Tian. Learning search space partition for black- box optimization using monte carlo tree search. In Advances in Neural Information Processing Systems, volume 33. Curran Associates Inc., 2020.
Yizao Wang, Jean-Yves Audibert, and Rémi Munos. Algorithms for infinitely many-armed bandits. In Advances in Neural Information Processing Systems, 22, 2008
Colin White, Mahmoud Safari, Rhea Sanjay Sukthanker, Binxin Ru, Thomas Elsken, Arber Zela, Debadeepta Dey, and Frank Hutter. Neural architecture search: Insights from 1000 papers. ArXiv, 2023.
Nathan Wycoff, John W. Smith, Annie S. Booth, and Robert B. Gramacy. Voronoi candidates for bayesian optimization. ArXiv, abs/2402.04922, 2024.
Chengrun Yang, Xuezhi Wang. Yifeng Lu, Hanxiao Liu, Quoc V Le, Denny Zhou, and Xinyun Chen. Large language models as optimizers. In The Twelfth International Conference on Learning Representations, 2024.
Kevin Yang, Tianjun Zhang, Chris Cummins, Brandon Cui, Benoit Steiner, Linnan Wang, Joseph E Gonzalez, Dan Klein, and Yuandong Tian. Learning space partitions for path planning. In Advances in Neural Information Processing Systems, volume 34, pp. 378-3911 Curran Associates, Inc., 2021.
Tong Yu and Hong Zhu. Hyper-parameter optimization: review of algorithms and applications.
arXiv preprint arXiv:2003.05689, 2020.
Dawei Zhan and Huanlai Xing. Expected improvement for expensive optimization: a review. of Global Optimization, 78(3)5007-54,, November 2020. ISSN 0925-5001. doi: 10.1007/
-0898-020-009223-X
Michael Zhang, Nishkrit Desai, Juhan Bae, Jonathan Lorraine, and Jimmy Ba. Using large language models for hyperparameter optimization. In NeurIPS 2023 Foundation Models for Decision Making Workshop, 2023.
14

---

Published as a conference paper at ICLR 2026
A ALGORITHM PSEUDOCODES
In this section, we present detailed pseudocode for the HOLLM algorithm in Algorithm 3 which complements Algorithm[] [ In the algorithm, we omit the subscript é for easier readability. Addition- ally. Algorithm _ describes the LLM- basdddaaddd--- global optimization baseline method used throughout our experiments. To ensure fair comparison, we configure the baseline to propose k b points per iteration (line 2 of Algorithm 2) matching the total number of proposals generated by HOLLM across all subregions, i.e., proposals each of the M subregions.
Algorithm 2: G1.0BAL-LLMM baseline
Data: Initialize D with ni points, budget T, batch size b, proposals k M do
for /. "o......
2. Propose k MM points with L.LM_ LLLM__EEEEE TOMM_GGEERA((X, D. M)
3. Evaluate the top b and add D
return best (z, y) D
B DETAILS ON TASKS, BASELINES AND EXPERIMENTAL SETUP
B. BASELINES
We compare HOLLM to the following baselines:
Random Search (RS) Bergstra & Bengio (2012) serves as a simple baseline that uniformly samples configurations from the search space without any learning or adaptation.
Regularized Evolution (RE) Real et al. 2019) is evolutionary algorithm that maintains population of candidate solutions and evolves them through mutation operations. The method reg- ularizes the population by removing the oldest individuals, preventing stagnation and maintaining diversity.
Conformalized Quantile Regression (CQR) Salinas et al. (2023) uses gradient boosted trees to predict performance quantiles and provides prediction intervals with statistical guarantees through conformal prediction techniques.
Bayesian Optimization by Density-Ratis Estimation (BORE) Tiao et al. (2021 reformulates Bayesian optimization as binary classification problem. It trains probabilistic classifier to distinguish between high-pe gigh-performing and low pwrfforrrming configurations, then uses the predicted class probabilities to construct acquisition function equivalent to expected improvement.
Gaussian Process BO with Expected Improvement (GP-EI) Snoek et al. (2012); Balandat et al. (2020) employs Gaussian Process as a surrogate model to capture the objective function's behavior and uncertainty. It uses the Expected Improvement acquisition function, implemented via BoTorch, to balance exploration and exploitation when selecting new evaluation points.
B.2 SYNTHETIC BENCHMARKS
In Section 4 we initially evaluated HOLLM and the baselines on 6 synthetic deterministic function with varying dimensionality and nature. In Table we provide details on each of them.
B.3 HYPERPARAMETER OPTIMIZATION BENCHMARKS
For our hyperparameter optimization experiments, we evaluate on four tasks from the FCNet bench- mark Klein Hutter 2019): PROTEIN, NAVAL, PARKINSONS, and SLICE. The FCNet bench- mark provides tabulated hyperparameter optimization setting where fully connected neural networks are trained on each dataset with different hyperparameter configurations. The search space consists of 9 categorical hyperparameters (network architecture and training parameters), yielding 62,208 possible configurations with pre-computed validation accuracies. To enable KD-tree partitioning on the categorical search space, apply ordinal encoding to convert categorical variables into numerical
15

---

positive transformed values Published as a conference paper at ICLR 2026
Algorithm 3: HIERARCHICAL OPTIMIZATION WITH LLMS (HOLLM) - DETAILED
Data:
Initial data D ( ((x))) : batch size b, regions sample from M, proposal count per leaf k. dimension d, initial leaf size mp, adaptive growth rate X, total evaluations T, exploration weights $1, B2, annealing Amins Omaz
)
+ Ro
// global evaluation counter
2. while T do
3.
(optional) Miaa <- + [Alog(1 1 + +t]
// adaptive leaf size K,
Build KD-tree D with leafsize meats obtaining K, leaves (X)
5.
f mmn + (0 0man) (1 + (os(Tt/TT)
// cosine annealing
schedule
fmin + min f(x)) 6.
7.
Y+ + (2+ fmin =)
8.
for / = 1 to K₁ do
9.
n4 < [Xel
// number of points this leaf // normalized volume
(l,, uz) & bounds of cell X²
Ve I (uey led) 1/d
11.
12.
n > then
13.
[g maxiel [+]
// best value in cell
14.
if me then
15.
0 + I Eien (Y+[1] [Y+11-Ya) Y2)²
// variance in cell
16.
else
of 0.01 // default variance for single-point cells log ferm +
17.
18.
ne)))
& - (v20 20, log Lerm/err// + log Jerm//)
19.
20.
else
L M4, Vi, + 0,
11 high exploration for empty cells
1. V + min-max normalize pe, Ve, & across all cells
B, + a (31 V2 E,)
// composite score
24.
Pr = Bp/ E B,
// normalize score across cells Sample M cells (K, MM ~ Categorical ((pe))
25.
X 0, F+1
for j = to M do
(,, + LLM LLL__LLMEEEELEE GENERATE(D, (44, Ui, Mi,).k)
Append x; to X: append f to
29.
+ argsort(F) (F)
// indices of sorted values (descending)
X new + top b points from X using indices
31.
for each x E nnew do
33.
Evaluate y = f(x) D DU ( y)} t+t+1
35.
36.
return best point (x.", f(x")) where x = arg max f(x)
16

---

Published as a conference paper at ICLR 2026
Table I: List of synthetic optimization functions and their main characteristics.
Function (dim.)
Landscape key traits
Global boundary
Global optimum
Hartmann 3D
Smooth, strongly multimodal surface
(21, 22, Ra)
1J
fuin ~ -3.62778
generated hy four weighted Gaussians inside unit cube; narrow, steep hasins punish local search.
Hartmann 6D
Six Gaussians in [0, 1 create even denser constellation of deceptive wells; still smooth but mildly ill-conditioned, and the search space grows exponentially.
(I1, (,,,,,,.,, E [0, 1)
fmin = -3.32237
Rosenbrock 8D
Classic curved "banana" valley stretched eight variables; unimodal yet highly ill-conditioned, requiring precise valley-tracking; non-separable.
(I1, 22....,
fmin 0
[-2.048, 2.048)
Rastrigin 10D
Quadratic core overlaid with cosine ripples forms a perfectly regular grid of 10 local minima; separable but brutally multimodal, exposing algorithms prone to premature convergence. Sine perturbations on quadratic backbone yield wide plateaus, sudden cliffs, deep and non-separable, stressing step-size control.
E21.x2....,.
Smin 0
55.12,
Lévy 1OD
(I 22, F10) E
fmnn = 0
[-10, 10
Ackley 20D
Exponential of radius plus averaged cosines: vast flat region, encircling ridge, and single sharp basin at the origin; tests exploration versus exploitation in very high dimension.
(J1, .200 E [-32. 328768)
fmin 0
split indices. Below we describe the four regression datasets used as the underlying machine learning tasks:
PROTEIN is a regression dataset containing physicochemical properties of protein tertiary structures. The task involves predicting protein properties from molecular descriptors across 45,730 protein samples.
PARKINSONS contains biomedical voice measurements from individuals with early-stage Parkinson's 's disease participating in six-month telemonitoring trial. The regression target is the progression of Parkinson's symptoms, with 5,875 samples and 19 acoustic features.
NAVAL consists of simulated sensor data from naval frigate's propulsion system, including gas turbine, propeller, gearbox, and control systems. The regression task predicts component degradation levels using 11 11,934 samples with 16 operational features.
17

---

Published as a conference paper at ICLR 2026
SLICE involves predicting the relative axial location of CT scan slices within the human body. The dataset contains 384 features extracted from 53,500 CT images, describing bone structures, air inclusions, and anatomical positioning.
B.4 REAL-WORDE OPTIMIZATION TASKS
To evaluate the practical applicability of HOLLM, we consider three real-world global optimization problems from Tanabe and Ishibuchi Tanabe & Ishibuchi (2020). These tasks are commonly used benchmarks that originate from engineering and bioprocess applications, each formulated as continuous blackbox optimization problem with simulators of real systems. Below, we provide detailed descriptions of each task and in Table 3 the input parameter names for each task and their respective value ranges.
This task models the optimization penicillin fermenta- tion process. The objective is to maximize the penicillin yield, by searching in a 7D continuous design space, consisting of cul- ture volume, biomass concentration, tem- perature, glucose concentration, substrate feed rate, substrate feed concentration, and hydrogen ion (H+) concentration. This problem formulation has been widely used as benchmark global optimization due to its nonlinear dynamics and practical rel- evance in bioprocess engineering.
Penicillin Production.
Table 2: Search space of the FCNet benchmark, list- ing each hyperparameter and its possible categorical choices.
Hyperparameter
Categorical Configuration Space
Initial LR
(0.0005, 0.001, 0.005, 0.01, 0.05, 0.1)
Batch Size
{8, 16, 32, 64) (cosine, fix
LR Schedule
Activation (Layer 1)
(relu, tanh)
Activation (Layer 2)
[relu, tanh)
Layer Size
(16, 32, 64, 128, 256, 512) [16, 32, 64, 128, 256, 512) [0.0, 0.3, 0.6)
Layer 2 Size
Dropout (Layer 1)
Dropout (Layer
[0.0, 0.3, 0.6]
Vehicle Safety. This task models the de- sign of a vehicle's frontal structure to im- prove crash safety. The primary objective to maximize crashworthiness, evaluated in terms of structural performance under frontal impact. The problem is formulated 5D continuous optimization problem, where the decision variables correspond the thicknesses of different structural components in the front of the vehicle. These parameters directly influence crash responses such as weight, acceleration, and deformation of the passenger compartment, making it well-established surrogate prob- lem for safety-critical vehicle design.
Table 3: List of real-world optimization tasks and their main characteristics.
Task
Inpui variable
Range
Calture volume
(60.0, 120.0)
Biomas concentration Temperature
(0.05, 18.0)
(293.0.303.0)
Penicillin Production
Gluease concentration
(0.05, 18.0)
Subvtrate feod mite
(0.01.0.5)
(500.0, 700.0)
H" concentration
(3.0,6.5)
member
(1.0,3.0)
Thicknes of member 2
(1.0,3.0)
Veticle Crubworthiness (SD)
Thickness of member 3
(1.0.3.0)
Thickness of member 4
(1.0.3.0)
Thickness of member 5
(1.0,3.0)
B-pillar
(0.5,
Door heam thickness
(0.45, 1.36)
Flour side thicknoss
(0.5, 1.5)
Car Side Impact. This task models vehi- cle performance under a lateral collision. The primary objective is to improve side- impact safety through careful adjustment of structural design parameters. It is 7D continuous optimization problem, where the decision variables correspond to aspects of the car body that affect passenger safety during side impacts.
Cur Side Imquet (7D)
Cossi-enhers thickness
(0.5, 1.5)
Dour thickness
(0.875, 2.625)
Roof thickness (freat)
(..4.1.2)
Roof thickness (rear)
(0.4. 1.2)
18

---

Published as a conference paper at ICLR 2026
C ADDITIONAL EXPERIMENTS
In this section, we provide additional experiments and ablations, complementing the ones conducted throughout Section 4 of the main paper.
C.I RESULTS USING GEMINI- 1.S-H
In this section, we present some preliminary results we obtained by running HOLLM and the global L.M optimizer with Gemini-1.5- on 6 synthetic benchmarks, the FCNet hyperparameter optimization tasks, and neural architecture search on NAS-Bench-201 (Dong Yang 2020
C.I.1 SYNTHETIC BENCHMARKS
Sguuee/showe the results when running HOLLM (Gemini-1.5-Flaste and the other baselines on the same synthetic benchmarks as in Section 4 We additionally ran on Ackley-20D, which is characterized by flat regions and single sharp global minimum the origin. We ran HOLLM and all other baselines 3 times with different random seeds for 100 iterations. Similarly as with 2.0-Fah,,, HOLLM consistently outperforms or matches the baselines, even on Rastrigin. Most notably, Ackley-20D with input range -28,766, 32. 22.766820 HOLLM locates the global maximum in just 50 iterations, while baselines struggle to improve beyond random search. However, more repetitions necessary obtain more conclusive results from this experiment.
Additionally, we evaluated space partitioning methods such as TurBO (Eriksson et al., 2019) and LA-MCTS (Wang al. 2020 Yang et al.! 202T), as well other popular blackbox optimizers such as TPE (Bergstra et al. 2011), SMACv3 (Lindauer et al. 2022), CMA-ES (Hansen & Auger (
and HEBO (Cowen-Rivers Rivers al., 2022). For TurBO, LA-MCTS, CMA-ES and HEBO, we use their official implementations, whilst SMACv3 and TPE are in SyneTTune. Table Fable|4show
shows the mean and standard error of 5 runs after 100 iterations on synthetic benchmarks. As can be seen, HOLLM still superior compared to all baselines, including the space partitioning ones. Moreover, even when running LA-MCTS for 1000 function evaluations, still is not able find the global minimum in Ackley 20D, while HOLLM does that in less than 50 evaluations.
Harmmann
Fannt
-100
11
100
-300
Rasti
-.00
-100
01
42
15C
100
Figure 9: Best function value across 100 iterations on the synthetic problems. HOLLM outperforms or matches the performance of baselines, especially on higher dimensional problems (e.g., Ackley). For this experiment we used Gemini- 1.5- and the results show the mean and standard error of runs.
C.1.2 HYPERPARAMETER OPTIMIZATION
In Figure 10 we show the results HOLLM compared to the same baselines as the experiments in
Section 4 of the main paper, however, this time using Semini--1.5-Foe the LLM model inside
19

---

Published as a conference paper at ICLR 2026
Table Comparison of the performance HOLLM and the global LLM optimizer using Gemini- 1.5-Flash to other non-LLM-bann --bddeed global optimization methods. We ran each method for 100 trials (or 1000 when specified so) and report the mean and standard error of 5 repetition.
Method
Rosenbrock 8D
Rastrigin 10D
Ackley 20D
CMA-ES
-970.56 + 689.68
-125.85 9.18
-20.96 + 0.15
CMA-ES (1000 iters)
-117.21 + 35.05
-67.12 6.52
-19.24 = 1.16
CQR
-295.79 + 17.04
-111.68 2.14
-20.59 + 0.03
BORE
-270.70 + 4.03
-75.73 1.73
-20.49 = 0.01
GP-FI
-512.86 + 27.64
2.3.
+ 0.01
RE
-11.86 + 27.64
-
20.39
RS
-51.866 + 27.64
-107.63
0.01
(1000 iters)
-194778 67.66
.1.655
-0.055 1 0.17
TPE
-52.33 + 29.05
107.63
20.399 0.01
SMAC
298.60 + 7.15
104.40
20.09 0.03
HEBO
-7.22 + 3.07
4.56
TuRBO
161.61 + 11.23
-87.00
17.81 0.03
LA-MCTS (RS)
212.10 + 69.59
-124.71 45.59
11.91 0.45
LA-MCTS (RS) (1000 iters)
-22.19 + 6.42
-57.05 6.02
19.11
LA-MCTS (BO)
236.23 + 79.80
-12.533 13.96
-17.75
LA-MCTS (BO) (1000 iters)
-26.60 + 10.71
43.59 6.00
-13.16
LA-MCTS (TuRBO)
-9836 + 73.54
-50.67 = 12.46
-16.33 1.89
LA-MCTS (TuRBO) (1000 iters)
-27.33 + 24.40
-14.16 8.65
-3.51 + 1.01
LA-MCTS (CMA-FS)
-30.56 + 27.37
-67.28 + 10.43
-9.04 - 0.42
LA-MCTS (CMA-ES) (1000 iters)
-6.31 1.16
-25. 16.12
-1.71 - 1.06
LLM
-4.66 + 0.50
-32.08 + 6.84
-..10 + 1.30
HOLLM (ours)
-7.03 + 0.01
-0.17 0.04
-0.00 + 0.00
HOLLM and the global LLM baseline. We ran each optimizer 3 times with different seeds and report the mean and standard error after 100 iterations.
C.1.3 NEURAL ARCHITECTURE SEARCH
Neural Architecture Search (NAS) (White et al. 2023), like hyperparameter optimization, aims to identify the best-performing neural network architecture for given dataset by maximizing validation accuracy. We use the NAS- Bench-201 benchmark (Dong Yang 2020), which provides precomputed validation accuracies for all architectures ImageNet 16 x 16 (Chrabaszez et al.) 2017). This tabulated format enables efficient benchmarking by eliminating the computational overhead of training each architecture from scratch. The search space is 6D, with each dimension representing a discrete choice among 5 possible layer operations: avg avg_pool. pool_ool30 3x3 (average pooling), cor__oov__3_ 3x3 (normal 3x3 convolution). skip_connees (identity connection), nor nor_cony 1xl (normal Ix1 convolution). and none (no operation).
CIFAR-10, CIFAR- 100, and Downsampled
-200
Figure 10: Hyperparameter optimization on 4 datascts from the FCNet search space. All baselines from Synetune are evaluated asynchronously using workers. For this experiment we used Gemini- 1.5-Flast and the results show the mean and standard error of 3 runs.
20

---

Published as a conference paper at ICLR 2026
C52
C100
Irapihel33
1430
10.10
Figure 11: Results the NAS-Bench-201 6 dimensional discrete function. We plot the negative regret vs. the number of iterations. We run each method 6 times and report the mean and standard error.
We ran experiments using F.5-Fash We also use a continuous representation [0. 1] of the input space and discretize it evaluate the true function. As seen in Figure 11 HOLLM always outperforms the LLM baseline that samples globally and is on par with BORE and CQR. The global LLM seems to get stuck in local minima, therefore leading to stagnating performance from early on.
Toxation 32
Fox stoiion 12
Ser bo
0.8
10
-32
933
Call Gener
Cal caree
Iaration
boration
-32
Cell Pre
Seeeter Prbbbii
C4 Cyme
Troico
-2
-29
Call tedtemi
texcuee Phebeld Do
(a) ID Levy problem.
(b) ID Rosenbrock problem.
Figure 12: Illustrative examples of optimizing 1D functions. The rectangles represent the input space partitions (top row). They are highlighted in orange whenever selected based their respective probabilities (bottom row). Both experiments used 5 initial points and a batch size of 4. All new points (red stars) are suggestions from the LLM.
21

---

Published as a conference paper at ICLR 2026
C.2 ABLATIONS AND FURTHER ANALYSIS
To assess the robustness of our method and understand the influence of key hyperparameters on performance, we conducted comprehensive ablation study. We employ the 10D Levy test function and examine 3 hyperparameters that directly impact the exploration-explotaaion balance and efficacy of our approach: (i) maximum leaf capacity Meat mg + [og + t)1 (A 0), which controls the granularity of space partitioning; (ii) candidate sampling rate (proposals generated per selected region), which determines the diversity of proposals within each selected region; and (iii) region selection parameter M (partitions selected per iteration), which governs the number of promising subregions explored simultaneously per iteration. The default hyperparameter configuration also used throughout the experiments in the main paper is: exploration parameter bounds O'max = 1.0 and O'nin 0.01, initial random sampling phase of no 5 evaluations, batch size b (points evaluated per iteration), At = 5, M = 5, and maximum leaf capacity ma = d/2, where d denotes problem dimensionality. We run each setting with 5 independent random seeds and report the mean performance + standard error in Figure B
Levy na (Lea) Siz)
Levy (Number of candidates per regian)
Lovy (Number of Partitions)
Punctton volvee
-10
-10
-15
Fundtonn
-15
Function
-15
20-
AOLLID 31
HS HALLM D8 1.4
-25
-25
30
0
100
30
100
Number evn luatoos
Number evb Ualions
Number of ext knstions
Figure 13: Impact of key hyperparameters optimization performance for the 10D Levy function. Left: Ablation on leaf size (m²) demonstrates that coarse partitioning (mo = 1d) yields the poorest performance, while finer partitioning enables superior exploitation of promising regions through more granular space decomposition. Middle: Varying the number of candidates per selected region (k) reveals optimal performance at intermediate values, where undersampling (k - 1) significantly degrades performance by limiting exploitation of high-potentia regions, and oversampling (k- 10) slows convergence due inefficient allocation of evaluation budget. Right: The number of partitions
selected per trial (M) governs exploration breadth, where single-region focus (M = 1) impedes convergence through insufficient exploration, while moderate values (M E (3, 5, 7)) accelerate optimization by enabling simultaneous exploration of multiple promising regions.
Impact of Leaf Size (m²). The leaf size parameter mi defines the maximum number of data points within a single leaf of the partitioning tree, directly controlling the granularity of search space decomposition. Our analysis across different values of mo factor of problem dimensionality d reveals a clear trade-off between partition resolution and statistical reliability (Figure T3] left). Coarse partitioning with mo d yields suboptimal performance due to overly broad regions that group diverse areas of the search space, diminishing the method's 's ability to precisely isolate promising subregions. Conversely, extreme fine partitioning with mo = 1 also degrades performance because singleton regions provide insufficient statistical information and the variance component becomes small constant across all regions, eliminating valuable uncertainty estimates necessary to guide exploration. We observe the best performance at mo d/4, which strikes an effective balance by enabling detailed space partitioning while maintaining sufficient data density within each region to compute meaningful variance estimates for the exploration term.
Impact of Number of Candidates per Region (k). We investigated the effect of varying the number of candidate points i sampled from each selected region, testing values & E (1,3, 5, 7, 10}. Results shown in Figure 13 middle) reveal clear trade-off between under- and over-sampling within regions. Setting i = leads to significant performance degradation as the method fails to adequately exploit promising regions by drawing only a single sample per region. Conversely, k = 10 results in worse performance during initial iterations compared to intermediate k values, which we attribute to increased risk of oversampling sub-optimal regions in the beginning. While the method can recover from this scenario oversampled sub-optimal regions receive lower scores in subsequent iterations, the initial performance penalty demonstrates that excessive sampling can be counterproductive. These findings showcase the importance of balanced exploitation within selected
22

---

Published as a conference paper at ICLR 2026
Navel
Parlissrrs
Skce
-0000
-30003
-3299
-33005
-20
-000
-1300
12m
-40022
-1000
10
-339
Honce Faaaaator
(a) Gemini-1.5- 1.5-Flash
hava
Protett
-0.0003
-
-AX00
-3320
M0L Srad
-30
-AX03
-13000
-330
(h) Gemini-2.0-Flash
Figure 14: Performance comparison of exploration parameter settings (O {0.2, 0, 1.0)) across four FCNet benchmark tasks. Each curve shows the mean objective function value over 5 runs, with shaded regions denoting standard error. Results illustrate that the optimal choice of O'max is task dependent, reflecting different landscape characteristics and exploration needs.
regions:
sufficient sampling to capitalize promising areas without overcommitiing computational budget to potentially sub-optimal regions.
Impact of Number of Partitions Selected per Trial (M), We examined how the number of partitions M E (1,3,5,7) selected per trial affects optimization performance. As seen in Figure 13 right, setting M = notably hinders performance, particularly during initial iterations, as HOLLM severely restricts exploration breadth by focusing all sampling efforts on a single region per iteration. When the initially chosen region lacks global promise, progress becomes slow, though the method can eventually exploit good regions once identified, leading to convergence that is typically slower than broader exploration strategies. Conversely, increasing M to moderate high values (3, 5, 7) generally improves initial exploration by enabling simultaneous consideration of multiple diverse regions. This expansion of the candidate selection pool allows the algorithm to benefit from a larger, more diverse set of proposals per trial, improving performance up to saturation point. The results demonstrate that balanced multi-region exploration through appropriate M values provides superior performance compared to overly focused single-region strategies, highlighting the importance of maintaining exploration breadth while preserving the ability to exploit promising areas effectively.
C.2.1 IMPACT OF EXPLORATION PARAMETER Oynax
We evaluated the effect of different exploration settings on the FCNet benchmarks across four tasks: PROTEIN, NAVAL, PARKINSON, and SLICE. Results in Figure 14 show that the impact of the exploration parameter Omux exhibits desk-dependent variation, with optimal settings determined by the underlying problem structure. Higher values of Omax bias the search toward less-explored regions, proving beneficial for highly multimodal or non-convex landscapes where diverse exploration is crucial for escaping local optima. Conversely, lower Oimax values reduce exploration of new regions and concentrate search efforts on exploitation, which may be more appropriate for smoother or convex solution spaces where intensive local search around promising areas yields better returns. These findings suggest that prior knowledge of the task landscape characteristics, such as modality, convexity, and noise structure, can effectively guide the selection of O'max to match the exploration-exploitatin balance to the problem's 's inherent difficulty and structure.
C.2.2 EFFECT OF HYPERPARAMETERS ON COMPUTATIONAL COST
The computational complexity of our method scales directly with the total number of candidate points generated per iteration, calculated as is M. When employing computationally expensive models such as large language models accessed via API calls or hosted locally, increases in either M (which amplifies the number of inference calls per iteration) (which extends the number of output tokens
23

---

Published as a conference paper at ICLR 2026
generated per call) result in proportionally higher inference times and associated costs per optimization step. This creates fundamental trade-off between optimization performance and computational efficiency that practitioners must carefully consider based their specific resource constraints and performance requiremenss. Our default configuration of and 5, yielding 25 candidate evaluations per iteration, represents calibrated compromise that balances exploration capability with computational practicality across the diverse benchmarks in our experimental evaluation.
D SAMPLE DIVERSITY ANALYSIS
To better understand the performance gap be- tween HOLLM and the global LLM baseline, we analyze the sampling diversity of the new proposals generated during optimization. Our goal is to investigate and quantify how strongly each method biased towards previously ob- served points versus exploring new regions of the search space. To this end, we introduce the ICL Divergence metric DICL. which is simply defined as the average Euclidean distance be- tween each newly generated candidate point z; and the set of historical points Iÿ provided as context:
Dier 0.12
Dier 0.28
x
x
x
x1
X ICL Examples
Proposala
Figure 15: ICL Divergence Illustration: An il- lustration of two distinct search behaviors as mea- sured by the ICL Divergence (DICL). The left panel shows an exploitative search with low DICL score, while the right panel shows an exploratory search with a high DICL score. The score the average distance (red lines) from each proposal its near- est ICL example.
M DICL. Σ
min |00 1=1 2,
22
M
where C the set of in-context examples shown to the model. A decreasing DICL indicates that the model re-sampling increasingly closer variants of earlier points, i.e., collapse of the proposal distribution toward previously seen optima. In Figure 15 we illustrate different instances of
this metric and how is computed.
Figure 16 shows the evolution of DICL on the real-world benchmarks from the main paper. Across tasks, the global LLM baseline exhibits a rapid decay of DICI, toward zero. This confirms that the vanilla LLM increasingly produces near-duplicate points around previously observed high- value candidates. In contrast, HOLLM maintains a substantially higher DICL throughout the whole optimization. The KD-tree partitioning prevents global collapse by constraining each LLM query to a specific region of the search space. Combined with global context conditioning and bandit-driven region selection, this structure encourages local refinement without allowing samples converge prematurely across the entire domain.
Results.
CarSidslmpact
Penteilim
IC1. Divengmece
0.4
ICL, Divergenne
0.9
8.6
0.8
0.4
0.2
0.2
0.0
2
10
Trials
Trink
HOLUM
LIM
HOLLM
L.S.
(a) CarSidelmpact
(b) Penicillin
Figure 16: DICL trials the a) CarSidelmpaac and b) Penicillin benchmarks. The global LLM baseline rapidly collapses toward zero diversity, while HOLLM maintains broad exploration.
24

---

Published as a conference paper at ICLR 2026
E ABLATION STUDY: SCORING, SELECTION, AND SAMPLING
To reinforce the results shown Figure 8b
the main paper, Figure 17 we show that consis- tent trends were observed in preliminary experiments NAS-Bencih 201 using Gemini-I.5-Flasi confirming that the UCB-V formulation is robust across different problems. In the same figure, also compare HOLLM against two variants: 1) KD-Tree RS: The LLM sampler is replaced with uniform random samples within the selected subregion. 2) KD-Tree + GP: The LLM sampler is replaced by Gaussian Process (GP) Surrogate which the partition history to fit local model, followed by Expected Improvement (EI) maximization within the selected subregion. We see that HOLLM significantly outperforms both variants, reconfirms that the LLM's in-context learning ability, when localized, provides unique and substantial benefit over both pure exploratory technique (RS) and a standard, highly competitive local BO surrogate (GP-EI).
F SURROGATE CALIBRATION ANALYSIS (LLM vs. GP VS. TABPFN)
A key component of HOLLM is using the LLM not just as a candidate sampler, but as surrogate model too to predict function values and rank candidates within region. To validate this design choice, we compared the LLM's predictive calibration against Gaussian Processes (GPs) and the recent TabPFN tabular foundation model (Hollmann et al. (025). In this experiment, we maintained the LLM candidate sampler and only replaced the surrogate model with the aforementioned models:
N8201-CIFAR100
HOLLM.
HOLLM canty esptore)
HOLLM (unifom))
HOLLM (UCB1)
HOLLM tonly explost)
Kottes RS
Kones GP
1. HOLLM GP: Uses the mean function value of the GP as the predicted function value. The train- ing data is the same the in-context examples given to HOLLM.
0.0
1.5
Regret
Figure 17: Ablation of the Scoring on the CIFAR-100 neural archi- tecture search task. The results show the mean and standard error of 6 repetitions. KDTree+RS and KDTrec+GP replace the LLM surrogate and candidate sampler with RS and GPs, respectively.
2. HOLLM + TabPFN: Uses TabPEN (Hollmann
et al. 2025), a Transformer- foundation model pretrained entirely synthetic tabular data. TabPFN provides quantile predictions and in our case, use the median as our predicted function value.
Figure 18 shows the Quantile-Quantile (Q-Q) plots of predicted values vs. ground truth the real-world benchmarks. These results were directly taken from our follow-up paper on Multi- objective HOLLM (Schwanke et al. 2026). The LLM surrogate (Red) consistently aligns with the diagonal, indicating robust calibration. GPs (Blue) tend to underestimate the spread on Penicillin, while TabPFN (Green) shows significant misalignment, suggesting its prior is less suited for these continuous engineering landscapes. In T we also report the regression (R², MSE) and ranking (Kendall 7, Spearman p) metrics for this experiment, computed using 700 data points total. The LLM achieves significantly higher ranking correlations (c.g., T ze 0.76 on VehicleSafety) compared GPs (T 0.71) and TabPFN (T 0.29). This confirms that the LLM's meta-learned prior provides superior signal for ranking proposals in sparse, localized regions.
G COMPUTATIONAL COMPLEXITY ANALYSIS
Due to the high inference time of LLMs, the primary bottleneck in the pipeline the M sequential LLM requests per iteration. Since both HOLLM and the global LLM baseline condition on the full history t points, the context length effectively identical. Consequently, the attention mechanism cost (t22) (or the associated API latency) is equivalent for both methods. HOLLM only adds negligible constant number of tokens to the prompt to define the local region boundaries. As shown in Table 6, the partitioning and scoring mechanisms introduce negligible overhead compared to the LLM inference time. HOLLM requires the same number of inference steps as the global LLM baseline, since they both generate the same number of points per iteration.
25

---

Published as a conference paper at ICLR 2026
Ored
1A49%
FOLLM -SAPP
WOLLD GP temmppc
Priject Direttat Sieaa
ZOLUM
Prites Dieed alo Strch
DOLUM
Priin Dbeet Slenk
JULLM
ZOLLM
(a) CarSideLmpact
(b) Penicillin
(c) VchicleSafety
Figure 18: Q-Q Plots of Surrogate Calibration. We compare the predictive distribution of the HOLLM LLM surrogate (Red) against substituting it with a Gaussian Process (Blue) or TabPFN (Green). The LLM consistently aligns closest the diagonal, indicating superior few-shot calibration.
Table 5: Quantitative Analysis of Surrogate Performance. We report regression metrics and ranking correlations for the Penicillin, Vehicle Safety, and Car Side Impact tasks. The LLM consistently outperforms GP and TabPFN baselines in ranking capability (Kendall т, Spearman p) and regression accuracy (R²).
Task
Method
R
MSE
Kendall r
Pearson r
Spearman
N
HOLLM (Ours)
0.742
8.06
0.684
0.868
0.851
700
HOLLM + GP
-9,721
314,59
0.274
-0.075
0.371
700
Penicillin
HOLLM + TabPFN
-173.697
5184.01
0.375
-0.138
0.487
700
LLM (Global)
0.756
6.20
0.730
0.875
0.854
700
HOLLM (Ours)
0.806
7.57
0.758
0.899
0.903
700
HOLLM + GP
-9038005
402521.05
0,709
-0.014
0.719
700
Vehicle Safety
HOLLM + TabPFN
16837.928
1272547.79
0,286
-0.011
0.306
700
LLM (Global)
0.874
7.21
0.869
0,936
0,955
700
HOLLM (Ours)
0.877
5,53
0.786
0,938
0,931
700
HOLLM + GP
0.010
43.35
0.781
0,679
0,802
700
Car Side Impact
HOLLM TabPFN
-5.616
242.86
0.061
0.176
0.037
700
LLM (Global)
0.807
6.35
0.745
0.901
0.887
700
Furthermore, we highlight that this approach offers advantageous asymptotic scaling compared to standard Bayesian Optimization. Exact Gaussian Process (GP) inference requires Cholesky decomposition with cubic complexity (t9)) In contrast, the Transformer attention mechanism scales quadratically (f2). While LLMS exhibit a higher constant latency factor, this difference implies that HOLLM is asymptotically more computationally efficient than exact GP-based methods as the optimization horizon grows.
Table Complexity Analysis of HOLLM, Global LLM, and GP Surrogates. f denotes the total number of observed points, L the number of leaf regions (with L s I/K), and M the number of candidate points generated per iteration.
Component
HOLLM
Global LLM
GP Surrogate
KD>-rce Maintenance
Otggt)
Region Scoring/Selection
O(L)
Surrogate Update
O (
Query/ Sampling Cost
O(Mt²)
O(MP))
Relative Cost
Dominated by LLM inference
Dominated by LLM inference
High for large
H IMPACT OF LLM CAPABILITY ON DIFFERENT DOMAINS
In the main text, we demonstrated on the Vehicle Safety task that HOLLM exhibits high robustness
to the underlying LLM's capability, performing well even with smaller models (e.g., Llama-3-8B) where the global LLM baseline fails. In this section, we extended this ablation to two additional benchmarks: CarSidelmpact (Figure 19a) and FCNet-Parkinson (Figure 19b).
On CarSidelmpact, regardless of the LLM model, HOLLM improves on the global LLM baseline. These experiments confirm that HOLLM, via its internal subroutines, reduces the reasoning
26

---

Published as a conference paper at ICLR 2026
CarSidelmpact
Parkinsons
0.00
17.5
-0.01
Function value
-20.0
Function value
-0.02
-22.5
LLM
-0.03
LLM
1154 (1-00-111--0
(ilemedd 1-00)
-25.0
L154 (ulame 3.3-7.00000.
LLM (Qwen3 300 4385
-0.04
LLM (Owen3 00003333--000 438)
-27.5
HCLUX
HCLUM
HGLLI Itleme-311000
-0.05
HILLIM (ilem-311-88)
-30.0
HOLUM 10wen3 A08-
HCLLUM [Umme-3
(ICLUM (1mmm-333-700)
HOLLM (0we:3:309.438)
-0.06
10
20
30
40
50
10
20
30
40
50
Number of evaluations
Number of evaluations
(a) CarSidelmpact (Continuous)
(b) FCNet (Discrete)
Figure 19: Performance across LLM Capabilities. We compare HOLLM vs. global LLM using
various LLM models.
plexity required from the LLM, enabling efficient models (e.g., Llama-3.1-8B) to solve tasks that typically require significantly larger compute budgets.
On the FCNee-Parkinsons hyperparameter optimization task, we observe a distinet "ceiling effect" with Gemini-2.0-Flan which solves the task almost immediately. With the open-weights models we observe distinct behaviors between different LLM models that highlight the complexity of solving discrete HPO using LLM-based optimization. This comparison demonstrates that while HOLLM universally beneficial in continuous domains (where the KD- Trer partitioning is straightforward), its interaction with discrete landscapes is more dependent on the LLM type rather than the search heuristics.
I PROMPTS
We design structured prompts to LLMs for both candidate generation and evaluation prediction within our optimization framework. Our prompting strategy consists of two complementary components: candidate generation prompt [Listing 3) that produces new solutions based on historical observations, and an evaluation prediction prompt (Listing 4) that estimates the quality of proposed candidates, or both in a single prompt. Each prompt follows systematic structure:
1. Task specification: We provide a description of the optimization objective and establish the problem context, including nature the search space and optimization goals.
2. Dynamic constraints: We define the feasible region constraints (boundaries) derived from the current KD tree partitioning. These constraints are automatically computed based on the selected leaf nodes and translated into natural language descriptions that specify valid input ranges alongside ssamplc-evaluation pairs.
3. In-context examples: We supply the model with historical observations consisting of previously evaluated points and their corresponding objective function values. These examples serve as demonstrations to guide the model's understanding of the optimization landscape and desired output format. Using global history is necessary for two reasons: 1) Better prediction in sparse regions: New or small regions often contain too few points for effective in-context learning. Global history ensures the LLM always has sufficient examples to reason about the function's general behavior, 2) Robustness: Global points act as anchors, preventing the LLM from "overfitting' to local patterns or misinterpreting a local maximum as global one.
4. Task-specific instructions:
We provide explicit directives tailored to each prompt's purpose. The candidate generation prompt instructs the model to propose new configurations, while the evaluation prediction prompt directs estimate performance for given candidates. Both prompts enforce structured ISON output formatting for automated parsing of model responses.
Throughout our prompts, we employ placeholder variables denoted with $ symbols to represent task-specific information that is dynamically populated during execution. Comprehensive examples
27

---

Published as a conference paper at ICLR 2026
and descriptions for each placeholder are provided in 7 For the NAS-Bench- experiments, we adopt a streamlined approach using unified prompt structure (Listing 5) that simultaneously elicits both candidate proposals and their predicted evaluations in single model query, reducing the computational overhead separate generation and prediction phases.
Table Description of placeholders for candidate proposal and prediction prompts.
Placcholder
Description
Example of Replaced Text
$metrics
The performance metrics for the specific task.
(lower is better)
$region_constraint
The allowable ranges or discrete values for parameters the configuration search space.
1r: range (float ([0.a
0.911
activation:
choice ( ("relu", "tanh"]), num_layer: range (int ([1, 201))
Crcgloon_ILL_xxanae
Examples of previously evaluated configurations and their performance metrics. These are the in-context learning examples.
{
es
flr: 0,4, activation: "relu", num_layer:
.,...)
Fl: 5.65
(ir: 0.03, activation: "tanh", num layer:
.,...
Fl: 3.23
Starget_number_ce dandidates
The numher of configurations that 15 the candidate sampler should generate.
$candidate sampler prosponse_format
The required JSON structure for each new candidate configuration proposed by the sampler.
1r: act vativation ?, num_layes ?
Searget_architer aectuutcttttuuuu
The set of new configurations for which the surrogate model predicts the performance metrics.
res
1: { iir: 0,4,
act activation ion: "relu",
num layer: 8,...
2: 1ir: 0.03,
activation: "tanh", num nam_aayer:
8, 8,...)
$surrogate_model. esponse format
The required ISON structure for the performance prediction output.
IFl:
}
28

---

Published as a conference paper at ICLR 2026
Suggest 100 randon samples for dimensions within the specified bounding box with a maximum of 3 decimal places.
Bounding Box:
x1_min: 0, ma_max:
x2 min 0, max:
x3 c3_min: 0, x3 max:
x4_min: 0, x4 max:
x5_min: 0, x5_max:
x6_min: 0, x6_maxx
x7 x7_min: 0, max:
mini 0, max:
Return the suggestions the following JSON format exactly, without any additienal text: 11"x1" float, "x2": float, "x3": flost, "x4": float, "x5": float, "x6"; float, "x7": float, "x8": float]]
Listing I: Prompt for LLMs simulating 100 8-D uniform random samples.
Suggest 80 sample points in 2 dimensions within the specified bounding box.
Bounding Box:
x1_min: 0, xl_max:
x2_min: 0, x2_max:
such that they are clustered around the points given below. Points:
Point 1: xl: 0,25, x2: 0.25
Point 2: xl: 0.75, x2: 0.75
Return the auggestions in the following JSON format exactly, without any additional text: [1"x1": float, "x2": float)]
Listing 2: Prompt for LLMs sampling 80 points around the minima.
aptimization task
# Problem Description
You are tasked with solving a optimization problem that requires finding optimal sclutions.
E+Eaalaation:: Configurations are measured by Smetrics
## Constraints
The allowable ranges for the hyperparameters are:
rregion_constraing
## Previously Evaluated Architectures
Below are examples of architectures that have been evaluated, showing their operations and performance metrics:
$Region_ICI_exampll
## Your Task Generate
architecture configurations
that:
1. Are likely to achieve lower Smetrics than the examples
2. Are different from all previcusly evaluated architectures
3. Satisfy all the speci fied constraints: $regiion_constraints
29

---

Published as a conference paper at ICLR 2026
When the region has negative constraints, make
take this into account when proposing candidate value.
Before providing the final output, you MUST perform a check to ensure every single parameter in your proposed configurations strictly adheres to the constraints specified.
Any configuration with value cutside its valid range is incorrect and will be rejected.
## Output Format Each configuration has to follow this format:
format
Provide response in JSON list containing each preposed configuration.
Return only the required JSON list output without additional text.
Listing 3: Prompt template used for candidate points generation in the LLM Generate function.
Configuration Performance Prediction
## Problem Description
You are tasked with predicting the performance of contigurations.
Evaluation Metric*:: Smetrics (to be predicted) Constraint The allowable ranges for the hyperparameter are: $Region__CCi_xxmmpees
# Reference configurations with Known Performance
Below examples of configuraticns that have been evaluated, showing their operations and performance metrics:
## Candicate configurations to Evaluate
You must predict performance for these new configurations:
starget_architectuur
## Your Task
1. Predict the $metrics value for each candidate configuration
2. Base your predictions on patterns the reference examples
I| Output Format
Each evaluation has to follew this format:
Ssurrogate_moder secteee_rrpppo__frrreeee format
Provide your response in JSON list containing each proposed evaluation. Return only the required JSON list cutput without additional text.
Listing 4: Prompt template used for performance prediction in the LLM Generate function.
Suggest 8 new candidate point (s) for maximizing a blackbox function in a 6-dimensional search space.
Below are some examples of previcusly evaluated points with their corresponding function values:
"x1": 030
"x2": 0.287,
"x3": 0.773,
"x4": 0.175,
"x5": 0.75,,
"x6": 0. 60608,
30

---

Published as a conference paper at ICLR 2026
"value": -37.093
"x1": 0.199,
"x2": 0.433,
"x3": 0.405, "x4": 0.779,
"x5": 0.186,
"x6": 0.594,
"value": -37.84
"x1": 0,447,
"x2": 0.342,
"x3": 0,97,
"x4": 0.087,
"x5"; 0.115,
"x6": 0.533,
"valuc": -2
"x1": 0. 949,
"x2": 0.127,
"x3": 0.659,
"x4": 0.546,
"x5": 0.049,
"x6"; 0.265,
"value"; -33.067
The search space is defined by the following bounding boxes:
xi_min: 0,492, x1_max: 1.000
x2_min: 0.000, x2_max: 1.000
x3_min: 0.000, x3_max: 1.000
x4_min: 0.000, x4_maxx 1.000
x5_min: 0.000, x5_max: 1.000
x6_min: 0.000, x6_max: 1.000
Based on the examples above, suggest candidate points that balance exploration (sampling new zegions) with exploitation (focusing promising areas where function values are good) Each candidate point must 1ie within the specified bounding boxes. addition, predict an estimated function value for each candidate.
Return the suggestions in the follewing JSON format exactly, without any additional text:
[1"x1": float, "x2": float, "x3": float, "x4": float, "x5": float, "x6": float, "value": float:]
Listing 5: Prompt example used for simultaneous candidate generation and performance prediction in the LLM Generate function for NAS- BensBench--0.
31


Reviews:
[{"review_id": "zA47P5NBgw", "reviewer": "Reviewer_Uiwf", "summary": "The authors propose a method called HOLLM that partition the space of search using KD-Tree and then using a bandit-style algorithm that select from which part of the space sample new candidates points using LLMs. The motivation of the method bases on the observation that LLMs are not good by covering the space optimally. The authors show extensive results, including a comparison with a vanilla LLM implementation.", "questions": "See weaknesses", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- Authors showcase extensive results demonstrating the applicability of HOLLM.\n- The authors provide ablation studies of the design choices on the appendix.\n- In general the paper is well written and presented.", "weaknesses": "- My only doubt is that the authors develop a method based on the main motivation that LLMs are not efficient covering the space of solution. However, it seems the method not necessarilly needs an LLM in its design. On this point why LLMs are revelant for this method? My first assumption would be because LLMs have strong inductive bias about the problem and make them sample more efficiently. However, the inductive biases would be given by the context that is given to the LLM. This is a point that is not properly discussed in the paper. Can you elaborate on this point?\n- If HOLLM actually needs a LLM to perform well then it seems that is necessary a more extensive comparison between HOLLM and the LLM baseline to understand better why the first perform better than the second. Some intuition is given as motivation, but I guessed some metrics can be tracked to prove this point with the actual experiments.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "4196I7hWMA", "reviewer": "Reviewer_dF12", "summary": "This paper addresses a key limitation of using Large Language Models (LLMs) for global optimization: their tendency to sample sparsely and inefficiently in high-dimensional search spaces. The authors propose HOLLM (Hierarchical Optimization with Large Language Models), a novel algorithm that integrates LLM-based candidate sampling with an adaptive search space partitioning strategy. HOLLM iteratively builds a KD-tree to divide the search space into smaller subregions. It then uses a bandit-inspired utility score to select the most promising subregions, effectively balancing exploration (large or uncertain regions) and exploitation (regions with good observed values). An LLM is then prompted to generate new candidate points specifically within these selected, smaller regions. The authors' empirical results on benchmark functions demonstrate that HOLLM matches or outperforms other global optimization methods and significantly improves upon a baseline \"global LLM\" sampler.", "questions": "- What is the computational complexity (e.g., wall-clock time) of the HOLLM framework, specifically the KD-tree construction and region scoring, relative to the (presumably expensive) LLM sampling calls? How does this overhead scale as the number of evaluated points t increases?\n- The paper's motivation (Fig. 1) suggests LLMs are bad at sampling (i.e., biased). The method's success implies this bias is now a useful \"prior\". Could you elaborate on what specific, useful properties this \"meta-prior\" is assumed to have, especially when constrained to small subregions?", "limitations": "", "rating": 4, "confidence": 5, "soundness": 2, "presentation": 3, "contribution": 1, "strengths": "- The paper clearly identifies and demonstrates a practical weakness of LLM-based samplers—their high bias and inability to cover a space effectively (as shown in Figure 1)—and proposes an intuitive solution.\n- The inclusion of a \"global LLM\" baseline is crucial, as it provides strong evidence that the partitioning framework itself, not just the use of an LLM, is responsible for the performance gains", "weaknesses": "- The paper's primary contribution appears incremental. The core components—hierarchical space partitioning (e.g., KD-trees) and bandit-based region selection—are well-established techniques in the black-box and hierarchical optimization literature (e.g., HOO, MABs). The method seems to primarily substitute a traditional sampler within this framework with an LLM, which may limit the work's fundamental novelty.\n- There is a notable disconnect between the paper's motivation and its empirical evaluation. The introduction suggests LLMs could unlock optimization for novel domains (e.g., language-based tasks) where traditional numeric methods are unsuitable. However, the experiments are confined to standard numerical benchmarks (synthetic functions, hyperparameter tuning) where classic methods are already effective. This evaluation fails to substantiate the primary motivating claim for using LLMs.\n- The framework's components, apart from the LLM sampler, are standard building blocks in black-box optimization. This reinforces the concern about incremental contribution, as it is unclear why an LLM is a necessary choice. The partitioning and selection mechanism  could seemingly be combined with other advanced samplers (e.g., from PSO, DE, or even a simple Gaussian sampler). The paper lacks a crucial ablation study to isolate and justify the unique benefit of using an LLM over these simpler, well-understood alternatives.\n- The results on synthetic benchmarks are difficult to interpret and potentially unconvincing. The authors should explicitly state the known global optima for these functions. There is a concern that some functions may have trivial solutions (e.g., $x^*=0$) that an LLM, due to its inherent biases, might guess easily. This suspicion is heightened by the surprisingly strong performance of the \"global LLM\" baseline, which reportedly outperforms most numeric methods (Table 4). This counter-intuitive baseline result calls the benchmark's difficulty and validity into question.\n- On the hyperparameter optimization tasks, the proposed HOLLM does not show a significant advantage over the simpler \"global LLM\" baseline. Given that the global method is far less complex, this result questions the practical utility of the added HOLLM framework. The paper needs to provide a cost-benefit analysis, including the computational overhead of the partitioning and scoring components. Furthermore, the work fails to disentangle whether the observed performance (of both HOLLM and the global baseline) is due to the algorithmic structure or simply the LLM's powerful, pre-trained \"intrinsic knowledge\".", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "dy3BjVanxm", "reviewer": "Reviewer_Y2NU", "summary": "- This paper introduces HOLLM, a global BBO algorithm that uses adaptive search space partitioning with LLM-based hypothesis proposal (within selected subregions) and evaluation (global)\n- It uses a bandit-style metric at the partition-level to trade off exploitation, geometric coverage, and statistical uncertainty\n- The algorithm follows the process of (1) partitioning the space, (2) ranking leaf regions, (3) sampling from within the regions of selected partitions, (4) using an LLM to propose local points, (5) evaluating and selecting the best proposals", "questions": "- The LLM surrogate does not include any uncertainty. How good is the mean prediction, especially with LLMs of different capabilities\n- Can the authors clarify the computational overhead (e.g., wall-clock time) for each aspect of their pipeline (e.g., dynamic re-partitioning, local LLM-based BO)\n- It would be interesting, but not a priority, to see results on benchmarks with more noise and understanding how the partitioning + variance-sensitive scoring function performs\n- How limiting are the axis-aligned partitions? Would this be applicable to different problems and where might it be a bottleneck?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "- The paper gives an intuitive and empirical evidence of LLM failure modes in high-dimensional spaces (biased coverage, mode-seeking), which it addresses through searching/sampling within bounded subregions\n- The algorithm makes sense, combining KD-tree partitioning, UCB-style scoring/selection over the partitions (as arms), and LLM-buided local BO. While each of those components exist in some form in BO/MAB/hierarchical bandit literature, the particular design feels suitable and well-motivated\n- The arm-level scoring function and stochastic selection policy is novel (AFAIK), based on best-observed improvement (exploitation), normalized HV (geometric exploration), and UCB-V (uncertainty)", "weaknesses": "- The arm-level scoring functions feels slightly ad-hoc and slightly complicated, it is not clear why the HV-based term and UCB-V (as opposed to other UCB variants) is necessary and used. The HOLLM vs global LLM-based BO is a good ablation, but it would also be interesting to isolate the different terms (e.g., UCB-variance) to see if they matter in practice.\n- It would be interesting to understand the LLM performance specifically, e.g., how well the surrogate performance is vs a GP\n- One possible limitation of this work is that even with KD-tree partitioning, this might degrade in higher-dimensional spaces. Based on my understanding, the paper's benchmarks are mainly < 10D, it is not clear whether this partitioning will give proportional empirical improvements (say in 50D) where TURBO-style trust regions or CMA_ES covariance structures might be more robust\n- It is not clear to me why the LLM sampling/surrogate sees the global results, and not just region-specific results\n- Minor: the cosine annealing for exploration is sensible, but we don't see quantitative ablations around sensitivity.\n- Minor: how are the number of partition regions selected (I might have missed this), it feels like an important hyperparameter.\n- Minor: in the results section, it is a bit confusing to interpret (with maximize vs minimize), it would be hlepful to standardize sign convention and introduce visual hints\n- Minor: this is not a major concern in my mind, but as the work draws parallels to HOO/bandit-style UCB-V, it is worth pointing out that this field of literature cares particularly about regret/concentration analysis.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "SYaphIM8cI", "reviewer": "Reviewer_LeUX", "summary": "The paper addresses black-box global optimization with LLMs and argues that asking an LLM to propose points globally often yields sparse, biased coverage—especially in higher dimensions or without domain priors. It introduces HOLLM, a loop that (i) adaptively partitions the search space with a KD-tree, (ii) treats each leaf as a meta-arm, (iii) scores leaves by combining best observed improvement, geometric size, and a variance-aware exploration bonus, and (iv) stochastically selects regions before prompting the LLM to generate local candidates, which are then evaluated and fed back. The method re-fits the KD-tree each round to avoid premature localization and uses a cosine-style schedule to front-load exploration. Experiments span continuous synthetic benchmarks (Hartmann, Rosenbrock, Rastrigin, Lévy), FCNet-9D hyperparameter optimization on four datasets, and three real-world continuous tasks (Penicillin, Vehicle Safety, Car Side Impact). HOLLM typically matches or shows modest gains over BO/ES and global-LLM baselines, but it is not consistently dominant.", "questions": "1. Experimental results are not particularly strong in my view. Based on the experimental evidence, can you provide a detailed, results-driven explanation of what the algorithm actually contributes, with pointers to the specific tasks and figures?\n2. The multi-LLM ablation is only on Vehicle Safety. Can you repeat it on more tasks (1 additional continuous benchmarks and 1 discrete task)?\nI would consider increasing my score if you both (i) provide a detailed, results-grounded explanation of the algorithm’s contributions (answering my first question), and (ii) expand the LLM ablation to more tasks.", "limitations": "", "rating": 4, "confidence": 3, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "1. Methodological novelty with a clean, coherent design. The paper marries KD-tree–based adaptive space partitioning with a bandit-style composite scoring rule—combining best observed improvement, geometric volume, and a variance-aware (UCB-V–like) exploration term—and then prompts the LLM to generate candidates locally within selected regions. Re-fitting the partition each round helps prevent premature commitment and keeps the search adaptive. Overall, the loop (partition, score, select, local LLM proposals, evaluate) is conceptually simple yet technically sound.\n2. Across diverse tasks, HOLLM’s trajectories are comparatively stable against strong baselines. Notably, on many settings the global-LLM variant tends to stall, whereas HOLLM continues to improve. This is likely because “partition + scoring + local sampling” mitigates sampling bias in LLM-guided optimization and reduces brittleness to model idiosyncrasies.\n3. Wide experimental coverage with appropriate baselines. The evaluation spans continuous synthetic functions, FCNet-9D hyperparameter optimization, and three real-world continuous tasks, and it compares against a solid suite of methods: BO (e.g., CQR, GP-EI), evolutionary strategies, random search, a carefully controlled global-LLM baseline, and RS+KD-Tree.\n4. Component and hyperparameter ablations that aid interpretation. Both the main text and appendix report ablations/visualizations for key knobs, such as the exploration annealing schedule (α), batch/selection sizes (b/M/k), and leaf capacity. This helps clarifying how each component contributes to performance and stability.", "weaknesses": "1. Average gains are modest. On continuous tasks, HOLLM is clearly best only on a subset (Lévy, Rosenbrock, Car Side Impact, Vehicle Safety). On the other continuous benchmarks, there exist methods that outperform it by a visible margin; several curves overlap within error bars, so the advantage is not consistent. On FCNet-9D (discrete HPO), HOLLM and the global-LLM baseline are essentially overlapping—partitioning brings little benefit in this regime. Because the paper does not provide theory, the empirical results carry the core argumentative burden. Since improvements on continuous tasks are modest, and those on FCNet are near zero, it’s hard to argue broad superiority; the main benefit appears to be robustness rather than overall performance.\n2. LLM ablation is under-scoped. The multi-LLM comparison appears only on a single task (Vehicle Safety). To demonstrate the practicality of the algorithm, the paper should repeat the ablation across multiple continuous benchmarks.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)