Decision Tree Induction Through LLMs via Semantically-Aware Evolution
Authors: Tennison Liu
Nicolas Huynh
Mihaela van der Schaar
Venue: ICLR 2025 Poster (2025)
Keywords: decision trees
LLMs
genetic programming
Primary area: optimization

Abstract: Decision trees are a crucial class of models offering robust predictive performance and inherent interpretability across various domains, including healthcare, finance, and logistics. However, current tree induction methods often face limitations such as suboptimal solutions from greedy methods or prohibitive computational costs and limited applicability of exact optimization approaches.
To address these challenges, we propose an evolutionary optimization method for decision tree induction based on genetic programming (GP). Our key innovation is the integration of semantic priors and domain-specific knowledge about the search space into the optimization algorithm.
To this end, we introduce $\texttt{LLEGO}$, a framework that incorporates semantic priors into genetic search operators through the use of Large Language Models (LLMs), thereby enhancing search efficiency and targeting regions of the search space that yield decision trees with superior generalization performance. This is operationalized through novel genetic operators that work with structured natural language prompts, effectively utilizing LLMs as conditional generative models and sources of semantic knowledge. Specifically, we introduce $\textit{fitness-guided}$ crossover to exploit high-performing regions, and $\textit{diversity-guided}$ mutation for efficient global exploration of the search space. These operators are controlled by corresponding hyperparameters that enable a more nuanced balance between exploration and exploitation across the search space. Empirically, we demonstrate across various benchmarks that $\texttt{LLEGO}$ evolves superior-performing trees compared to existing tree induction methods, and exhibits significantly more efficient search performance compared to conventional GP approaches.

Published as a conference paper at ICLR 2025
DECISION TREE INDUCTION THROUGH LLMs VIA SEMANTICALLY-AWARE EVOLUTION
Tennison Liu', Nicolas Huynh" & Mihaela van der Schaar DAMTP, University of Cambridge
Cambridge, UK
(t1522, nvth2, mv472) cam. ac uk
ABSTRACT
Decision trees are crucial class of models offering robust predictive performance and inherent interpretability across various domains, including healthcare, finance, and logistics. However, current tree induction methods often face limitations such as suboptimal solutions from greedy methods or prohibitive computational costs and limited applicability of exact optimization approaches. challenges, we propose an evolutionary optimization method for decision tree induction based on genetic programming (GP). Our key innovation is the integration of semantic priors and domain--peciffc knowledge about the search space into the optimization algorithm. To this end, we introduce LLEGO, a framework that incorporates semantic priors into genetic search operators through the use of Large Language Models (LLMs), thereby enhancing search efficiency and targeting regions of the search space that yield decision trees with superior generalization performance. This is operationalized through novel genetic operators that work with structured natural language prompts, effectively utilizing LLMs conditional generative models and sources of semantic knowledge. Specifically, we introduce crossover to exploit high-perffee ter--oerfoniin regions, and diversity- guided mutation for efficient global exploration of the search space. These operators are controlled by corresponding hyperparameters that enable a more nuanced balance between exploration and exploitation across the search space. Empirically, we demonstrate across various benchmarks that LLEGO evolves superior-perfore performing trees compared existing tree induction methods, and exhibits significantly more efficient search performance compared to conventional GP approaches.
address these
INTRODUCTION
Decision trees are fundamental models, which are widely utilized across various domains, including finance, healthcare, and bioinformatics (Morgan & Sonquist, 1963; Che et al., 2011; Soleimanian et al., 2012). These hierarchical models recursively partition the feature space, creating a tree-like structure where internal nodes represent decision rules based on feature values, and leaf nodes correspond to class labels or predicted values. Decision trees are particularly appealing as they offer both predictive accuracy and interpretability, which have stood the test of time against recently developed black-box predictive models (Borisov et al., 2022; Grinsztajn et al., 2022).
However, decision tree induction represents challenging optimization problem. Finding the optimal tree given a training dataset is NP-complete (Laurent & Rivest, 1976), often necessitating the use of heuristic algorithms (Quinlan, 1986). While computationally efficient, these heuristics yield approximate, locally greedy solutions that sacrifice some degree of performance and global optimality (Rokach & Maimon, 2005). Exact optimization methods have been developed address these limitations, but they face their own constraints. Their computational complexity typically scales exponentially with problem size, limiting their applicability to restricted search spaces, and specific problem types (e.g., binary classification) (Verwer & Zhang, 2019; Lin et al., 2020).
Genetic programming (GP) a class of evolutionary algorithms which offers promising middle ground for decision tree induction, balancing computational efficiency with global optimization
"Equal contributions.

---

Published as a conference paper at ICLR 2025
Pop. of decision
Creswveer (L1e00x0) Section 3.2
trees pis)
Sample parents: S. ( e PI]
Cso dieciti
Compute desired fitness: 5 vin Sa.
Section 3.4
Eval
Sample ofrping by LiMool C.
Compute fireees:/ Yug Oxo Lu Outer
Training dataset: Dusis
N
Mutation (LT) Section 3.3
Seiertion
Trak acmuniic di
Sumple parents: S ((4,./(t))ss e [+]]
gidt (o planti DeL. pe.i
probiem doa:
Easple baggrig (6%, +1630 L.burrt (
Chart idquutt
Weighted offapring sompling: a(6,).r
Update population
Figure/ LLEGO Overview. In each generation g E [G], population of trees pl9) is evolved through crossover OMUT LLEGOxo (P(s), ,C; and mutation OMUT = LLEGOMUT(Plo) C; r). Subsequently, the offsprings Oxo U OUTT are evaluated for fitness on Drain: and selection preserves the top-N trees, p(gt1) & SEL(p(e+1) ,N), where p(s) p(9) U Oxo U OUT-
of the tree structure (Koza, 1994a;b). Inspired by principles of evolution, GP algorithms evolve population of candidate solutions through iterative application of genetic operators such as crossover and mutation. They are particularly well-suited for optimizing combinatorial problems with discrete, variable-length search spaces, as is the case in decision tree induction (Koza, 1990; Tanigawa & Zhao, 2000; Kuo al., 2007; Lahovnik, 2024). While much research in GP has focused on designing genetic operators to enhance search efficiency, these approaches face inherent limitations that constrain their exploratory effectiveness. Key challenges include the difficulty of incorporating semantic information into genetic operations
in primarily structural, unguided mechanisms and the narrow operational contexts that limit global exploration.
Key considerations. The key insight of this work is to employ large language models (LLMs) to design semantically-aware genetic operators for decision tree induction. LLMs are powerful generative models capable of learning distributions over discrete and variable-length sequences given only few-shot examples (Radford et al., 2019; Brown et al., 2020). We utilize LLMs as the foundation for crossover and mutation operators, leveraging their encoded semantic priors create meaningful distributions over potential offspring. Building on LLMs, our approach introduces fimess- guides crossover and space exploration that contrasts with the structural focus and unguided nature of conventional genetic operators. By representing decision trees in natural language, we also enable higher-arity genetic operations, capable of operating over multiple trees simultancously.
mutation operators within GP framework, enabling efficient search
Contributions. Conceptually, we propose a novel GP algorithm that leverages semantic priors contained in LLMs to enhance search efficiency and performance on challenging decision tree induction problems. Technically, we introduce LLEGO Enhanceed Genetic Operators), which uses L.MSs to define two key search operators: fitness-guided crossover that steers the search towards promising regions using target fitness; and diversiry-guided mutation that employs log-probabilities to evolve solutions in under-explored search regions. Empirically, on wide range of classification and regression tabular benchmarks, we demonstrate that LLEGO significantly improves search efficiency and consistently evolves trees with superior generalization performance.
PRELIMINARIES
2.1 DECISION TREE INDUCTION
Decision tree induction the problem of learning a decision tree T from a training dataset Durain = {(x,, y)) where x; X' C Rd denotes a d-dimensional input and ys E ) denotes the output. Decision trees recursively partition the input space X' into hierarchical, disjoint regions. In this work, we focus on binary decision trees, where splits partition regions in two subregions. These regions define set of leaf nodes R = (R,, R2, ..., Rr). where each leaf RG is assigned a constant (Hastie et al., 2009). This in turn yields a predictor t X - y which is defined by (xx) Σte a Txx Ri), where T(-) is the indicator function. Constructing decision trees from
2

---

Published as a conference paper at ICLR 2025
training dataset is challenging optimization problem, since full tree optimization has been proven to be an NP-complete problem (Laurent & Rivest, 1976). Greedy algorithms like CART (Breiman et al., 1984) build trees top-down, offering computational efficiency but sacrifices performance by only finding locally optimal trees. By comparison, exact optimization methods (Lin et al., 2020) provide theoretical guarantees of glohal optimality, but scale exponentially with problem size, and apply only to classification tasks and objective fuunctions of specific forms.
2.2 GENETIC PROGRAMMING
Genetic Programming (GP) is class of evolutionary algorithms for searching combinatorial spaces and offers a flexible middle ground between greedy and exact optimization methods. The fundamental objective of GP is to evolve trees 1 E T to maximize fitness function f T + R, where T is the combinatorial search space (Koza, 1994a). In GP, each individual is described by the tuple (t. f(t)) containing tree and its fitness. We denote this population of N individuals P ((x))), with N E N. The algorithm evolves the population across G E N generations. In each generation E [G', the population p(s) undergoes three key genetic operations: selection and two variation operators (crossover and mutation).
((1)) (f2, ((2)),
Selection. The selection mechanism preserves performant trees across generations, placing selection pressure on sufficient exploitation and ensures convergence (Goldberg, 1989). The N-ary selection operator is defined as SEL: : T x R. + A(T where A ((TN) represents the probability simplex over T Often, selection operators implicitly create this probability distribution over P, wherein trees with higher fitness are more likely to be preserved.
Crossover. The crossover operator combines the genetic material of multiple candidate trees to generate performant offspring (Langdon & Poli, 2013). Crossover is an ı--ar operator, denoted XO T + AT)). taking in // parents to generate an offspring o T, sampled as, o Pxo S). where S is usually pair of parents (u 2) sampled uniformly from the population. xo induces the offspring distribution Pxo. which can be interpreted as a uniform distribution over all trees producible by the crossover operator. For example, a popular version of XO is subtree crossover, where randomly selected subtrees from two parent trees are swapped (Koza, 1994a).
The mutation operator promotes global exploration, thus mitigating premature convergence to local optima (Goldberg, 1989). An и-ary mutation operator MUT : T + A(7) performs random modifications to parents to generate a single parent tree (v = 1) and Paur is uniform over the set of trees that can be generated through a mutation operator (e.g., random insertion replacement of nodes).
Mutation.
offspring o Paur(- S). Traditionally, mutation operates on
2.3 DESIDERATA
We can conceptualize variation operator, v, as implicitly defining sampling distribution (o I S) that depends on the parent trees S and its rules. More formally, Po S) = po.(o) S, () (S) dg. where (g ()) represents the prior distribution of applying specific genetic operation (e.g.. pairs of nodes when v corresponds to subtree crossover), and p. S,g) is the likelihood of an offspring given parents and genetic operation (generally, if producible, and otherwise). As such, genetic operators be viewed as sampling from posterior distribution over offspring, where the prior is encoded through the operator's 's design. While these variation mechanisms are core to GP, they present notable limitations that negatively impact search performance, leading to the following desiderata:
Semantic priors: Conventional variation operators encode inductive biases on structural properties, crucially lacking any considerations for tree semantics, i.e., (r[g) is independent of the semantics. This can be problematic, as small changes to the structure can lead to disruptive changes to the functional behavior (an issue known as rough eenotyp-pphenotype mapping Rothlauf et al. (2011)). This can be improved through integration of problem semantics into the prior (g |S, (,C), where C represents the semantics, leading to the sampling distribution p. S,C).
Guided variations: bution over any genetic operations. For example, they might consider any structural operations as equally likely, i... p(g S) is uniform. This lack of search guidance can lead to inefficient exploration and slower convergence, which can be improved with more informative priors that
Generally, conventional variation mechanisms place an uninformative distri-
3

---

Published as a conference paper at ICLR 2025
prioritize offsprings that are more semantically meaningful, or likely improve fitness and cover unexplored regions of the search space.
Broader context: The designs of existing operators often constrain the arity of allowed operations (e.g.. it is difficult define valid operators > 2 trees), restricting offsprings to local exploration. In contrast, including more parents in genetic operations can improve global exploration.
line of work has aimed to address some of these desiderata. Notably, previous works in semantic GP have attempted to address the first two limitations with variation operators, which consider the semantics of solutions (Krawiec Lichocki, 2009; Moraglio et al., 2012; Krawiec & Pawlak, 2013). In the semantic GP literature, solution's s semantics typically refers the functional output of solution, i.c., htt) = [t(xz), )) (x2). {(x)]] ]] E R", contrast, our work uses the term to describe domain knowledge about the solution space encoded in the LLM. Additionally, semantic GP is limited to application- speciifc definitions of semantics that restrict its broader applicability. Crucially, no comparable semantically-awwe method has been developed for decision tree induction.
3 LLEGO: GENETIC OPERATORS WITH SEMANTIC PRIORS
Designing genetic operators that satisfy the aforementioned desiderata using conventional methods has proven difficult. In this work, we build on the insight that LLMs are powerful generative models that can be employed properties that make them appealing (Meyerson et al., 2023; Lehman al., 2023). Firstly, we utilize LLMs source of semantic prior (LIM(g S, C)C), as they contain rich semantic knowledge of the problem and tree solutions, forming the basis of variation operators (Xie et al., 2021). Secondly, they are able to reason over and learn from in-context examples to identify high-potential patterns in candidate trees and produce guided variations towards desired regions. Lastly, their relatively large context window facilitates utilization of broader context, increasing arity of feasible genetic operations. Building on this semantic prior, we design genetic operators through mechanisms that incorporate fitness information and log probabilities of generated offspring. to further improve exploration and exploitation abilities.
semantically aware variation operators. Indeed, LLMs possess several
Method overview, At high level, LLEGO represents trees and frames genetic operations in natural language. Specifically, each genetic operation is realized through a distinct prompt which contains parent trees, semantics, and auxiliary information. We introduce fitness sunnessguided crossover LLEGOXo that performs in- no-context learning over solutions and their fitness, and generates offspring conditioned on a desired fitness f", steer evolution towards geggh-erforming regions. Additionally, we propose diversity-guide guidedd mutation LLEGOMUT. which uses the log probabilities of candidate offspring to con- struct a weighted sampling distribution, prioritizing efficient exploration that cover unexplored search regions. We note that the level of fitness- diversity- guidance is intentionally controllable through two hyperparameters. a and 7 that correspond to different degrees of exploitation vs. exploration. An overview of our method is visualized in Figure 1.
diversity-guidance
3.1 LLEGO PROMPT DESIGN
The genetic operations are performed through natural language queries to the LLM. While the specific structure of each query differs, they constructed from three essential components. For extended description of prompts and examples, please refer to Appendix B.
1. Task context. Denoted as C and encapsulates the semantic description of the problem, including se- mantic and statistical descriptions of the input space features X. output label y, and characteristics of the overall dataset, e.g., number of samples or features.
2. Parent solutions. This contains the solution representation and fitness of each parent, which are serialized into natural language and provided as few-shot examples in each genetic operation. Task-specific instructions. For each genetic operator, we include task-specific instructions on offspring generation and guidelines the format of the response.
3.2 Fiiness-GUIDER CROSSOVER OPERATOR
Traditional crossover operators are not semantically aware, as they randomly select subtrees from parents to be recombined into offspring. This ignores patterns in the parents, introducing the
4

---

Published as a conference paper at ICLR 2025
possibility for performant substructures to be destroyed through random perturbations. Additionally, they do not make use of parent fitness explicitly to guide offspring generation (i.e., no fitness guidance), foregoing any informative signals on correlations between fitness and solution structure. We seek to address these factors in our fitness- guided crossover operator. More specifically, the crossover operation includes three steps: (/) sampling subset of parents, weighted by parent fitness, (2) compute desired fitness f based on parent fitness, (3) constructing the prompt and querying the LLM generate offisprings, conditioning on f (see Figure 2).
Parent sampling. Each crossover operation is conditioned on parents, which are sampled from the current population. We utilize a roulette-w -hheel mechanism (Blickle & Thiele, 1996) to favour existing solutions with high fitness. Specifically, we aim to sample a set of E N parents for each crossover operation, where the sampling weights (0), Pe....... 0x) are proportional fitness. These weights define a categorical distribution Cat (0), based on which we sample parents without replacement. Intuitively, solutions with higher fitness are more likely to be involved in genetic
the solutions'
operations. We use Sy to represent the set of parents sampled for operation & E [s]. where E N is the number of crossover operations performed.
Crossover through fitness guidance. To perform crossover, we utilize both the tree structures t, and the fitness metric/ ft,,) to create few-shot prompts. For each of the sampled parents in S, we serialize the tree into natural language as a nested dictionary, which we denote as t where each intermediary key represents the splitting condition (feature name and splitting value) and the leaf item represents the value of the leaf node. Please refer to Figure 10 for more description of this representation. Each example is then constructed as "fitnes:: [(14), free: 1n!"" and we use Sul to represent the serialized few-shot prompt. We further condition the generation by specifying a desired fitness f* in the prompt to steer the generation towards high-fitness gions. This steering is controlled by a hyperparameter a, where f = fmax a(f fmin), with fmax and fiiu the best and worst fitness in S respectively. Intuitively, f is defined relative to the best parent fitness, with the improvement proportional to the observed variability. A positive a defines f" to improve over the best fitness in the parent set, whereas -1 < a 0 results in more conservative target fitness that is within the observed fitness range.
Croesover (LLBaOsu)
Mudaifa
by LUNOO
Pecplo Menn
Figure 2: LLEGOXO- In each operation, the crossover operator (I) samples a set of par- ents S weighted by their fitness, (2) com- putes the desired fitness f using S and o, and (3) samples offspring via the LLM.
We generate offsprings as 0j L- S J*), by sampling from LLM conditioned on the parents Sah the task context C, and target fitness f" controlled by a. We write the complete crossover operation Oxo, LLEGOxo (P(s), a)), where Oxo, k is the set of offspring generated from the operation & E [Kt]. a controls the level of extrapolation, and we systematically investigate its effect in Section 5.2. By framing crossover using natural language, our crossover operator naturally allows for an arity v strictly than 2, by including additional parents as in-context examples through S
3.3 Diversity-GUIDED MUTATION OPERATOR
On the other side of the coin is the mutation operator, where the objective is to efficiently traverse under-explored areas in the search space to improve diversity and escape local minima. Traditional mutation operators can be viewed as inducing a uniform distribution over the space of solutions one random mutation away from the parent. However, this does not consider whether such mutations semantically meaningful. To contextualize this, imagine the space one mutation away from a decision tree; many of these mutations highly unlikely to be interesting or optimal given some degree of domain knowledge, resulting in inefficient exploration. In this setting, our mutation operator uses its semantic prior to effectively guide exploration, enabling more efficient diversity-driven exploration.
Parent sampling. As before, each mutation operation is conditioned on a set S of v parents. However, whereas for crossover, parents are sampled based on their fitness, for mutation, parents are randomly sampled from the population to increase the diversity of S. Specifically, S = {(tz, f(t))) i [y],t, Uni fo (P where each solution has uniform probability of being selected as parent, Future works should consider more advanced sampling schemes to improve parent diversity.
5

---

Published as a conference paper at ICLR 2025
Mutation with diversity guidance. To perform mu- tation, we only include the tree structure ły to create few-shot prompts: each parent is serialized as "tree: inl",, to create St Subsequently, we generate X' (where X' 2 λ) candidate offsprings oj. and track the nega- tive log probabilities of the candidates obtained from the LLM. represented s(ā,) = -p(o S). Intuitively, s(ō,) reflects the likelihood of the candidate offspring given the set of parents, with smaller values indicating that the candidate offspring has low probability under the current population distribution and hence that its inclusion can introduce more diversity at the population level. We provide further justification for the use of log probabilities in Section 5.2 and Appendix D.4, demon- strating their correlation with functional and structural distances between parent and offspring trees.
Musation (LLEDO
Hidtdie (Bya) (1)) Lot (S,C)
ne 1
Figure 3: LLEGOMUT- In each operation, the mutation operator: (1) samples set
of X' candidate offsprings O, (2) computes sampling weights, 0, inversely proportional to logprobs, with temperature 7 controlling diversity, and (3) sample offspring via this weighted distribution CO)
As such, the candidate sampling step is represented (ōj. x(ā,)) LLMMUT St ,C). Given this set of X' candidates, we select Ả offspring based on their log probabilities, i.e., OUTT = [(o,, ((;) Ij E
05 (o)) where 0- (0), ,00,,..,000 0x) and 03 Σx' exp(st,,)/=)
exp(=(0,)/7) Here, is the sampling
temperature, where larger values of r > 1 results in more uniform distribution over the candidate offspring, and lower values of 0 < < would put more weight on candidates with lower likelihood. As such, we use t and the log probabilities
guide the sampling of offspring with controllable levels of diversity. In Section 5.2, we empirically investigate the effect of 7 on offisprnng diversity. In summary, we define the k-th mutation operation as OUTT = LLEGOMur(P(o) C,;)).
3.4 END-TO-END ALGORITIMM
Having detailed our LLM-based genetic operators, we now put together the end-to-end GP algorithm. The search is initialized with a set of N solutions, p(u), In each generation, we sample N crossover offspring, represented as O and mutation offsprings, represented as O This is performed through genetic operations, with each operation involving parents, and generating À offsprings. The fitness of each solution is then calculated against the training set D For selection, we consider the set of candidates the union of the solutions from the previous generation and the newly
XO generated offsprings, i.e. p(++1) = p(a) U ( U MLT- We use the elitism selection to select the
top-N unique solutions from the candidate population to preserve the highest quality solutions across generations (Goldberg, 1989). Here, top-N is selected based on training set fitness. More formally, we denote this process as p(++1) <- (F N). After G generations of evolution, we select the solution with the highest validation fitness as the final solution.
RELATED WORKS
Our work relates to multiple strands of research, which we summarize in brief below. We provide an extended literature survey in Appendix A.
Tree induction algorithms. Existing algorithms for decision tree induction can be broadly categorized into three main classes: greedy, globally optimal, and GP algorithms. Greedy algorithms recursively construct a tree in a top-down approach, heuristically making locally optimal splits at each node (Breiman et al., 1984; Quinlan, 1986; 1993). While computationally efficient, these methods do not pursue global optimality. Recent works have proposed exact combinatorial methods to construct optimal decision trees (Verwer& & Zhang. 2019; Hu et al., 2019; Lin et al., 2020; Aglin et al., 2020). These methods face two key limitations: exponential complexity scaling with tree depth and number of splits, and restricted applicability to specific objectives (primarily classification problems).
Genetic programming. GP approaches present a middle ground between search performance and computational efficiency (Koza, 1990; Tanigawa & Zhao, 2000; Lahovnik, 2024). GP builds on genetic operators that operate over structure (genotype) but can have disruptive effects because of the complex genotype-phenot mapping (Rothlauf et al., 201 1). This observation has motivated works

---

Published as a conference paper at ICLR 2025
Table 1:
Performance on classification tasks. Balanced accuracy (1) on 7 datasets, reporting (sa) and emboldening best results. We also report average rank ()) for comparing baselines.
Method
Breast
Compas
Credit
Diabetes
Heart
Liver
Vehicle
Rank
depih
CART
0.941 (0.003
0.655qu.aaa
w.808
0.710
(.734
0.903)
2.9.0.a3
04.5
0. 0.650 019))))
0.579hu.mon
0.687
0.704
0.509
0. 0.857
4.9
DL.5
0.947
(u ous)
0.685 oun)
0.703huuzz)
0.658
0.508 (u00s))
0.
3.0
308DT
(. o0s)
0. 0.641 (664,o.004 o01;
0.681 000)
0.681 (o.08))
0.65 (ocos))
0.656 (u.58)
4.4
GATRES
0,912
0.8 0.647ro
0 0.6418o.045)
0,069 (o009
0.626
0. 0.2.0.02
1.311.11
LLEG3
0111
0.652
0.009
0.713
0. 0.004)
0. 0.672
0. 0.937 (0.010
1.6 70)
depti =
CART
0.600 a01;
0.675)u.01))
0.701)
0.713
(u.ws))))))
2.9r0.00
24.5
0.913
0.660) 2oo;
0.622
0.634)
0.714
0.583 0.04))
0.921
3.6
DL85
(0.01
0.661.o 004;
0.576
0.67 (..71
(.
(o.o1)
0. (.850.0000
1.100000)
30SDT
(0.077
0.641
0. 0.680)o.ac2)
0,701
0.677 (o.cas)
(.600))
D. 1.886 8..5
1.39)
GATRES
(o.us))
0.650
(..75)
(.675)
4.6 4.6g0.05)
T.RGO
0.95 (a 007)
0.663 to ous)
0.684 (0.013)
0.72 (0.03T:
0.75 (0.022)
0.676
0.9 (a.01))
0.00000)
on semantic GP (Krawiec & Lichocki, 2009; Moraglio et al., 2012; Krawiec & Pawlak, 2013), aiming to produce offspring that inherit semantics (phenotype) from parents. However, these approaches are highly domain-specific, and have not extended to tree induction, which is the focus of our work.
LLMs and optimization. Recent studies have explored LLMs for optimization tasks, with some works employing LLMs as variation operators (Meyerson et al., 2023). Examples of applications include code evolution (Lehman et al., 2023; Nasir et al., 2024; Brownlee et al., 2023), Bayesian Optimization (Liu et al., 2024), and prompt optimization (Fernando et al., 2024; Guo et al., 2024), where unguided variations are sampled from LLMs. In contrast, LLEGO generates guided variations by considering search dynamics at the population level, modulating fitness and diversity through hyperparameters a and T, while exploiting in-context learning with parent solutions.
5 EXPERIMENTS
Benchmark datasets. We empirically evaluate LLEGO's ability to find performant decision trees for 12 open-source tabular datasets from OpenML. curated benchmarks (Vanschoren al., 2014), including 7 classification and 5 regression datasets. These datasets were selected based on the number of features, samples and the presence of semantically meaningful feature names and descriptions. We provide further details on this selection of datasets and preprocessing in Appendix C.I.
Baselines. We compare LLEGO against a comprehensive set of competitive decision tree induction methods across major categories: greedy induction (CART (Breiman, 2017) and C4.5 (Quinlan, 1993)). sparse optimal tree induction (GOSDT (Lin et al., 2020) and DL8.5 (Aglin et al., 2020)), and a GP approach using conventional genetic operators (GATree (Lahovnik, 2024), which is an implementation of GP for decision tree induction in Python). More details on these baselines, their implementation, hyperparameters, and experimental details are given in Appendices C.2 and C.3. For GP-based methods (LLEGO, GATree), we initialize the population with CART models bootstrapped on 25% of the training data. We report results using G = 25, N 25, and we use 0.1, 10 and v = 4 as the hyperparameters for the variation operators LLEGO.
Evaluation. For classification tasks,
balanced accuracy (ACC), and for regression tasks, mean squared eгror (MSE), computed on a held-out test dataset Diest- Each metric is averaged over runs with different random seeds, due different dataset splits, and we present these averages with their standard deviations. For LLEGO, we use gpt-3. 2....tt.. 5-turbo version 0301 as the underlying LLM. For fair comparison, each method is allowed 10 minutes of wall clock time per seeded run, which includes time spent on hyperparameter tuning.
5.1
TREES ACHIEVE SUPERIOR GENERALIZATION PERFORMANCE
We first compare the performance of the complete LLEGO algorithm against baselines for decision tree induction. We report in Table I and Table 2 the test performance on classification and regression datasets, respectively, for maximum tree depths of 3 and 4. For regression, we report the results for CART and GATree since other baselines cannot optimize regression objectives. The results demonstrate that LLEGO outperforms baselines comprehensively. We observe that this performance advantage becomes more pronounced in the space of trees with depth 4, which is intuitive since
7

---

Published as a conference paper at ICLR 2025
Table 2: Performance on regression tasks. MSE (L)
Molian Pup Fitzess
Median Pop Divenilt
across 5 regression datasets, best results emboldened.
Method
Abalone
Cars
Cholesterol
Wage
Wine
depoh
44
0.50
GATRES
CARC
0.500 940)
0.198 o now)
0.2220/..025)
1.427 ter
2.50
0.811 o.046
1.150ho
0.825
000)
0,00
D. 0.584
1.324
CARC
0.561a
0 0.1
depoh
1,522 nas:
1.185 mo)
0.807
Gimerationn
GATRES
0. (.
1.348
1,188 1601
0.847
Figure 4: Search efficiency. Median fit-
31.303
0.099
1 1.322
1 1.07
(.828
ness and diversity across 25 generations.
it represents a substantially larger search space compared to the set of trees with depth 3. In the more constrained space of trees with depth 3, sparse optimal induction methods such as DL85 and GOSDT demonstrate increased competitiveness. This suggests that LLEGO'S ' efficiency gains are particularly evident when navigating more complex and expansive search spaces. We also compare in Appendix D.1 the generalization gap across all methods. Notably, LLEGO consistently achieves the lowest generalization gap, with this advantage becoming especially pronounced depth 4.
The results also underscore the impact of incorporating semantic priors. Indeed, LLEGO consistently outperforms the GP baseline GATree, which cannot take into account contextual information. Further analysis in Appendix D.5 demonstrates that LLEGO produces superior trees even when compared to GATree configuration utilizing substantially larger search budgets. Notably, LLEGO achieves this superior performance while requiring fewer evaluations, highlighting its efficiency and effectiveness.
Takeaway: LLEGO optimizes decision trees that are superior against diverse benchmark of methods, while being more applicable to a wider range of optimization objectives (c.g., regression).
Search efficiency. Having shown the superior generalization performance of decision trees evolved by LLEGO, we now compare search efficiency between LLEGO and the GP baseline GATree. We evaluate population dynamics via normalized population fitness and diversity between the two methods across all classification datasets, when optimizing trees with depth 3. Fitness values (i.e., balanced accuracy) were normalized to enable comparison across different seeds and datasets (refer to Appendix C.4 for details). Figure 4 (Left) shows the median population fitness, where LLEGO demonstrates superior search efficiency, finding fitter individuals more efficiently.
Figure 4 (Right) shows that the populations evolved by LLEGO exhibit decreasing diversity as the search progresses, whereas GATree maintains roughly the same level of diversity in its population. This is expected, LLEGO uses its semantic priors to focus the search on semantically meaningful regions, which naturally reduces diversity. A similar effect has been observed when employing semantically aware GP in other domains (Krawiec & Pawlak, 2013). In comparison, GATree, which is semantically unaware, performs random structural perturbations that maintain a certain level of diversity in the population. In Appendix D.8, we investigate search efficiency on problems with depth 4 and show search dynamics on individual tasks, observing the same effects at play.
Takeaway: LLEGO leverages its semantic priors for more efficient search convergence, although this can sacrifice population diversity, requiring this trade-off to be carefully balanced by its operators.
5.2 UNDERSTANDING THE SOURCES OF GAIN
Having demonstrated enhanced search efficiency of LLEGO in the previous section, we now examine the contributions of its operators to this improvement. In what follows, we analyze how offspring characteristics are influenced by different values of the hyperparameters a and T, which give control over the desired solution fitness and population diversity.
0.7
Pop Fitness
0.6
11,500
GATURR
0.5
0.4
Pop Diversity
Results. (1) Crossover: We examine the effect of (-0.25, 10.1, 0.1, 0.25) on offspring generation, where a de- termines the target fitness f" that conditions the offssring gen- eration. In Figure 5, we visualize the median population fitness and diversity function of a. Offspring fitness improves as a increases from -0.25 to 0.1, but regresses beyond this point as the target fitness f leads extrapolation in less reliable
11,000
0.2
-0.25
-0.10
0.10
0.25
Figure 5: XO dynamics.
8

---

Published as a conference paper at ICLR 2025
regions. Interestingly. the best offspring fitness emerges at 0.1, suggesting LLEGOxo ability to perform a reasonable degree of extrapolation. Corresponding, diversity decreases with increasing a, reflecting sampling from progressively smaller search regions. Hence, compared to GATree, LLEGO produces higher quality offspring but with lower diversity, which is consistent with our findings in Section 5.1.
(2) Mutation: We investigate the role of LLEGOMUT in main-
0.46
taining diversity by considering range of (5, 10, 25, 50). In Figure 6, we observe that lower values of r increases pop- ulation diversity, as they prioritize offspring that have low like- lihood given parents. As such, the offspring introduce greater diversity at the population level, which complements the dy- namics of the crossover operator mentioned above, crucial in balancing exploitation and exploration during search. Results for individual datasets can be found in Appendix D.S.
Pop Diversity
0.44
GÀTREE
0.42
0.40
10
25
50
Figure 6: MUT dynamics.
5.3 ABLATION STUDY: ALL COMPONENTS CONTRIBUTE TO ENHANCED SEARCH EFFICIENCY
Having demonstrated the superior performance of LLEGO against existing baselines, we finally scrutinize the contribution of each algorithmic component to its optimization performance. Specifically, we aim to investigate the effects of (1) leveraging the LLM's semantic prior evolve solutions, (2) the fitness- guided crossover and diversity- guided mutation, and (3) the higher arity of genetic operations. Now, systematically ab- late each component: LLEGOm semantics removes any semantic information from the prompts (see Appendix B.1 for detailed description), which is equivalent to removing C from p(o S, C); LLEGOno removes the fitness-guided crossover, using only the mutation operator during search; LLEGO,o mut removes diversity-guided mutation, using only crossover during search; and LLEGO,=2 restricts the context to 2 parents, akin to traditional genetic operators. We evaluate search efficiency in Figure 7, observing that best performance is obtained when both operators are used in tandem, likely as they balance exploration of higher fitness regions (guided by (*) and exploration of less visited regions (guided by r). The semantic information leveraged by the operator also improves performance, although note that even without it, LLEGOno semantics performs very competitively. highlighting the strong few-shot learning capabilities of LLMs. Finally, using binary operators in LLEGOy=2 is suboptimal, underlining the often overlooked importance of using wider context in genetic operations. We provide more fine-grained ablation results Appendix D.7.
Median Pop Pitness
0.80
0.60
LLBS0ma
LLB30
0.50
LLS00 amantion
20
Generations (G)
Figure 7: Ablation study, Compar- ing search efficiency of ablations.
Takeaway: Our ablation experiment demonstrates that all algorithmic components contribute the enhanced optimization performance of LLEGO.
5.4 ADDITIONAL RESULTS.
In the interest of space, we relegated additional investigations to Appendix D. Specifically, we investigated memorization concerns by evaluating generalization performance on datasets with removed identifying information and context, as well as testing LLEGO on unseen proprietary datasets (detailed in Appendix D.2). In Appendix D.3, investigated LLEGO's ability to mitigate negative bias by optimizing fairnes--reullaiized objectives. We also provide additional analyses into LLEGO'S performance and its individual components.
DISCUSSION
In summary, we introduced LLEGO, a novel GP method for decision tree induction that integrates semantic priors over the search space by using LLMs variation operators. Our approach leverages the semantic understanding and domain knowledge of LLMs to evolve decision trees through crossover and mutation operators, while incorporating fitness and diversity guidance and flexible

---

Published as a conference paper at ICLR 2025
operation arity. Empirical results across diverse datasets demonstrate LLEGO's superior optimization efficiency, yiclding high-pes perffrerfrn decision trees compared to existing baselines.
Limitations and future works. However, work is not without its limitations. Performing inference through LLMs incurs a larger computational footprint than conventional GP algorithms. Our findings indicate that LLEGO trades off computational requirements for improved search efficiency and generalization performance, making it particularly appealing in performance-sensinn or problems where evaluation costs exceed search costs. Future works should prioritize reducing computational requirements while retaining performance, such as through inference acceleration (Leviathan al., 2023) and memory- model architectures (Han al., 2015). Additionally, while LLEGO can operate effectively without semantic priors, its performance can be further improved when such knowledge is available. Future works could explore finetuning strategies and prompt augmentation strategies to incorporate semantic knowledge in specialized domains. We also recognize that using black-box L.MSs could potentially lead to the propagation of negative biases into the solutions returned by LLEGO-to to this end, we presented initial steps to mitigate bias via the design of objective functions. Outlook. the long run, we believe this work illuminates the promise of employing LLM capabilities for enhancing efficiency and performance in complex combinatorial optimization problems beyond decision tree induction.
domains
10

---

Published as a conference paper at ICLR 2025
ETHICS AND REPRODUCIBILITY STATEMENTS
Ethics. In this work, we evaluate both public benchmarks and private datasets. The private datasets are de-identified and used following the guidance of the respective data providers. We follow recommendations to use the Azure OpenAl service when using GPT models, where via the agreement we ensure the medical data is not sent for human review stored, hence respecting the guidelines given by the dataset providers.
Reproducibility. We provide all the details the datasets, the implementation of baselines and the LLM Appendix C. Furthermore, we detail the prompts used by the crossover and the mutation operators in Appendix B. We provide the code to reproduce our results https: /ttptp::::p//// thub. htnnnnicolssuuuhh LLEGO, and https: / //tpthut.bb//////g///////t/.hhtt wow/tenniseeeoocccccccc.c www.eennnnnnnnnnnnno.oo
ACKNOWLEDGMENTS
We thank the anonymous ICLR reviewers, members of the van der Schaar lab, and Andrew Rashbass for many insightful comments and suggestions. TL would like to thank AstraZeneca for their spon- sorship and support. NH thanks Illumina for their support. This work was supported by Microsoft's Accelerate Foundation Models Academic Research initiative.
REFERENCES
Statlog (Heart). UCI Machine Learning Repository. DOI:
Gaël Aglin, Siegfried Nijssen, and Pierre Schaus. Learning optimal decision trees using caching branch-and-bowk search. In Proceedings of the AAAI conference artificial intelligence, volume 34, pp. 1146-315,, 2020.
Takuya Akiba, Shotaro Sano, Toshihiko Yanase, Takeru Ohta, and Masanori Koyama. Optuna: A next-generation hyperparameter optimization framework. In Proceedings of the 25th ACM SIGKDD international conference on knowledge discovery data mining, pp. 2623-2631, 2019.
Julia Angwin, Jeff Larson, Lauren Kirchner, and Surya Mattu. Machine bias. ProPublica: May
https//wwwppppbiiiccaaggtttttttt sitaiia---rkksssssseee----e--nie 2016.
Douglas Adriano Augusto and Helio JC Barbosa. Symbolic regression via genetic programming. In Proceedings. Vol. I. Sixth Brazilian symposium neural networks, pp. 173-178. IEEE, 2000.
Rachel KE Bellamy, Kuntal Dey, Michael Hind, Samuel C Hoffman, Stephanie Houde, Kalapriya Kannan, Pranay Lohia, Jacquelyn Martino, Sameep Mehta, Aleksandra Mojsilović, al. Ai fairness 360: An extensible toolkit for detecting and mitigating algorithmic bias. IBM Journal of Research and Development, 63(4/5):4 2019.
ER Berndt. Determinants of wages from the 1985 current population survey. The practice of econometricx: classic and contemporary, pp. 193-209, 1991.
Philip Bille. A survey tree edit distance and related problems. Theoretical computer science, 337 (1-3):217-239, 2005.
Bernd Bischl, Giuseppe Casaliechio, Matthias Feurer, Frank Hutter, Michel Lang, Rafael G. Manto- vani, Jan N. van Rijn, and Joaquin Vanschoren. Openml benchmarking suites. arXiv:/708.037331
[stat.ML), 2019.
Tobias Blickle and Lothar Thiele. A comparison of selection schemes used in evolutionary algorithms. Evolutionary Computation, 4(4):361-394 1996.
Marko Bohanec. Car Evaluation. https//bbi.oggl102444///sPs
UCI Machine Learning Repository, 1997.
DOI:
Also available the wider lab repository httppp///.thhh...ccm//aaeeecrrrr
11

---

Published as a conference paper at ICLR 2025
Vadim Borisov, Tobias Leemann, Kathrin Seßler, Johannes Haug, Martin Pawelezyk, and Gjergji Kasneci. Deep neural networks and tabular data: A survey, IEEE Transactions on Neural Networks and Learning Systems, 2022.
Leo Breiman. Classification and regression trees. Routledge, 2017.
Leo Breiman, Jerome Friedman, Charles J Stone, and RA Olshen. Classification and Regression Trees. CRC Press, 1984.
Cliford Broni-Bediako, Yuki Murata, Luiz HB Mormille, and Masayasu Atsumi. Evolutionary nas with gene expression programming of cellular encoding. In 2020 IEEE symposium series on computational intelligence (SSCI), pp. 2670-2676. IEEE, 2020.
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. Advances in neural information processing systems, 33:1877-1901, 2020.
Alexander EI Brownlee, James Callan, Karine Even-Mendoza, Alina Geiger, Carol Hanna, Justyna Petke, Federica Sarro, and Dominik Sobania. Enhancing genetic improvement mutations using large language models. In International Symposium on Search Based Software Engineering, pp- 153-159. Springer, 2023.
Dongsheng Che, Qi Liu, Khaled Rasheed, and Xiuping Tao. Decision tree and ensemble learning algorithms with their applications in bioinformatics. Software tools and algorithms for biological systems. pp. 191-199, 2011.
Paulo Cortez, A. Cerdeira, F. Almeida, T. Matos, and J. Reis. Wine Quality. UCI Machine Learning Repository, 2009. DOI: http:///bbl.rr//11244433/6666663
CUTRACT. Cutract. https:/prostuutccnneerk..orr 2019.
Chrisantha Fernando, Dylan Sunil Banarse, Henryk Michalewski, Simon Osindero, and Tim Rock- täschel. Promptbreeder: Self-referential self-improvement via prompt evolution. In Forty-first International Conference Machine Learning, 2024.
Sebastian Felix Fischer, Matthias Feurer, and Bernd Bischl. Openml-ctr23-a curated tabular regres- sion benchmarking suite. In AutoML Conference 2023 (Workshop), 2023.
DE Goldberg. Genetic algorithms in search, optimization, and machine learning, 1989.
Léo Grinsztajn, Edouard Oyallon, and Gaël Varoquaux. Why do tree-based models still outperform deep learning on typical tabular data? Advances in neural information processing systems, 35: 507-520, 2022.
Alexandre Guillaume, Seugnwon Lee, Yeou-Fang Wang, Hua Zheng, Robert Hovden, Savio Chau, Yu- Wen Tung, and Richard Terrile. Deep space network scheduling using evolutionary computational methods. In 2007 IEEE Aerospace Conference, pp. 1-6. IEEE, 2007.
Qingyan Guo, Rui Wang. Junliang Guo, Bei Li, Kaitao Song. Xu Tan, Guoqing Liu, Jiang Bian, and Yujiu Yang. Connecting large language models with evolutionary algorithms yields powerful prompt optimizers. In The Twelfth International Conference on Learning Representations, 2024.
Song Han, Huizi Mao, and William J Dally. Deep compression: Compressing deep neural networks with pruning, trained quantization and huffmmnn coding. arXiv preprint arXiv:1510.001 2015.
Trevor Hastie, Robert Tibshirani, Jerome H Friedman, and Jerome H Friedman. The elements of statistical learning: data mining, inference, and prediction, volume Springer, 2009.
Daniel Hein, Steffen Udluft, and Thomas A Runkler. Interpretable policies for reinforcement learning by genetic programming. Engineering Applications of Artificial Intelligence, 76:158-169, 2018.
Xiyang Hu, Cynthia Rudin, and Margo Seltzer. Optimal sparse decision trees. Advances in Neural Information Processing Systems, 32, 2019.
12

---

Published as a conference paper at ICLR 2025
Northpointe
Compas risk scales, wwtp:pppppbbllc.......oo /ot..//oogaa.ttttteaa/aaatttttt.
Inc.
2016.
URL
https: //www
Accessed: 2023-10-
01.
Andras Janosi, William Steinbrunn, Matthias Pfisterer, and Detrano Robert. Heart Disease. UCI Machine Learning Repository, 1988. DOI:
Markelle Kelly, Rachel Longjohn, and Kolby Nottingham. The uci machine learning repository. https://archivv..cs.....c
John R Koza. Concept formation and decision tree induction using the genetic programming paradigm.
In International Conference Parallel Problem Solving from Nature, pp. 124-128. Springer, 1990.
John R Koza. Genetic programming means for programming computers by natural selection. Statistics and computing. 4:87-112, 1994a.
John R Koza. Genetic programming II: automatic discovery of reusable programs. MIT press, 1994b.
Krzysztof Krawiec and Pawel Lichocki. Approximating geometric crossover in semantic space. In Proceedings of the I Itth Annual conference on Genetic and evolutionary computation, pp. 987-994, 2009.
Krzysztof Krawiec and Tomasz Pawlak. Approximating geometric crossover by semantic backpropa- gation. In Proceedings of the 15th annual conference Genetic and evolutionary computation, pp. 941-948, 2013.
Chan-Sheng Kuo, Tzung-Pet Pei Hong, and Chuen-Lung Chen. Applying genetic programming technique in classification trees. Soft Computing, 11:1165-1172, 2007.
Tadej Lahovnik. Gatree gatree 0. documentation. https: /atp:://ggteeeeeeeeeeee readthedocs. http://gggteeeeeehheeedsosss. o en/latest/ 2024. (Accessed on 05/19/2024).
William B Langdon and Riccardo Poli. Foundations of genetic programming. Springer Science & Business Media, 2013.
Hyafil Laurent and Ronald L Rivest. Constructing optimal binary decision trees is np-complete. Information processing letters, 111):15-17, 1976.
Joel Lehman, Jonathan Gordon, Shawn Jain, Kamal Ndousse, Cathy Yeh, and Kenneth O Stanley. Evolution through large models. In Handbook of Evolutionary Machine Learning, pp. 331-366. Springer, 2023.
Yaniv Leviathan, Matan Kalman, and Yossi Matias. Fast inference from transformers via speculative decoding. In International Conference Machine Learning, pp. 19274-19286 PMLR, 2023.
Jimmy Lin, Chudi Zhong, Diane Hu, Cynthia Rudin, and Margo Seltzer. Generalized and scalable optimal sparse decision trees. In International Conference on Machine Learning, pp. 6150-6160. PMLR, 2020.
Tennison Liu, Nicolás Astorga, Nabeel Seedat, and Mihaela van der Schaar. Large language models to enhance bayesian optimization. The Twelfth International Conference on Learning Representations, 2024. URL https: //pp:::///// wpwpreveeeeeeeeeeoooeeee. net - hww.:/nutttttmeeeeeeeee
net
Vadim Liventsev, Anastasiia Grishina, Aki Härmä, and Leon Moonen. Fully autonomous program- ming with large language models. In Proceedings of the Genetic and Evolutionary Computation Conference, pp. 1146-1155, 2023.
Yecheng Jason Ma. William Liang, Guanzhi Wang. De-An Huang, Osbert Bastani, Dinesh Jayaraman, Yuke Zhu, Linxi Fan, and Anima Anandkumar. Eureka: Human-level reward design via coding large language models. In The Twelfth International Conference on Learning Representations, 2024.
13

---

Published as a conference paper at ICLR 2025
Sascha Marton, Stefan Lüdtke, Christian Bartelt, and Heiner Stuckenschmidt. Gradtree: Learning axis-aligned decision trees with gradient descent. arXiv preprint arXiv:2305.03515, 2023.
Elliot Meyerson, Mark Nelson, Herbie Bradley, Adam Gaier, Arash Moradi, Amy K Hoover, and Joel Lehman. Language model crossover: Variation through few-shot prompting. arXiv preprint wrXiv:2302.12170, 2023.
Brad Miller, David E Goldberg, et al. Genetic algorithms, tournamett selection, and the effects of noise. Complex systems, 93):193-212, 1995.
Alberto Moraglio, Krzysztof Krawiec, and Colin G Johnson. Geometric semantic genetic pro- gramming. In Parallel Problem Solving from Vature-PPSN XII: 12th International Conference, Taormina, Italy, September 1-5, 2012, Proceedings, Part 12, pp. 21-31. Springer, 2012.
James N Morgan and John A Sonquist. Problems in the analysis of survey data, and a proposal. Journal of the American statistical association, 58(302):415-434, 1963.
Warwick Nash, Tracy Sellers, Simon Talbot, Andrew Cawthorn, and Wes Ford. Abalone. UCI Machine Learning Repository, 1995. DOI: https://bbl.org/0.2444////
Muhammad Umair Nasir, Sam Earle, Julian Togelius, Steven James, and Christopher Cleghorn. Limatic: Neural architecture search via large language models and quality diversity optimization. In Proceedings of the Genetic and Evolutionary Computation Conference, pp. 1110-1118, 2024.
Mowforth Pete and Barry Shepherd. Statlog (Vehicle Silhouertes). UCI Machine Learning Repository. DOL:
Justin K Pugh, Lisa B Soros, and Kenneth O Stanley. Quality diversity: A new frontier for evolutionary computation. Frontiers in Robotics and AI, 3:202845, 2016.
Zhaozhi Qian, Krzysztof Kacprzyk, and Mihaela van der Schaar. D-code: Discovering closed-form odes from observed trajectories. In International Conference Learning Representations, 2022.
J. Ross Quinlan. Induction of decision trees. Machine learning, 1:81-106, 1986.
J Ross Quinlan. C4. S: Programs for Machine Learning. Morgan Kaufmann, 1993.
Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, llya Sutskever, et al. Language models are unsupervised multitask learners. OpenAl blog, 1(8):9, 2019.
John Roemer and Alain Trannoy. Equality of opportunity. In Handbook income distribution, volume 2, pp. 217-300. Elsevier, 2015.
Lior Rokach and Oded Maimon. Top-down induction of decision trees classifiers-a survey, IEEE Transactions on Systems, Man, and Cybernetics, Part C (Applicatiions and Reviews), 35(4):476-4877 2005.
Franz Rothlauf et al. Design of modern heuristics: principles and application, volume Springer, 2011.
Jack W Smith, James E Everhart, WC Dickson, William C Knowler, and Robert Scott Johannes. Using the adap learning algorithm to forecast the onset of diabetes mellitus. In Proceedings of the annual symposium on computer application in medical care, pp. 261. American Medical Informatics Association, 1988.
Farhad Soleimanian, Peyman Mohammadi, and Parvin Hakimi. Application of decision tree algorithm for data mining in healthcare operations: case study. Int J Comput Appl, 52(6):21-26, 2012.
Xingyou Song, Yingtao Tian, Robert Tjarko Lange, Chansoo Lee, Yujin Tang, and Yutian Chen. Position: Leverage Foundational models for black-bor optimization. In Forty-first International Conference Machine Learning, 2024.
W Nick Street, William H Wolberg, and Olvi L Mangasarian. Nuclear feature extraction for breast tumor diagnosis. In Biomedical image processing and biomedical visualization, volume 1905, pp. 861-870. SPIE, 1993.
14

---

Published as a conference paper at ICLR 2025
Colin Sullivan, Mo Tiwari, and Sebastian Thrun. Maptree: Beating "optimal" decision trees with bayesian decision trees. In Proceedings of the AAAI Conference on Artificial Intelligence, vol- ume 38, pp. 9019-9026, 2024.
Toru Tanigawa and Qiangfu Zhao. A study on efficient generation of decision trees using genetic programming. In Proceedings of the 2nd Annual Conference on Genetic and Evolutionary Computation, pp. 1047-1052, 2000.
Joaquin Vanschoren, Jan N Van Rijn, Bernd Bischl, and Luis Torgo. Openml: networked science in machine learning, ACM SIGKDD Explorations Newsletter, 15(2):49-66) 60, 2014.
Vassilis Vassiliades, Konstantinos Chatzilygeroudis, and Jean- Baptiste Mouret. Using centroidal voronoi tessellations to scale up the multidimensional archive of phenotypic elites algorithm. IEEE Transactions Evolutionary Computation, 22(4):623-635 630, 2017.
Sahil Verma and Julia Rubin. Fairness definitions explained. In Proceedings of the international workshop on software fairness, 1-7, 2018.
Sicco Verwer and Yingqian Zhang. Learning optimal classification trees using binary linear program formulation. In Proceedings of the AAAI conference on artificial intelligence, volume 33, pp. 1625-1632, 2019.
Shuhei Watanabe. Tree-struche rree-structure parzen estimator: Understanding its algorithm components and their roles for better empirical performance. arXiv preprint arXiv:2304. //27, 2023
David H Wolpert and William G Macready. No free lunch theorems for optimization. IEEE transactions on evolutionary computation, 1(1):67-82 1997.
Chih M Wong, Nathaniel M Hawkins, Mark C Petrie, Pardeep Jhund, Roy S Gardner, Cono A Ariti, Katrina K Poppe. Nikki Earle, Gillian A Whalley, Iain B Squire, et al. Heart failure in younger patients: the meta-analysis global group in chronic heart failure (maggic). European heart journal, (S(39):2714-7222 2014.
Sang Michael Xie, Aditi Raghunathan, Percy Liang, and Tengyu Ma. An explanation of in-context learning as implicit bayesian inference. In International Conference on Learning Representations, 2021.
Chengrun Yang. Xuezhi Wang, Yifeng Lu, Hanxiao Liu, Quoe Le, Denny Zhou, and Xinyun Chen. Large language models optimizers, 2024. URL https: www.xiiooog/aaio 2309. 03409.
Haoran Ye, Jiarui Wang, Zhiguang Cao, and Guojie Song. Reevo: Large language models as hyper-heuristies with reflective evolution. ArXiv, abs/2402.01 145, 2024. URL https://appt bemanticscholn
Huimin Zhao. A multi-objective genetic programming approach to developing pareto optimal decision trees. Decision Support Systems, 43(3):809-826, 2007.
Arman Zharmagambetov, Suryabhan Singh Hada, Magzhan Gabidolla, and Miguel Carreira- Perpinán. Non-greedy algorithms for decision tree optimization: An experimental comparison. In 2021 International Joint Conference on Neural Networks (IJCNN), pp. 1-8. IEEE, 2021.
15

---

Published as a conference paper at ICLR 2025
A ADDITIONAL DISCUSSIONS
A. EXTENDED RELATED WORKS
Tree induction algorithms. Greedy algorithms sequentially grow trees by optimizing a given objective myopically. Popular methods in this class of algorithms are CART (Breiman et al., 1984), ID3 (Quinlan, 1986) and C4 5 (Quinlan, 1993). These algorithms differ in the predictive tasks in which they can be applied. These algorithms mainly differ in the criterion used to split the nodes at each local node, including Gini impurity (Breiman et al., 1984) or information gain (Quinlan, 1993). Owing to their greedy nature, they are computationally efficient in searching the combinatorial space. In contrast, a branch of work employs exact combinatorial optimization techniques to search for sparse, optimal trees, c.g. branch and bound (Lin al., 2020) and dynamic programming (Aglin et al., 2020). Notable works include Bi nOCT (Verwer & Zhang, 2019), DL85 (Aglin et al., 2020), OSDT (Hu et al., 2019), and GOSDT (Lin et al., 2020). These approaches are fundamentally limited by the P-haaddness of the tree induction problem, and struggle to scale to larger size problems. Additionally, they have exclusively focused on the classification setting, and are limited in the types of feature (e.g. binary or continuous features) and objective functions that can be optimized. We compare LLEGO with representative tree induction methods in Table 3.
Table 3: Comparison with the related works. LLEGO provides a general framework for global optimization of decision trees, contrasting with prior works along several dimensions: computational complexity, support for different objective and regularization functions, task types, and incorporation of structural and semantic priors.
Method
Algorithm
Worst-case complexity (
Objective
Arbitrary regularization
function
Classitication
Regression
Structural
Semantic
CART (Breiman et
Greedy
Gini impurit/MSS
x
al.,
1984)
c4.5
Greedy
(12%)
Information gain
X
(Quinlan,
1993)
DL8 5
DP
(dd)
Additive functions
V
X
V
X
(Aglin
al.,
2020)
GOSDT
DP
(d!)
Monotonic functions
X
(Lin et al.,
2020)
LLEGO
GP
OGGN)
Any
Genetic programming. GP an evolutionary optimization framework, particularly effective for variety of combinatorial optimization problems, since it only requires the provision of fitness function to evaluate and evolve a population of solutions to find optimal solutions (Koza, 1994a). As such, GP has been used in diverse tasks including tree induction (Tanigawa & Zhao, 2000; Kuo et al., 2007: Zhao, 2007; Koza, 1990), discovery symbolic mathematical expressions (Augusto & Barbosa, 2000; Qian et al., 2022), scheduling problems (Guillaume et al., 2007), neural architecture search
et al., 2020), and policy design (Hein et al., 2018). While the design of genetic operators differ significantly across domains, genetic operators share several limitations, being agnostic to the solution semantics, relying on stochastic perturbations without any search directionality, and narrow contexts. Several works in semantic genetic programming have considered the first two limitations and proposed variation operators (Krawiec & Pawlak, 2013; Moraglio al., 2012) or rejection sampling mechanisms (Krawiec & Lichocki, 2009) to obtain semantic consistency between the offspring and their parents. However, these methods are domain-specific: for example, (Krawiec Pawlak, 2013) considers convex combinations in the particular case of symbolic expressions. This limits their generalizability. and we note that semantic operator has been designed for the tree induction setting which is the focus of our work.
LLM and optimization. Recent studies have explored LLMs for optimization tasks, exploiting their domain priors to enhance optimization efficiency (Song et al., 2024). Notable applications include prompt (Yang et al., 2024), reward-function (Ma et al., 2024), and code optimization (Liventsev et al., 2023). Particularly relevant is research employing LLMs as variation operators. (Lehman et al., 2023; Nasir et al., 2024; Brownlee et al., 2023) use LLMSs as mutation operators for code
16

---

Published as a conference paper at ICLR 2025
evolution, sampling mutation instructions from predefined sets. LLMs also have been utilized as variation operators for prompt optimization (Fernando et al., 2024), where task prompts contain explicit directives for generating variations. These approaches generate unguided variations, primarily capabilities. For example, in (Guo et al., 2024), crossover is prompt".
utilizing L.MSs instruction-follovinn
performed using the prompt template: "Cross over the following prompts and generate Recent works have also considered the integration of LLMS with advanced evolutionary frameworks, namely quality-diversity algorithms (Pugh et al., 2016), evolve both neural architectures and variation prompts (Nasir et al., 2024). In contrast, LLEGO generates guided variations, utilizing in- context learning of patterns in parent solutions to generate intelligent variations. Specifically, LLEGO desired fitness, while LLEGO controls
steers offspring towards high-fitno fitnefiessss regions by conditioning diversity and exploration with the hyperparameter to define the offspring sampling distribution. Finally, recent work (Ye et al., 2024) has proposed using L.M for meta-heuristic optimization. It differs from LLEGO as it focuses on finding general meta-hcuristics for a set of optimization tasks rather than tailoring the search with datasct-specific characteristics and relevant domain knowledge as LLEGO does.
A.2 No FREE LUNCH
In accordance with the principle of No Free Lunch (Wolpert Macready, 1997), we expect LLEGO to excel in domains with the following characteristics:
Natural language representation: Problems where solutions are expressible in natural language. enabling LLEGO to employ the LLM's semantic and contextual understanding for effective variations.
Complex genotype-phenotype mapping: Tasks with low locality, where LLEGO's semantic prior enhances variation efficacy.
3. Contextual knowledge: Domains benefiting from broader knowledge, where contextual knowl- edge (e.g. clinical guidelines for risk scoring) can be flexibly incorporated via prompt design (C). This integration remains non-trivial for traditional evolutionary algorithms.
4. Challenging operator design: Areas where conventional semantic operators are difficult craft (e.g. preserving semanties program synthesis). LLEGO offers broadly applicable and flexibly customizable semantic variation operators.
These characteristics are prevalent many applications, including decision trees, mathematical equations, and symbolic programs.
B COMPLETE PROMPTS
Prompt design. In this section, we describe the details of the prompts. To recap, each of the genetic operations is realized through natural language queries to the LLM. Each prompt is constructed of three essential elements:
1. Task context. characteristics of the dataset D, e.g. number of samples, categorical features, continuous features. It also includes feature summary characteristics that are computed on the training set.
This includes information about the input space X' the output space V. and the
2. Parent trees. This contains the tree structure of each parent and possibly the fitness metric (in the case of crossover). These are translated to natural language and provided as few-shot examples to perform ICL in each genetic operation.
3. Task-specific instructions. For each genetic operator, include task-specific instructions offspring generations and guidelines on the format of the response.
The structured prompt for mutation is described in Figure 8. Descriptions enclosed in (), such as [task_dees des_eritsssoon represent placeholder values that are populated dynamically at run-time. For a concrete example of this, the mutation prompt credit dataset is shown in full in Listing 1. Similarly. the structured prompt for crossover is described Figure with a concrete example shown in Listing 2.
17

---

Published as a conference paper at ICLR 2025
fask_descriptionn The dataset contains (n_samples) samples and (n_attributes) features, of which (n_ [__uumerical) are numerical and (n_categorical) are categorical. The target variable [target_name), it is [target type], [label information). The features and their ranges are: cecaturr_semaaticssss ). You should generate a diverse decision tree that is more interpretable. Please generate decision trees in the desired ISON format, you can use any of the features, but are only allowed to use operators ( >, <=, >=]. Return only the JSON in the format ### tree #t.
Figure 8: Prompt structure for mutation operation.
flask_descriptions The dataset contains (n_samples) samples and [n_attributes] features, of which [n_numerical] are numerical and (o_categorical) are categorical. The target variable is (target name), it is (target_typs) (feature_ setnrrtcsmmm Generate a different, interpretable decision tree which should have the improved fitness. Please generate decision trees in the desired JSON format, you can any of the features, but are only allowed to use operators [<, >, <=, >=]. Return only the JSON in the format #tf tree ##:.
(label information). The features and their ranges are:
Figure 9: Prompt structure for crossover operation.
The task is to classify people described by a set of attributes as good or bad credit risks. The dataset contains 360 samples and 20 features, of which are numerical and 13 target variable class, it is binary, the label distribution is [0: 29.178, 1: 70.838. The features and their ranges are: [checking_status (int) [o, 31, duration (float) [5.00, 60.00), eredit_histon (int) [0, 41, purpose (int) [0, 91, erodit_amount
categorical. The
(float) [276.00, 15672.00], savings_status (int) [0, 4], employment (int) [0, 4], cetttalinnet_ccmmitmee personal_statup (int) [o, 3], other_parties (int) [0, 2],
(float) [1.00, 4.00),
residence_since (float) (1.00, 4.00), property_magnitude (int) [0, 31, age (float) 119.00, 74.00), other sacee__pammnt__aare (int) 10, 21, housing (int) 10, 2), existing_crediee (int) [0, 31, num nun_dependents (float) [1.00, 2.001, own_telephone (int) [0, 11, foreign_worker (int) [0, 111. You should generate a diverse decision tree that is more interpretable. Please generate decision trees in the desired JSON format, you can use any of the features, but are only allowed to use operators [s, 2, <=, >=]. Return only the JSON in the format ## tree #
(float) (1.00, 11.00, 4.001, job
Expression: # ('oredit_history': ('<= 1.5000'; ("property_magniuude''
['<= 0.500''; l'employment's ('<= 1.500''; ('value'; 11, "> 1.5000
['value': 0}}), 0.5000'; ('value': 0)]), '> 1.5000':
('savings_status" {'<= 3.5000': ("property_magnitudee" 0.5000': ('valuef: 02, <> 0. 0.500'': ('value': 111,, "> 3.5000':
('<=
['employment': <'<= 2.5000 l'value' 11, <> 2.5000' : I'value'
111111 ##
Expression: ## l'other
paameett_laant: <'<= 1.5000':
[ "prroperty_magnituee [ {<<- 1.5000': [' ('own_telephone': ('<- 0.5000':
['value': 0), "> 0.5000"; ('value': 0)1), <> 1.5000':
d'nun_dependents': ('<= 1.5000'; ('value': 11, '> 1.500'' l'value':
01 >> 5000' !'purpose:; ('<= 6.5000'; (rresidenc__since
1,50000 I l'value's 11, <> 1.5000': I'value': 1111, r> 60000"
['housing' <'<= 0.5000' ( ('value': 1), <> 0.5000': ('value':
1|1 ##
Expression: ## ('eredit ('<= 3.5000'; ( ('duration': {'<=
34.5000': (' ('checking_statue"
('<= 1.500'' ( ("value': 11, />
1.5000'; ; ['value': 1111, <> 34.5000'; 1' e'eredi_aamuunt <'<=
10552 ('valuef: 11, t> 5000' ('value': })))}, />
3.5000' : ('credit_amount'' (*<= 9597.5000' (*employment's K'
18

---

Published as a conference paper at ICLR 2025
1.5000' ('value': 11, <> 1.5000': ('value': 11F1, <> 9597.5000'
('value': 0)))) FF
Expression: At ("proprrty_agnitude"" ('<= 0.5000'; ('duration': ('<=
33.0000": ("housing"; <'<= 1.5000'; ('value': 1h, <> 1.5000';
['value': 0]]},, <> 33.0000": ('employment': ("<= 0.5000': ('value': 0), 1> 0.5000': ('value': о}}}},, <> 0.5000': : ('employment's ('<= 0.5000': ('credit_amount"
{'<= 5000' ('value': 0), t>
5309; I'value': 11th, (> 0.5000'; : l'purpose:: {*<= 5.5000': ('value': 11, t> 5.5000'; ("value': f133)333 #
Expression: ##
Listing I: Example mutation prompt. On credit dataset.
The task is to classify people described by a set of attributes as good or bad credit risks. The dataset contains 360 samples and 20 features, of which 7 are numerical and 13 are categorical. The target variable is class, it is binary, the 1abel distribution is [0: 29.178, 1: 70.838 The features and their ranges are: barcking__tattue (int) [0, 31, duration (float) 15.00, 60.00), credit history (int) [0, 41, purpose (int) [0, 91, credit amount credit_amount
(float) [276.00, 15672.001, savings_status (int) [0, 41, employment (int) [0, 4], cnstallinent_comitnnee personal_status (int) to, 31, other_parties (int) [0, 2],
(float) [1.00, 4.00),
residence_since (float) [1.00, 4.00], propertyy_agnitzude (int) [0, 31, age (float) [19.00, 74.00], rcher_payment_plans (int) [0, 21, housing (int) [o, 2], existing_credits. (int) [0, 31, dem_ndeesssnts (float) [1.00, 2.00], own_telephone (int) [0, 1l, foreign_wordd worker (int) [0, 1]]. Generate a different, interpretable decision tree which should have the improved fitness. Please generate decision trees in the desired JSON format, you can use any of the features, but are only allowed to use operators [<, >, <=, >-]. Return only the JSON in the format di tree ##.
(float) (1.00 4.00], job
fitness: 0. 5882, Expression: # ('purpose': {'<= 5.5000'; l'housing's
[ {'<= 5.500'' I'residence_sineee since': <'<= 2.5000'; ('value': : 0), '>
2.5000'; : ('value'; : 111, <> 0.5000'; j'job': ('<= 1.500''; ('value';
0), <> 5.50'''; I'value': 11HH), <> 5.500''; ('duration': {'<=
25.5000' (Meredit_history''s ('<= 3,5000'; ('valuc'; 11, <> 3.5000' ('value': 1111, <> : ("residence_since
('<= 3.5000':
['value': 1), '> 3.5000':
0111111 #
f fitness: 0.5930, Expression: # ("savings_status'' (*<= 2.5000'
{1 ['credit_amount'
( 9597.5000': (/orettt
<*<- 0.5000':
('value': 01, <> 0.5000': ('yalee': <> 9597.5000' v'value':
0)11, <> 2.5000's I'ehecking_status': {'<= 0.5000':
0'500' l'vaue'' 01, <> 0.5000':
['value': 1)1), <>
('<- 2.5000':
['value': 1), 1> 2.5000'; :::00:::('vvlllllal 2 = 41
fitness: 0.6162, Expression:
("property_mmannttuers ('<= 0.5000':
['duration': ['<= 33.0000'; ('housing': ('<= 1.5000'; ('value's 11,
<> 1.500' l'value': 0ith, 3>.0000''' : ('employment': {*<=
0.5000': ('value': 0), <> 0.500'' ( 01111), <> 0.500'':
['employment': {'<= 0.500' :0::500000("rree_____"0"1""1 ['erediit_mountt' ('<= 3359.5000';
('value': 0), <> 3359.5000'; (tap:000000(11l0000 (/value': <> 0.500'r: 1 t'purpose':
('<= 5.5000': ('value': 11, <> 5.5000':
1111111) ##
fitness: 0. 0.6815, Expression: # ("checking_status'' ('<= 1.5000r:
ppproperty_mannitiee 1.500000" ["other_parties' ('<- 0.5000r:
['value': 0}, "> 0.5000': ('value': =))e) 1.500'' ('duration':
['<= 20.5000"; I'value' : 11, 1> 20.5000'; ('value': 111111, '>
1.5000': l'credit hiaattthistory'sss ('eo ft== 2.5000' ('num dependents' {'<=
1.5000': l'value': 11, <> 1.500' v'val''' 0tH,, <> 2.5000'
pescher_pammen__laane" (?<= 1.5000r; ('value'; 1), >> 1.5000';
('value': 1))))))}
fitness: 0.6908, Expression:
Listing 2: Example crossover prompt. On credit dataset.
19

---

Published as a conference paper at ICLR 2025
Tree representation. We represent trees in natural language nested dictionary. This dictionary represents a decision tree where each key is a feature and the subsequent nested dictionaries corre- spond to decision rules and their outcomes. An example is illustrated in Figure 10 on the iris dataset. In this example, if "petal width (cm)' is less than or equal to 0,80, the classification is O; otherwise, further splits are made on 'petal width (em)" 1.75, leading to classifications of 1 or 2 depending on the condition.
(Em) <H sompliyy 750
"potol width (cm) ":
0.80"; ("value"; 0)r
solzan
"s 0.80%:
dri-
pita wish ure 1.9
"petal widch (=m)*: {
comples 201
<<= 1.75*; ["valus": 11,
"> 38"s (*value": 2)
campies 06 +50
COSS VNSSCAE
Figure 10: Example decision tree, And its corresponding natural language representation as a nested dictionary.
B.I ABLATION PROMPTS
In our ablation study, we removed all semantic information from the prompts, with examples illustrated in Listing and 4. Here, we remove the semantic description of the task, and replace its features names with X,.
The task is to generate interpretable and high-performing decision trees given a set of attributes. The dataset contains 360 samples and 20 features, of which 7 are numerical and 13 are catogorical. The
target variable is y, it is binary, the label distribution is [0: 29.178, 1: 70.70.83 83.833). The features and their ranges are: [X_0 (int) [o, 31, X_1 (float) [5.00, 60.00), X_2 (int) [0, 4], x_3 (int) [c, 91, X_4 (float) 1276.00, 15672.00), X_5 (int) [0, 41, X_6 (int) [0,
41, X_7 (#loat) 11.00, 4.001, X_8 (int) [0, 31, X_9 (int) 10, 21,
x.10 (float) 11.00, 4.00), X_11 (int) IC, 31, X_12 (Eloat) 119.00,
74.001, X_13 (int) [0, 21, x_14 (int) [C, 21, x_15 (float) [1.00,
4.001, X_16 (int) 10, 31, x_17 (float) I1.00, 2.00], X_18 (int) [o, 11, X_19 (int) [0, 1il. You should generate diverse decision tree that is more interpretable. Please generate decision trees in the desired JSON format, you can use any of the features, but are only allowed to use operators [<, 2, <=, >=]. Return only the JSON in the format ## tree IF.
Expression: # 1''_6''; ('<= 1.5000'; 12X_1''; ('<= 38.5000'; 1'X_4': ['<= 2443. l'value' 11, "> 2443.0000' ('value': 0)]}, '> 38.5000': ('X_1': 11.-21.0000
('value': 0), 1> 21.0000'
['value': 1))))}, <> 1.5000 ('x_0': (f<- 1.5000' ('X_3': ('<-
5.5000': I'value': : 01, <> 5.5000'; l'value': 1111, <> 1.5000':
['X_1': ['<= 19.0000's ('value': 11, <> 19.0000'; ('value's 11EHE ##
Expression: H1 ('x_0': (r<- 0.5000': ('x_4': ('<- 976.5000' ('x_3':
0}),, [f<- 3.5000'; ('value'; 01, 1> 3.5000': ('value'; O)))) />
976.5000' ('x_5'; ('<= 1,5000': ('value': 01, <> 1.5000'; l'value':
1|HH, "> '> 0.5000'; ('x_4': ('<== 13765.5000', ('x_12': ('<=
22.5000' ['value's : 0), <> 22.5000': l'value': 111,, t> 50065
('value': 0)]1)) ##
Expression: ## ('X_5': ('<= 2.5000' (''_4': {'<= 9597.5000' 2'x_2':
{<<= 0.5000'; I'value's : 01, <> 0.5000'; ('value's ; 11]],, >>
9597.5000 l'value': 0tH,, <> 2.5000'; ('x_0': ['<= 0.500'';
['x_il': ['<= 0.5000'; ('value': 0), <> 0.5000": ('value': ; 1111, i>
0.5000': : ('x_10': {'<- 2.5000" : ('valuel: :1, <> 2.5000': ('value!:
1EEEEE] ##
20

---

Published as a conference paper at ICLR 2025
Expressicn: ## 2': (<<= 11, <> 23.50 I'valuer; 011,, f> 0.5000'; ('X_5'; (/<= 3.5000'
2
0.5000': |'X_12'; ( (<<= 23.5000': l'value's
['X_12'; <'<= 2000'' ('value'; 0), (> 25.5000'; ('valuo's 1)),, <> 3.500''; ('X_4'; <<<= 1034.5000'; : ['valuc'; 0}, >> 1034.
('value's 1}}))}}} #4
Expression: ##
Listing 3: Example mutation prompt with semantics removed. On credit dataset.
The task is to generate interpretable and high-peri decision trees given a set attributes. The dataset contains 360 samples and 20 features, of which are numerical and 13 are categorical. The target variable is it is binary, the label distribution is [0: 29.178, 1: 70.838]. The features and their ranges are: [x_0 (int)
[0, 31, x_1 (float) [5.00, 60.00], x_2 (int) [0, 41, x_3 (int) [C,
9), x_4 (Elost) [276.00, 15672.00), x_5 (int) [0, 4), X_6 (int) [0,
4), X_7 (float) [1.00, 4.001, x_8 (int) [C, 31, X_9 (int) [0, 21,
x_10 (float) [1.00, 4.001, X_11 (int) [C, 31, X_12 (float) [19.00,
74.001, X_13 (int) [0, 21, X_14 (int) [C, 21, X_15 (float) [1.00,
4.001, X_16 (int) [0, 31, X_17 (float) 11.00, 2.00], X_18 (int) [0,
11, X_19 (int) [0, 11l. Generate a different, interpretable decision tree which should have the improved fitness. Please generate decisien trees in the desired JSCN format, you can use any of the operators 1s, 2, <=, >=]. Return only the JSON in the format ## tree #t.
features, but are on-y allowed
fitness: 0.5882, Expression: 44 ('X_3'; ('<= 5.5000'; ('X_14'; f'<=
0.5000'; ('x_10': {'<= 2.5000'; ('value'; : 01, <> 2.5000'; ('value'; 11H,, <> 0.5000'; {'x_66'; ('<= 1.5000'; l'value'; 0), "> 1.500' ['value': 1))1)), 1)]]),, t> 5.5000'; ['x_1': {'<= 25.5000': ('x_2': ('<=
3.5000': ['value': 1), (> 3.5000'; ('value': 11H, "> 25.5000': ['X_10': ('<= 3.5000': ('value': 1), >> 3.5000': : ('value': оуهеиии #4
fitness: 0.5930, Expression: # ('X_5'; <'<= 2.5000' ('x_4'; I'<=
(r<= 9597.5000'; : 2'x_2'; ("<e= 0.5000'; l'value's 01, <> 0.5000':
['value': 11)), 9597.5000": ('value': 6HH, r> 2.5000': ('x_0'; ['<- 0.500'': ['x_11f; (f<- 0.5000': ('value's 0}, 1> 0.5000': {'<- 2.5000'; ('value': 1),
['value': 11)), <> 0.5000': ('x_10':
2.5000': I'value': 1 #
fitness: 0.6162, Expression: # 11x ['<= 1'<= 0.5000': 1 {'<=
33.0000': 04t,, <> 33.0000'; ['X_6'; ('<= 0.5000'; :::::::vvlllll ('value': 0), 1> 0.5000';
( 1.5000': l'value': <> 1.5000'; [ l'value':
('value': 0))))), (> 0.5000'; ('X_6': 0,50000'
('<=
3359.5000'; ('value': 0), (> (faal000
0.5000";
['X_3': ('<= 5.5000': ['value': 1> 5.5000' : ##
fitness: 086815, Expression: #
('<=1.5000: {(<=
1.5000': ["X_9': ('<= 0.5000': ('value': 0), 0.5000': [ ('value':
0111, <> 1.5000': I'Xi'; ('<= 20.5000'; ('value': 11, "> 20.50000''
('value':
11111,, 111111, <> 1.5000": ("500:
('<=
1.5000': I'value': 1;, t> 1.5000'; l'value': 011,, i, <> 2.5000'
[3x_13': ('<- 1.5000': ('value': 1), >> 1.5000 : ('value': 13131ии
fitness: 0.6908, Expression:
Listing 4: Example crossover prompt with semantics removed. On credit dataset.
C DETAILS OF EXPERIMENTAL PROCEDURES
In this section, we outline the benchmark datasets employed in our evaluations, as well as implemen- tation details of our method and considered baselines.
21

---

Published as a conference paper at ICLR 2025
C.I DATASET DETAILS
We employ a total of 12 datasets for our evaluation, of which 7 are classification tasks, and 5 are regression tasks. Additionally, we consider propriety datasets Appendix D.2, for which the LLM would not have seen during pretraining, and thus used to check for any memorization concerns.
Open-source datasets. The open-source tabular datasets are sourced from OpenML (Vanschoren et al., 2014). The classification datasets were selected from the curated suite OpenML-CCI8 (Bischl et al., 2019) with the following criteria: < 20 features, I 10000 samples, binary labels and no missing data. This stems from the fact that optimal tree induction methods scale exponentially with the number of features and samples, and some baselines only support binary classification. Additionally, excluded datasets lacking semantically meaningful feature names and descriptions, required by LLEGO. Regression datasets were selected from OpenMI.-CTRO (Fischer et al., 2023) with identical criteria. We detail dataset characteristics, including OpenMI. ID, number of attributes, number samples and label distribution in Table 4. These datasets can be loaded by querying their OpenML IDs. The datasets describe:
credit (Kelly et al.): This dataset classifies people as good or bad credit risks.
diabetes (Smith et al., 1988): This dataset classifies patients based on WHO definition of diabetes. compas (Inc., 2016): Contains criminal history, jail and prison time, demographics, and used to predict two year recidivism.
heart (hea): Prediction of heart disease in patients.
liver (Kelly et al.): This data set contains 416 liver patient records and 167 non liver patient records. The data set was collected from north east of Andhra Pradesh, India. The class label divides the patients into 2 groups (liver patient not). This data set contains 441 male patient records and 142 female patient records.
heart (Street et al., 1993): Features are computed from digitized image of a fine needle aspirate (FNA) of breast mass. They describe characteristics of the cell nuclei present in the image. The target feature records the prognosis (malignant or benign).
vehicle (Pete & Shepherd): The dataset classifies a given silhouette as one of four types of vehicle, using set of features extracted from the silhouette. The target label is re-relabelled, where the majority class as positive ("P") and all others as negative ('N').
cholesterol (Janosi et al., 1988): The dataset predicts the cholesterol level among patients diagnosed with heart disease.
wine (Cortez et al., 2009): The task is to predict quality of white and red wine.
wage (Berndt, 1991): The task is predict individual wages using the Current Population Survey
(CPS), used to supplement census information between census years.
abalone (Nash et al., 1995): Predicting the age of abalone from physical measurements. The age of abalone is determined by cutting the shell through the cone, staining it, and counting the number of rings through microscope a boring and time-consuming task.
cars (Bohanec, 1997): Dataset of the suggested retail prices (column Price) and various character- istics of each car.
Table 4: Open-source datasets. Details of open-source datasets from OpenML (Vanschoren et al., 2014). Cat: number of categorical attributes, Num: number of numerical attributes, Label dist: label distribution.
Dataset
ID
# Samples
# Attributes
# Num
Cat
Label
Label distr
credit
31
1000
20
7
13
binary
0: 29.17%, 76, 1: 70.83%
diabetes
37
768
0
binary
0: 66.30% 1: 33.70%
compas
42192
5278
binary
52.50%. 1: 47.50%
heart
270
binary
0: 52.58%, 1: 47.42%
liver
1480
583
binary
0: 67.94%, 1: 32.06%
breast
15
699
9
9
O
binary
0: 65.34%, 1: 34.66%
vehicle
994
846
18
18
O
binary
0: 73.03% 1: 26.97%
cholesterol
continuous
wine
287
6497
11
11
0
continuous
wage
534
534
3
7
continuous
abalone
44956
4177
8
7
continuous
44994
804
17
16
continuous
22

---

Published as a conference paper at ICLR 2025
Dataset preprocessing. We preprocess the dataset using wrain-validation-ess split ratio of [0.2, 0.4, 0.4]. The low training split is used to accentuate the difference in performance as given sufficient training data, all methods perform comparably. For each run, we only vary the seed used for data splitting, such that for seed O, we use train_test_spp littt (seed=0). For any algorithms that have inherent randomness (i.e. CART and GATree), we seed them with seed=42. As such, the randomness reported is induced only by different datasets.
We do not apply any additional preprocessing continuous features. For categorical features, we follow the recommendations provided in $9.2.4 of (Hastie et al., 2009), where we rank each category of the predictor by calculating the proportion of observations that fall into the outcome class (Hastie et al., 2009). This results in ranking of the categories based on these proportions. No additional preprocessing is applied to categorical or continuous labels.
C.2 IMPLEMENTATION DETAILS
Baselines. To assess the performance of LLEGO, compare it against comprehensive set of state-of-the-art algorithms, covering representative methods from main categories of tree induction. Specifically, CART and C4.5 are greedy tree induction methods, GOSDT and DL8.5 optimal tree induction methods, and GATree a genetic programming based approach:
CART (Classification and Regression Trees) (Breiman et al., 1984): CART is a decision tree algo- rithm that splits data into subsets based on feature values, creating binary tree for classification regression tasks using measures like Gini impurity or mean squared error. We use the implemen- tation provided in sklearn. tree, https: ptttps:/scikkieee wttsssiiiiitteaeeeeeeeeeeeeeeeae
org /sstsableeoouuuut html.
generated/ss hetdd/skeearr tree.
C4.5 (Quinlan, 1993): C4.5 is an extension of the ID3 algorithm that generates decision trees by handling both categorical and continuous data, and uses information gain ratio to choose splits. We use the implementation provided in the PyPI package c45-decis cetooottree -tree, https: 1.55-decsiinn-
GOSDT (Lin et al., 2020): GOSDT constructs decision trees by optimizing a trade-off be- tween accuracy and complexity, ensuring sparsity and interpretability through global opti- mization techniques. We use the implementation provided by the original authors https:
//github. com/ uuc-uucco systopia
DL8.5 (Aglin et al., 2020): DL8.5 is a decision tree learning algorithm that focuses constructing optimal decision trees given specific constraints, using dynamic programming to find the best tree structure. We use the implemented provided in the PyPI package d18. 5, https: / github.
http///osssdssseeeeeeeeeee guesses.
GATree (Lahovnik, 2024): GATree is a Python library designed for implementing evolution- ary decision trees using genetic algorithm approach. We use the official implementation www.reaaeeeeee hedacs io/ eo/ee//ne wwwennaaas/////eeeeeeeeeeeeeaaee and keep the defaults settings of the implementation (i.e. tournament selection, subtree crossover and subtree mutation).
Hyperparameter search ranges. Next, we detail the hyperparameters of each method, and their respective search ranges. Across experiments, we keep deathhhphhhhhe fixed to enable fair comparison, the details of tunable hyperparameters detailed in Table 5.
Hyperparameter tuning. We use Optuna (Akiha et al., 2019) and the default Tree-Parzen Estimator for hyperparameter tuning (HPT) (Watanabe, 2023). For all baselines, we permit wall-clock time to maximum of 10 minutes. This allows 50 iterations of HPT for CART and C4.5, and 10 iterations for the computationally more intensive DL8.5, GOSDT, and GATree. In each iteration of HPT, we evaluate the objective on the validation set, selecting the best configuration to evaluate on the test set.
Computer resources. We run all experiments on an AMD EPYC 7V13 64-Core Processor.
C.3 LLEGO IMPLEMENTATION DETAILS
For our instantiation of LLEGO in Section 5, we use N - 25 and G 25. We seed the algorithm with a population of trees generated by CART, where each tree is fitted on 25% of the Duain- We use the same population to initialize GATree. In each iteration, we generate 25 crossover offspring and 25 mutation offspring, using a rejection mechanism where invalid solutions are discarded (in Section 5, ~ 86% of crossover and ~ 88% of mutation offspring are syntactically valid). We use
23

---

Published as a conference paper at ICLR 2025
Table 5: Hyperparameter search ranges. Hyperparameler search ranges for all baselines.
min_ Lin_samples___ ssppls min min_samples, sesppees_ppittttt max sux_depth splitter
[int, 2, 16] [int, I, 16] fixed best
CART
criterion
['squared_ere eeroor" (reg), "gini" (clas)
min min_samples_sppe split
[int, 2, 16]
C4.5
min jin_ssamlee__sppee
[int, 1, 16] fixed
max_depth
min_sup max_depth
[int, I, 10]
DL8.5
fixed
regularization
[float, 0.001, fixed
GOSDT
max cix__dpphh__
population size
iint, 10, 50]
mutation_prot
[float, 0.1, 0.5]
GATree
crossover_probe
[float, 0.1, 0.95]
iterations
100
tournament size max_depth
fixed
elitism selection to preserve the top 25 trees after merging the offspring of the crossover and the mutation. To compute the desired fitness, the value that balanced diversity and fitness. We use = 10 for diversity guidance. For each genetic For experiments,
0.1, based on observations in Section 5.2 as
operation, we À parent with default hyperparameters temperature = 0.7 and top 0.95.
gpt-35 335-turbo, version 0301
Function and terminal set. For both LLEGO and GATree, the function is {<,>, S 2} and the terminal set includes numerical constants based on target feature values.
In Section 5.2, we perform steps of crossover starting from the initial population, for both LLEGO and GATree to obtain Figure 5. We similarly perform 3 steps of mutation starting from the initial population obtain Figure
C.4 EVALUATION METRICS
MSE. For regression dataset, we report MSE www.:meetrii....
1 Σ
MSE f) [lf(a,) - Unl
N
Balanced accuracy. For classification datasets, we report balanced accu- racy, which is equivalent (sklearn.mett metrics walaaaaan aalllancddcccce segpeed__accurc___orcee equal importance both the positive and negative classes, thereby mitigating the impact of class imbalance and providing more reliable assessment of the classifier's 's performance across all classes:
accuracy with class-balanced sample weights
This has the effect of giving
TP BAcc TP+FN TN FP
TN
Difference equal opportunity. When evaluating fairness, we consider difference in equal opportu- nity (DEO). This score measures the difference in recall between unprivileged and privileged groups, where a value of DEO = 0 indicates equality opportunity.
DEO = IP = group = 1, y = 1) - p(O group = 0, =1]
We utilize the implementation ai a363.0.0.0 welleann.metrice equal .pportunity..................... difference provided in https: //tts/s////ff//0 rew.et...eeeeeeeeeeceeeeee readthedocs. io/ (Bellamy et al., 2019; Roemer & Trannoy, 2015).
24

---

Published as a conference paper at ICLR 2025
Population Fitness. In order to assess the fitness of the populations evolved by the GP-based algorithms, we compute for given population P:
Fitness = (Median([((t))
where (t) denotes the normalized accuracy, calculated as max (t)-minges f(t)* where f(t) here denotes the accuracy. P' is the union of all individuals produced by all methods for particular seeded run on a particular dataset. In other words, the best accuracy obtained by any method on particular seed for a particular dataset will have (t) and the worst will have f'(() = 0. This
1(t) f(t)-min.gp f(t)
normalization allows accuracy results from different datasets, seeds, and methods to be compared.
Population diversity. In order to assess the diversity of the populations evolved by the GP-based algorithms, we compute for given population P:
Diversity = Median ((]lp(t) - ((t))ll I (t,t) E p2))
where (t) denotes the functional signature of t, i.e. the vector (f(æ1), ..... t t(xn))))))))))))))))))())))))t
D ADDITIONAL RESULTS
In this section of the appendix, we provide additional empirical results. Specifically;
1. In Appendix D.I, we analyze the train-test generalization gap.
In Appendix D.2, we report generalization performance on tasks with all semantics removed. The objectives of this experiment are to (1) check for memorization and (2) evaluate the contribution of semantic priors to search efficiency. We also evaluate LLEGO on proprietary datasets.
In Appendix D.3, we investigate the potential for bias and the flexibility of LLEGO optimizing for reiulis--e objectives.
4. In Appendix D.4, we demonstrate that the log- probbabbaillssss of candidate trees is directly correlated with the structural distances.
In Appendix D.5, we compare LLEGO against version of GATree running with larger population sizes and more generations than LLEGO.
6. In Appendix D.6, report the runtimes of LLEGO and the baselines.
7. In Appendix D.7, we perform additional ablation studies key variables, including prompting strategies, choice of underlying I.M., genetic operation arity, and different parent sampling mechanisms, and population initialization strategies.
8. In Appendix D.8, provide fine- fineigrained results of the aggregate analysis presented in the main paper.
D.1 GENERALIZATION GAP
averaged across the classification datasets for each method. The results show that LLEGO consistently achieve lower generalization gap compared to the baselines. In particular, the difference between LLEGO and the baselines is more noticeable at depth 4, where the baselines are more susceptible to overfitting, such as the optimal induction method 5L85. This aligns with empirical observations in recent works (Zharmagambetov et al., 2021; Marton et al., 2023; Sullivan et al., 2024).
In Table 6, we report the generalization gap (defined as BAce Baaoqaii-BAcsaaal BArcuoia
BAcecat)
Table 6: Relative generalization gap. Averaged over the classification datasets.
Method
Depth=3
Depth=4
C45
0.073
0.086
CART
0.078
0.101
DL85
0.131
0.161
GATREE
0.086
0.092
GOSDT
0.044
0.052
LLEGO
0.043
0.043
25

---

Published as a conference paper at ICLR 2025
D.2 INVESTIGATION INTO MEMORIZATION
As with any LLM application, there is a concern about LLM memorization, Although it is highly unlikely that the LLM has encountered the optimal trees for the considered datasets- especiaaly given that high-perforning solutions can vary significantly across different training splits, seeds, and preprocessing steps-we empirically investigate this concern. This is done by removing any dataset-specific metadata or semantic information that could identify the underlying data. For prompts with semantics removed, please refer Appendix I. We refer to this setting as LLEGO semantics and compare its performance against LLEGO with semantics included in Table 7. We observe that LLEGOno semantics achieves similar performance, even outperforming LLEGO on two of the tasks.
Table 7: Performance on classification tasks. Comparing LLEGO with LL no semantics (i.e. all
semantic information removed). Best results are emboldened.
Method
Compas
Credit
Diabetes
Heart
Liver
depth 3
LLEGOno ge_nannnnniss semantics
0, (0.04(0.010)
1.683(0.012)
0.700
0.726(0.030)
0.643
LLEGO
(0.04)
0.679
0.713
0. 0.736(0.024)
0.672
depth = 4
LLEGOno semantics
(..5)))
0.667(0.024
(..03)
0.716(0.098)
(o.025)
LLEGO
0.663(0..00)
(. 0.721(0.017) 0.751
0.676 (..01)
To further verify that LLEGO's superior performance does not rely on memorization, we evaluate it on two proprietary datasets (requiring authorized access, and hence extremely unlikely to be in the LLM training corpus): MAGGIC (heart failure, (Wong et al., 2014)) and CUTRACT (prostate cancer, (CUTRACT, 2019)). We report the results against CART and GATree for depth in Table showing that LLEGO achieves superior performance on these private datasets, further demonstrating that it relies on generalized semantic priors rather than dataset-specific memorization.
Table 8: Performance on proprietary datasets. Comparing LLEGO with CART and GATree, with
depth- = 4. Best results are emboldened.
Method
MAGGIC
CUTRACT
CART
0.610
0.694(0.038)
GATree
0.619
(.706
LLEGO
0.623 (0007) (..10000.009)
D.3 ADDRESSING BIAS VIA REGULARIZATION
As illustrated in the previous experiments, the genetic operators in LLEGO benefit from the properties
of LLMs (i.e. semantic priors and wide context). It is then natural to wonder if, conversely, negative artifacts of LLMs may propagate to the decision trees found by LLEGO.
Setup. this experiment, we focus in particular on bias. More precisely, we assess group fairness (Verma Ru- bin, 2018) by computing the Difference in Equality of Opportunity (DEO) metric, defined as the difference in recall between unprivileged and privileged groups (cf. Ap- pendix C.4 for an exact definition). We show an illustrative example on the dataset COMPAS, which is known to be biased on the sensitive attribute race African American (Angwin et al., 2016). Our objective is to mitigate bias with a DEO-based regularization, by defining LLEGO's new fitness function, i.e. ft) (t) + (t) for any t T.
Table 9: Fairness aware objective.
Compas
Method
FA?
ACC (1)
DEO (L)
CART
X
(.651 (.255
C4.5
X
0.650
0.258
DL85
0.666.o.oomm
(0000 0.264(0.008))
GOSDT
X
(.64](p.003)
0.187 (0.019)
LLEGO
x
0.308 0.632(0..00)
(0.70)
LLZG0
v
0.651 (o.02)
0.161
As be seen in Table 9, LLEGO does not natively return fair decision trees when the fitness functions are based purely on accuracy. However, the DEO regularization permits LLEGO to find decision trees with less bias compared to the other baselines. This highlights the flexibility
Results.
26

---

Published as a conference paper at ICLR 2025
of LLEGO, which can handle composite search objectives unlike the other baselines. LLEGO also returns a population of individuals, which makes it possible to trade-off predictive performance with fairness metrics. We show this in Figure 11, where one choose individuals returned by LLEGO with acceptable tradeoffs.
Depth
Depth
0.07
0.68
ACC (1)
0.00
0.50
0.63
0.00
0.25
0:30
0.18
0.30
0.55
DEO
(4)
Figure 11: Accuracy-fairnes tradeoff. On compas dataset.
D.4 CORRELATION BETWEEN LOG-PROBABILITIES AND TREE EDIT DISTANCE
We show in this experiment that the bog-probabilities of the offispring trees (utilized in LLEGO's mutation operator) are negatively correlated with the structural distances between parent and offspring solutions.
Experimental setting, We generate 1000 offssring trees using the LLEGO's mutation operator with single parent tree. For each offspring individual, we assess its structural distance to the parent tree by computing the Tree Edit Distance (TED) (Bille, 2005) between this individual and the parent.
Observations. As shown in Figure 12, we observe strong negative correlation (correlation coeffi- cient -0.85) between log-proball probabiliiiie of the offspring and TED values. This relationship indicates that offspring with lower log- pr-obabbiliitess tend to exhibit greater structural differences from their parent, as measured by the TED, This demonstrates that LLEGO's 's Ses---obbbiity-bbaae selection mechanism inherently promotes diversity in the population by favoring mutations which introduce varied structural changes.
Correlation = -00,55
10
Log Probability
-20
-30
60
0.0
0.2
0.4
0.6
0.8
Tree Edit Distance
Figure 12: Correlation between log-rrobabilities and stuctural distances. There a strong negative correlation between log-probabilities of the offspring and their TED values with respect to the parent tree.
27

---

Published as a conference paper at ICLR 2025
D.5 ADDITIONAL COMPARISON WITH GATREE
We extend our comparisons against GATree by increasing the population size to N = 100 and the number of generations G 200, while keeping LLEGO's default hyperparameters. We report the results for the classification and regression tasks in Table 10 and Table 11. Despite GATree's 's larger number of evaluations, LLEGO evolved superior trees. This underscores the importance of LLEGO's integration of semantic priors, search guidance, and broader context to enhance search efficiency. This superior search efficiency is especially important in settings where evaluation costs significantly exceed search costs (e.g. complex simulations, hardware optimizations, robotics control).
Table 10: Comparison against GATree. Balanced accuracy (1) on classification tasks (depth d = 4).
Method
Compas
Credit
Diabetes
Vehicle
GATREE (N= 100, G = 200)
0.848 0.951 oor)
D. owo))
0.6877 0.6677
0 0.68-k).m3)
0. 0.738
0.636gooz
0 0.939 (o92))).
LLEGO
(, o05)
0.721mmm m.1
0.751,e 042)
Table 11: Comparison against GATree. MSE (L) on regression tasks (depth d = 4).
Method
Abalone 0.566
Cars
Cholesterol
Wage
Wine
GA GATRES (N = 100, G 200)
0. (0.013
1 1,30 201)
1. 1.148
0.829 ogm
(N 25, G 25)
(.5
1.322 145:
1.06770.208 (0.20))
0.828 o cos)
D.6 RUN-TIME COMPARISONS
We provide the total runtimes for the different methods in Table 12, averaged across the 7 classification datasets used in Section 5.1. We also report in Table 13 the detailed timings for LLEGO and CATREE with varying population sizes (P E [25, 100}) and generations (G' E (25, 100, 200}), and also report the number of functional evaluations. These results, along with the ones presented in Section 5. highlight that LLEGO evolves superior trees compared to GA GREREE while necessitating less functional evaluations and wall-clock time. Nevertheless, we acknowledge that there is room for improvement for the runtime of LLEGO. Potential solutions include (1) reducing runtime through inference acceleration techniques such as speculative decoding and vLLM serving (Leviathan et al., 2023) and (2) reducing memory requirements through specialized fine-tuned models or quantization (Han et al., 2015).
Table 12: Runtime comparisons (all methods). Total runtime (in seconds), averaged across 7 classification datasets.
CART
C4.5
DL85
GOSDT
CATREE
LLEGO
Total run time (depth d 3) Total time (depth d
0.0022
0.08
22.10
261.14
15.50
407.66
0.0023
0.13
172.70
234.44
15.77
430.32
Table 13:
Runtime comparisons (GP methods). Per-generation, total runtime (in seconds), and total number of fitness evaluations (depth d 4, averaged across classification datasets).
Per Per-generation runtime
Total run-time
Functional evaluations
GATRER (N 25, G = 25)
0.63
15.77
620
CATREE (N 100, G 100)
2.65
264.95
9600
GATREE (N 100, G 200)
3,86
772.97
19200
LLEGO (N = 25. = 25)
17.22
430.32
1250
D.7 ADDITIONAL. ABLATION RESULTS
This subsection performs additional investigations into several key variables affecting LLEGO perfor- mance. We investigate the impact of diverse prompting strategies (Appendix D.7.1), the selection of different L.MMs as genetic operators (Appendix D.7.2), and the impact of arity of genetic operations (Appendix D.7.3). Additionally, we analyze how various parent sampling mechanisms for crossover and mutation influence outcomes (Appendices D.7.4 and D.7.5), alongside an evaluation different population initialization strategies (Appendix D.7.6).
D.7.1 PROMPTING STRATEGIES
Experimental setting. We compare LLEGO to LLEGO a variant which removes the crossover operator and changes the mutation prompt to "improve the solution"-type -ttye of prompt.
28

---

Published as a conference paper at ICLR 2025
Results. We report the results in Table 14, where that LLEGO consistently outperforms LLEGOuaive- This demonstrates the importance of explicit fitmess diness-guidance via the hyperparameter a in order steer the search towards high-fitness, regions.
Table 14: Performance naive prompting.
Test balanced accuracy (f) on classification tasks
(depth d seeds), reporting (sa)
Method
Breast
Compas
Credit
Diabetes
Heart
Liver
Vehicle
L..RGoaalvv
(.000)
(0.11) 0.663
0.708
0.714
(.62))
0.945)
LUBGO
0..00)
(.721 0.684(..011)
0.676 (..52)
0.029)o.015)
D.7.2 DIFFERENT LLMS
key property of LLEGO's design is that is LLM-agnostic. To demonstrate the advantage this flexibility, evaluate LLEGO's performance using gpt-4, comparing it to gpt 3.5.. 5. We report the results in Table 15 for all the classification datasets, for depth 4 problems, across 3 seeds. We that the gpt-< -4 variant of LLEGO achieves superior performance than its gpt-3. counterpart. These results have two sigmificant implications, as they indicate that (1) LL LLEGO's effectiveness is robust across LLM architectures, and importantly that (2) its performance can scale with advances in capabilities of the underlying L.L.s.
Table 15: Performance different LLMs. Test balanced accuracy (1) on classification tasks (depth
d = 4, seeds), reporting mean
Method
Breast 0.951 (0))
Compas
Credit
Diabetes
Heart
Liver
Vehicle
1..60 (gpt-35)
0.663 o0s)
(..84
0.721 (0o01))
0.751 (.75))
(.676
0.929 0.951 o.65)
LLEGO (gpt-4)
0.957 006)
0.671
(.684
0.741 (.741(0.028)
0.751
D.7.3 ARITY OF GENETIC OPERATIONS
In Figure 5, compared the crossover dynamics between LLEGO with 4 parents and roulette wheel selection, and GATreexo with v 2 parents and uniform parent sampling. In Figure 13 (Left), we compare LLEGOxO with 2 parents and uniform parent sampling against cATreexo with v = 2 parent and uniform parent sampling. In Figure 13 (Right), we compare LLEGOXo with 4 parents and uniform parent sampling against GATreexo with parent and uniform parent sampling.
We observe similar dynamics in Figure 5, where varying enables control the population fitness and diversity. Additionally, y = 4 leads to significantly improved offspring fitness at the cost of lower diversity, highlighting the nuanced impact of higher arity on search efficiency (corroborating the ablation results in Figure 7).
11000
Pop Piaees
0.6
Pop Divesiity
Pop Firnsss
0.6
1LE00 GATTZ
LLEGO G472
LLEGC
0.4
0.4-
547R2E
Pap
0.2
0.2
0.25
-0.10
0.10
0.25
-0.25
-0.3
10
0.25
-0.25
-0.10
0.30
0.25
0.25
-0.10
0.10
0.25
Figure 13: XO dynamics. Effect fitness guidance (a+ on population and diversity using uniformly sampled parents. (Left) 2 parents, (Right) 4 parents
D.7.4 PARENT SAMPLING: CROSSOVER
The objective of this experiment is to analyze the impact of an alternative selection mechanism on the balance between population fitness and diversity in the crossover operator.
Experimental setting. Specifically, we replace the roulette wheel selection cetness-proportion mechanism with tournament selection mechanism (Miller al., 1995) with varying tournament
29

---

Published as a conference paper at ICLR 2025
sizes k (1,2,355) We then compute the median offspring fitness and diversity function of k following the experimental setup described in Section 5.2.
Observations. The results, shown in Figure 14, demonstrate clear trade-off between fitness and diversity which is modulated by the tournament size. As shown in Figure 14a, larger tournament sizes consistently yield higher population fitness, while Figure 14b shows corresponding decrease in population diversity. Indeed, larger values of intensify selection pressure by increasing the probability that highly fit individuals win multiple tournaments, thereby reducing population diversity. Conversely, smaller values of k lead to an increased population diversity, For example, when k = 1, tournament selection corresponds to random sampling, which maximizes diversity at the cost of fitness performance. In comparison to tournament selection, the roulette wheel selection mechanism employed in LLEGO achieves a good anddle-ground by striking a balance between fitness and diversity.
0.20
0.550
Touraament
Pop Fitness
0.525
Pop Diversity
0.15
Roulette wheel
0.500
0.475
0.10
0.450
0.425
Tournament Roulette wheel
0.05
0.400
Tournament size
Tournament size
(a) Offspring fitness.
(b) Population diversity.
Figure 14: Crossover dynamics with tournament selection. (a) Population fitness increases
monotonically with tournament size, demonstrating enhanced selective pressure. (b) Population diversity exhibits an inverse relationship with tournament size, with smaller tournaments preserving higher diversity the cost of reduced fitness.
D.7.5 PARENT SAMPLING: MUTATION
In this experiment, we investigate an alternative choice for the selection mechanism in LLEGO's mutation operator.
Experimental setting. We replace the random parent selection in the mutation operator with the quality-diversity algorithm CVT-MAP-Elites (Vassiliades al., 2017), which requires defining behavioral space. Given n training samples, we define the behavioral space for classification tasks as algorithm then
H. = [0, 1]"" encompassing the trees' functional signatures. The CVT-MAP-Eltn partitions H into M niches using uniformly distributed centroids found with k-means clustering. We then select parents for the mutation operator by uniformly sampling niches and selecting the best individual in the sampled niches. Finally, we compute the offspring diversity, similarly as in Section 5.2.
Observations. We report the results in Figure 15, averaged across the classification datasets. We see that the total number of niches M serves as control parameter for offspring diversity, with an increasing relationship between diversity and the number of niches M. When M = 1, the process reduces to repeatedly sampling the population's 's best individual, resulting in minimal diversity for the generated offspring. Conversely, when solutions spread into distinct niches, the sampling process becomes equivalent to uniform sampling without replacement from the population, yielding high diversity. Furthermore, see in Figure 15 that the random selection of parents employed in LLEGO comparatively yields high diversity, justifying its use in the diversity-guine guiieey-ggueddddd mutation operator.
D.7.6 POPULATION INITIALIZATION STRATEGIES
In this experiment, we investigate the impact of a different population initialization on the search performance of L.EEGO.
30

---

Published as a conference paper at ICLR 2025
Pop Divensity
Quality- Diveesityyy
Random
0.0
20
Number of niches
Figure 15: Mutation dynamics with Queeity-Diversity selection, Offssring diversity increases with the number of niches employed in CVT-MAP-Elites for parent selection.
1.
Mean Pop Fitness
0.8
0.6
0.4
0.2
mL
LLEGOmai
0.0
0
15
20
25
Generations (G)
Figure 16: Ablation the population initialization. LLEGO initializes populations using CART models trained on 25% bootstrap samples, while LLEGOmnt. 2 uses minimal training subsets of size Results are aggregated across all classification datasets for one seed.
Experimental setting. Specifically, we compare two variants of LLEGO. The baseline variant, LLEGO 1 corresponds to the instanciation of LLEGO described in Section 5, which initializes the population with CART models trained bootstrap samples comprising 25% of the training data. In contrast, LLEGO initializes trees using CART models trained on minimal random subsets of just two training samples, resulting in weaker initial decision trees.
Observations. Figure 16 illustrates the convergence of the mean population fitness across genera- tions, aggregated and normalized over all classification datasets for one random seed. The results demonstrate that LLEGO exhibits slower convergence compared to LLF.O0mi which shows the role of effective population initialization in improving search efficiency and faster discovery of high-quality solutions. Nevertheless, we remark that LLEGO 2 still achieves good performance in the later stages of the search (after G 20 generations), showing the effectiveness of LLEGO's variation operators in steering the search towards promising regions, independent of the initialization scheme.
D.8 FINE-GRAINED RESULTS
In this subsection, present detailed analysis of the results from the main paper. Our examination includes: (1) mutation dynamics observed in individual tasks; (2) convergence analysis across varying depths, including ablation studies; and (3) convergence trajectories for specific tasks.
Mutation dynamics. We provide the mutation dynamics for each individual dataset in Figure 17, showing that meaningfully controls the diversity in the population for 5 of the classification datasets, where the diversity metrics are computed between parents and offspring (Top) and among the offspring (Bottom).
31

---

Published as a conference paper at ICLR 2025
Oage
Figure 17:
MUT dynamics. Effect of diversity guidance (7) on (Top) median parent-offspring distance and (Bottom) median offspring distance.
Convergence analysis. We provide separate convergence plots in this subsection, obtained when optimizing trees of depths 3 and 4, under the experimental setup described in Section 5.1. The results reported in Figure 18a and Figure 18b, In these settings, LLEGO leads more efficient search compared to GATree. This improved efficiency also comes with reduced diversity, showing that LLEGO concentrates its populations in the later generations in high-fitness regions.
Media Pop Fitess
Mis Pop Fiteess
Medion Pop Divasily
Medon Pep Fitoos
Siax Pup Flioes
Nodian Plo Divenily
11000
LLaG2
LLEGO
21350
GATTEE
simee
065
13
Genmitton
Gemmitioo
(a) Depth = 3.
(b) Depth 4.
Figure 18: Convergence dynamics. Comparing LLEGO with GATREE.
Ablation study. We report the ablation study results for depth 3 and in Figure 19 and Figure 20. These results align with the observations made in Section 5.3, highlighting the importance of using crossover and mutation in tandem, the importance of incorporating more than 2 parents for the operators and using semantic information. With a higher maximum depth, the space of possible trees becomes more complex, and accentuates the need for both exploration and exploitation, which explains why the mutation only (ILEGOmo. xo) and Crossover only (LLEGOno_ma) baselines perform worse than LLEGO.
0.70
Median Pop Fitncss
0.70
0.2
Max Pop Fitnoss
0.00
0.58
0.00
0.95
L1E03
LLEGO LLBOO
0.00
15
Generations (G)
Generations (G'
(a) Median population fitness.
(b) Max population fitness.
Figure 19: Additional ablation results. Depth 3.
Individual task results. Convergence plots comparing LLEGO and GATree for individual tasks are given Figure 21 and Figure 22. They show that LLEGO consistently leads better search efficiency compared to GAtrec.
32

---

Published as a conference paper at ICLR 2025
0.73
0.00
Median Pop Fitbess
0.00
Max Pop Fitncee
0.00
0.89
0.03
0.00
E1R20
0.82
0.40
LLB
L1B
CL000
0.78
16
Generations (C)
Generations (G)
(a) Median population fitness.
(b) Max population fitness.
Figure 20: Additional ablation results. Depth 4.
bocast
cocupas
credit- gg
14300
11820
11420)
G4m2z
547222
06:722
8:00
0.00
0,000
0.03
6.0
0.03
Giabelus
Instri-siaatoo
600
0.15
0.50
6.00
0.74
0.54
Mean
0.72
0.50
0.00
0.20
Liver
velicce
LLEGO
11E50
GATREZ
0.00
0,00
8.00
Generatione
(G)
Figure 21: Convergence plots. Mean population fitness (f) of LLEGO and CATREE individual tasks across 25 generations (depth=3).
33

---

Published as a conference paper at ICLR 2025
bovsel
0.5
6.70
0.00
1000
diobetes
heart-starloog
0.08
OATREL
0.76
0.70
0.74
0.72
0.20
0.00
Vehiele
8.94
11802
11800
0.72
ONTRES
0.54
ONTREE
0.70
8.00
0.00
Meon
0.00
061
180
20
(G)
Figure 22: Convergence plots. Mean population fitness (f) of LLEGO and GATREE on individual tasks across 25 generations (depth=4).
34


Reviews:
[{"review_id": "gLuEjMq0jv", "reviewer": "Reviewer_v13n", "summary": "This paper presents an application of LLMs as a crossover and mutation operator in an evolutionary algorithm. The authors propose “LLM-Enhanced Genetic Operators” (LLEGO), which uses fitness guided crossover and diversity guided mutation to infer decision trees from data. The proposed method (LLEGO) is compared against other decision tree induction algorithms on classification and regression tasks.", "questions": "- The choice of using natural language as the representation for the evolutionary algorithm is fitting  as they need to be used by an LLM-based genetic operation. Do you imagine there being any ways to improve the syntactic validity from ~86% of the decision trees being outputted by your genetic operators?\n- Is there any reason (beyond representation in natural language) that this method could not be used for symbolic regression or more general function induction?\n- How well would the method work when starting without initializing with CART models on 25% of the training data?", "limitations": "", "rating": 8, "confidence": 4, "soundness": 4, "presentation": 4, "contribution": 3, "strengths": "- The paper is very well written and easy to follow. The motivations for the proposed methods are detailed well.\n- The experiments are well detailed, fitting, and the proposed approach is compared against a good number of baselines. There were very rigorous additional results, addressing issues such as fairness and generalization. Upon the first read, I was concerned that the results might be highly conditional on the solutions existing in the training set of the LLMs, but Appendix D.2 settled my concern.\n- The use of LLMs in the loop of an evolutionary algorithm for decision tree induction is, to my knowledge, novel and, in my opinion, very promising.", "weaknesses": "- I would have been very interested in seeing a different use of selection algorithms, as fitness-proportionate (roulette wheel) selection is somewhat outdated and known to result in low population diversity. For example, perhaps the use of tournament or lexicase selection could adequately balance fitness and diversity?\n- You use mutation as purely a technique to inject diversity into the population. I would have been interesting in seeing how “fitness-guided” mutation would perform (i.e. remove XO, and prompt the LLM to generate a better version) as a baseline. This would help motivate the use of XO/mutation as a means to explore/explore, respectively.\n\nsmall typo:   \n- At the beginning of section 3 - “buid” should be “build”", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "MK2bzNwYgV", "reviewer": "Reviewer_71f5", "summary": "The paper proposes an innovative method for decision tree induction using genetic programming (GP) enhanced by semantic knowledge from large language models (LLMs). The key contribution is the LLEGO framework, which introduces semantically-aware genetic operators: fitness-guided crossover and diversity-guided mutation. These operators leverage semantic priors from LLMs to improve search efficiency and decision tree generalisation performance. The method is empirically validated on various tabular datasets, demonstrating superior performance compared to traditional and state-of-the-art tree induction methods.", "questions": "* How does the computational overhead of using LLMs in LLEGO compare to traditional GP approaches in terms of runtime and memory usage? Are there any scenarios where the increased cost might not be justified? Provide a more detailed analysis of the computational cost and scenarios where the trade-off is most beneficial. Consider discussing potential optimisations or the use of smaller LLMs.\n\n* How does the performance of LLEGO change if the underlying LLM lacks semantic priors relevant to a particular problem domain? Are there strategies to mitigate performance degradation in such cases?\n\n* Include a theoretical explanation of how semantic priors impact search efficiency and decision tree quality. This would strengthen the paper’s scientific contributions.\n\n* Do the authors have any preliminary insights or results on how well LLEGO could be applied to other GP-based problems, such as symbolic regression or evolutionary program synthesis? Test how LLEGO might perform in other GP-based problems, such as symbolic regression or program synthesis, to show the method’s broader applicability.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "* The integration of LLMs into GP is a significant advancement. Using LLMs to inform genetic operators with semantic priors is a creative and impactful idea, enhancing both search efficiency and the quality of generated decision trees.\n* The paper provides a detailed explanation of the LLEGO framework, including how LLMs are utilised and how genetic operators are implemented. The end-to-end algorithm is clearly described, making it easier to understand the overall workflow.\n* The paper addresses the limitations of existing decision tree induction methods by offering a scalable and generalisable approach, which could be beneficial across various domains.", "weaknesses": "* The effectiveness of LLEGO depends heavily on the semantic priors embedded in the LLM. If the LLM lacks relevant knowledge for a particular domain or problem, the performance might degrade. A discussion on this limitation and possible solutions would be helpful.\n* While the empirical results are strong, the paper lacks a rigorous theoretical analysis of why and how the semantic priors from LLMs improve search efficiency. This could strengthen the understanding of the method’s underlying principles.\n* The paper focuses on decision tree induction, but it is unclear how well the approach generalises to other GP-based tasks. A brief discussion or some preliminary results in this direction would make the contribution more comprehensive.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "wb1QpcoRVt", "reviewer": "Reviewer_mCct", "summary": "In this paper, the authors propose using LLMs as evolutionary operators to evolve decision trees.", "questions": "Here are several questions that need to be addressed:\n1. The most important issue in this paper is the claim that existing optimal decision tree methods are computationally complex, with complexity scaling exponentially with depth and the number of possible splits in the dataset. However, in the experimental results, the authors show their method with depths of 3 and 4, which can actually be solved by optimal decision tree induction methods, such as DL8.5 [1]. As shown in the DL8.5 paper, with a depth limit of 4, DL8.5 can find the optimal decision tree in 19 out of 24 cases within 10 minutes. It is unclear why the authors chose to use LLMs, an expensive method, to address this issue.\n\n2. The idea of using large language models for evolving programs with diversity considerations is not novel. For example, the GECCO 2024 paper \"LLMatic: Neural Architecture Search via Large Language Models and Quality Diversity Optimization\" [2] already considers this. Also, using random selection for parent selection in diversity-guided mutation is insufficient. Please consider quality-diversity optimization for parent selection.\n\n3. Please report the training accuracy as well. It is hard to understand why the proposed method would outperform optimal decision trees within depths of 3 and 4.\n\n4. The proposed method is tested on GPT-3.5. Are the conclusions applicable to other models, such as GPT-4 or Claude? Please conduct experiments to verify this. Analyzing the sensitivity of the proposed method to different LLMs is also important.\n\n5. The paper claims one advantage of the proposed method is using LLMs to apply semantic information to guide the search. However, the conclusion of Table 7 shows that LLMs work well even when semantics are removed. The authors should reconsider why LLMs perform well in this context.\n\n6. The term \"semantic information\" seems more like \"domain knowledge\" rather than the commonly understood meaning of semantics in the genetic programming domain. Please consider changing the term or at least add an explanation to clarify the difference. Otherwise, it could be confusing.\n\nReferences:\n\n[1]. Aglin, Gaël, Siegfried Nijssen, and Pierre Schaus. \"Learning optimal decision trees using caching branch-and-bound search.\" Proceedings of the AAAI Conference on Artificial Intelligence. Vol. 34. No. 04. 2020.\n\n[2]. Nasir, Muhammad Umair, et al. \"LLMatic: Neural Architecture Search via Large Language Models and Quality Diversity Optimization.\" Proceedings of the Genetic and Evolutionary Computation Conference. 2024.", "limitations": "", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "The strength of this paper is that it explores decision tree induction in the era of large language models and shows promising results compared to traditional evolutionary algorithms and optimal decision tree induction methods.", "weaknesses": "The weakness of this paper is that the motivation for using LLMs for decision tree induction is not well justified. It is unclear why LLMs would be helpful for decision tree induction, especially considering that LLMs can still function without semantic priors, as shown in Appendix D.2.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "WIDUteWMva", "reviewer": "Reviewer_BF7z", "summary": "The paper introduces a novel framework LLEGO that leverages LLMs to enhance genetic programming (GP) for decision tree induction. LLEGO incorporates semantic priors and domain knowledge into the optimization process through LLM-based fitness-guided crossover and diversity-guided mutation operators. The framework represents decision trees in natural language, enabling the use of broader contexts and higher arity operations. Empirical results on various benchmarks demonstrate LLEGO's superior performance over existing tree induction methods in terms of search efficiency and the quality of the evolved decision trees.", "questions": "- Can LLEGO be extended to other string optimization problems, or is it currently too specialized for the Decision Tree Induction benchmarks it addresses?\n- What does \"LLEGO_no_prior\" mean? Have you compared LLEGO to vanilla LLM evolution without semantics guidance in crossover or diversity guidance in mutation?\n- Recent work has incorporated semantic priors generated by LLM reflections into the evolutionary process [1]. A discussion of how this approach relates to LLEGO would be beneficial.\n\n[1] ReEvo: Large Language Models as Hyper-Heuristics with Reflective Evolution, NeurIPS 2024", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- Combining LLM and EA for decision tree induction is novel.\n- The introduction of fitness-guided crossover and diversity-guided mutation is well-motivated.\n- Strong empirical performance compared with the conventional GP method.\n- The paper is well-written and easy to follow, and it is accompanied by good visualization.", "weaknesses": "- **Crossover**: Is fitness guidance truly necessary? Could we instead prompt with \"improve on the above decision trees\" and still achieve comparable performance? Have you conducted any ablation studies on this?\n\n- **Mutation**:\n  - Are the log probabilities of within-population solutions consistently higher than those of out-of-population solutions? Have you verified this in your experiments?\n  - According to your experiments, LLEGO sacrifices population diversity compared to traditional GP due to its semantic priors. Could you implement mutation alone to preserve population diversity? If not, does this suggest that the designed diversity-guided mutation operator fails to maintain sufficient diversity?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)