Partition to Evolve: Niching-enhanced Evolution with LLMs for Automated Algorithm Discovery
Authors: Qinglong Hu
Qingfu Zhang
Venue: NeurIPS 2025 poster ()
Keywords: Automated algorithm discovery
Evolutionary computation
Large language model
Primary area: applications
TL;DR: PartEvo introduces feature-assisted niche construction in abstract language spaces, allowing the integration of niche-based search strategies into LLM-assisted evolutionary search, ultimately offering more efficient automated algorithm discovery.

Abstract: Large language model-assisted Evolutionary Search (LES) has emerged as a promising approach for Automated Algorithm Discovery (AAD). While many evolutionary search strategies have been developed for classic optimization problems, LES operates in abstract language spaces, presenting unique challenges for applying these strategies effectively. To address this, we propose a general LES framework that incorporates feature-assisted niche construction within abstract search spaces, enabling the seamless integration of niche-based search strategies from evolutionary computation. Building on this framework, we introduce PartEvo, an LES method that combines niche collaborative search and advanced prompting strategies to improve algorithm discovery efficiency. Experiments on both synthetic and real-world optimization problems show that PartEvo outperforms human-designed baselines and surpasses prior LES methods, such as Eoh and Funsearch. In particular, on resource scheduling tasks, PartEvo generates meta-heuristics with low design costs, achieving up to 90.1\% performance improvement over widely-used baseline algorithms, highlighting its potential for real-world applications.

Partition to Evolve: Niching-enhanced Evolution with LLMs for Automated Algorithm Discovery
Qinglong Hu
Qingfu Zhang
Department of Computer Science City University of Hong Kong
Department of Computer Science
City University of Hong Kong
Hong Kong, China qinglhu2-c@my cityu. edu.
Hong Kong, China qingfu. serffu.zaan@@ittyp ter..eiittueee edu hk
Abstract
Large language model-assisted Evolutionary Search (LES) has emerged a promis- ing approach for Automated Algorithm Discovery (AAD). While many evolution- ary search strategies have been developed for classic optimization problems, LES operates in abstract language spaces, presenting unique challenges for applying these strategies effectively. To address this, we propose general LES framework that incorporates feature- assisted niche construction within abstract search spaces, enabling the seamless integration of niche-based search strategies from evolution- ary computation. Building on this framework, we introduce PartEvo (Partition to Evolve), an LES method that combines niche collaborative search and advanced prompting strategies to improve algorithm discovery efficiency. Experiments on both synthetic and real-world optimization problems show that PartEvo outper- forms human-designed baselines and surpasses prior LES methods. In particulan on resource scheduling tasks, PartEvo generates meta-heuristics with low design widely-used baseline
costs, achieving up 90.1% 1% performance improvement algorithms, highlighting its potential for real-world applications.
Introduction
Algorithms are essential for solving real-world problems. As demands across domains become increasingly diverse and complex, Automated Algorithm Discovery (AAD) has emerged as a critical approach to improving the efficiency of algorithm development []] Techniques such as genetic programming, auto-regressive models, and reinforcement learning have successfully automated tasks like algorithm configuration and composition 2 However, the automated generation of more sophisticated algorithms remains ongoing pursuit.
Recent advancements in Large Language Models (LLMs) have provided transformative tools for algorithm generation, owing their impressive capabilities in natural language understanding and code generation [5]. Among these, the LLs-assisted Evolutionary Search (LES) paradigm has shown great promise in AAD tasks. LES represents search objects (e.g., algorithms, code, heuristics) as individuals in an evolutionary framework, where LLMs, guided by specialized prompts, act as evolutionary operators to improve these individuals iteratively 96. While carly LES methods have demonstrated the fcasibility and potential of LES in AAD, they often rely on oversimplified search mechanisms (e-g., greedy selection), which limit their efficiency.
In Evolutionary Computation (EC), search efficiency is enhanced through better exploration- exploitation trade-offs, which can be achieved via computational resource allocation techniques such as niching [10] and search space partitioning (Τ.. It is natural to consider adapting these established EC techniques to the LES to enhance algorithm discovery. However, applying them to LES poses new challenges due to the fundamental shift the nature of the search space.
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

With the integration of LLMs, the search space extends beyond traditional numerical or manually designed discrete spaces into language spaces [[2]. Unlike traditional search spaces, language spaces lack explicit dimensionality and well-defined structures. Instead, they are implicitly shaped by the interaction between the I.IMss and the specific task context. This abstraction complicates the adoption of advanced EC techniques. For example, in numerical domains, niches can often be defined using distance thresholds [3]. However, in language spaces, it is inherently challenging to compute distances between algorithms, which hinders the application of niche-based EC techniques.
To address these challenges, we present a practical pipeline for partitioning language search spaces and constructing niches, integrated into a general LES framework to improve search efficiency. This framework facilitates the seamless incorporation of niche-based EC techniques into LES, enabling more effective allocation of sampling resourees (i.c.. queries to LLMs) during the search process. Critically, it also establishes a methodological blueprint for incorporating a diverse range of advanced EC methods into future LES pipelines. Building on this foundation, we propose PartEvo, an LES method that combines advanced prompting strategies with effective EC techniques. PartEvo significantly improves search efficiency and excels in AAD tasks, particularly under limited sampling budgets. Our contributions can be summarized as follows:
A general LES framework for integrating advanced EC techniques: We propose general LES framework that incorporates assiste abstract search space partitioning, enabling structured (non-random) niche construction for the algorithm discovery process. This framework allows LES to integrate niche-based EC techniques, enhancing the efficiency of sampling resource allocation in language search spaces.
We develop PartEvo, a novel LES method that combines verbal gradients with both local and global search strategies to discover high-performing algorithms. PartEvo exemplifies the seamless integration of EC techniques with LES, demonstrating significant performance and efficiency gains.
PartEvo development:
We evaluate PartEvo on AAD tasks over both synthetic and real-world optimization problems. It consistently outperforms human-designed meta-heuristic baselines and achieves substantial gains in algorithm discovery efficiency. Furthermore, extensive ablation studies validate the effectiveness of feature-assisted niche construction, advanced prompting strategies, and the integration of advanced EC techniques into LES.
Comprehensive evaluation:
2 Related works
Automated algorithm discovery Automatic algorithm discovery automates the configuration, bination, or generation of algorithms tailored to specific problems, yielding significant improvements in efficiency and scalability [14 15). It has become possible automatically tune hyperparameters
or combine different algorithmic components [16] [7]. Genetic programming [18 [9] provides an interpretable approach to algorithm design. Significant efforts have also been made incorporate machine learning techniques 220 3 into the automatic algorithm design process. More recently, pre-trained LLMs, with their extensive knowledge repositories, have shown considerable promise in the field of automated algorithm discovery [23] 24]. However, the full potential of LLMs for effective algorithm discovery remains largely untapped current LES approaches.
LLM-assisted evolutionary search EC is powerful optimization paradigm inspired by the principles of natural evolution 225 26]. The integration of LLMs into EC has led to significant
advancements in areas such as code generation (27 28 and text generation [29 30). Moreover, the combination of EC and LLMs, particularly through prompt engineering, has shown remarkable potential across various domains, including scientific discovery 66 [1], algorithmic component
design [32 33 reward function optimization (9 34], and neural architecture search [35]. Recent studies have also investigated specialized individual encoding strategies 7 and verbal gradients 8 to further elicit the contextual understanding capabilities of L.LMs during iterative evolutionary processes. Building on these developments, this work introduces general framework incorporating niche-based EC techniques to improve the search efficiency of LES.
Niching principles Niching is widely used strategy in optimization, designed to improve efficiency by dividing the search space into smaller, more manageable subregions [36] 37]. It has shown strong
2

---

benefits in black-box optimization [1]] and Bayesian optimization [8 In AutoML, partitioning improves the construction of machine learning pipelines by reducing search time [3 In numerical optimization, niche-based methods are often employed to avoid convergence to local optima in complex problems [10 40). Inspired by these successes, we seek to expand the concept of niching to language search spaces, integrating it into LES- algorithm discovery. This enables more structured and controllable allocation of sampling resources, leading to improved search efficiency.
3 Niching-enhanced LLM-assisted evolutionary search
3.1 General framework
In the LES process, the language search space is inherently defined when using LLM for specific task. A single search step involves selecting parent algorithms, embedding them into few-shot prompts, and querying the LLM to generate new candidate algorithms. This can be viewed sampling operation within the language search space. Thus, the LES process can be interpreted as iterative sampling in this abstract space facilitated by the LLM. Our goal is to partition this space into subregions (i.c., niches), enabling a structured allocation of sampling resources across the entire space that strikes a balance between exploration and exploitation, ultimately enhancing search efficiency.
key challenge arises from the lack of explicit representations of language search spaces. Unlike numerical spaces, where niches can be defined via geometric or distance- based thresholds, language spaces do not have clear dimensionality or structure. Each sampled point corresponds to a complete text specific semantic expression, with no numerically defined "distance" between points. This implicit nature complicates the analysis and partitioning, limiting the application of evolutionary strategies that depend on structured subspaces for efficient searching.
We address this challenge by recognizing that each sampled point originates from this abstract search space. By analyzing the sampling results, we can indirectly infer and partition the original language space. To achieve this, we propose a feature-assisted partitioning pipeline that allows for non-random niche construction, making it possible to incorporate tiche-based EC techniques into LES.
In this framework, sampled individuals are projected into an interpretable feature space, where clustering techniques are applied to indirectly partition the language search space, Once partitioned, effective search strategies are designed allocate sampling resources across niches. The key components of the framework are as follows:
Generate an initial population candidate algorithms. This population can be derived from an existing database or dynamically generated during the search process.
Population initialization:
2. Feature space projection: Project individuals to a feature space using a chosen representation. This step connects the abstract language space with computationally manageable space.
3. Niche construction: Apply clustering methods in the feature space to group individuals based on similarity, thereby forming niches that serve as indirect partitions of the language search space.
4. Collaborative search: Perform iterative search using both prompt-centric operators and EC- inspired operators, continuing until the target problem is solved. These operators enhance the efficiency of algorithm discovery in complementary ways:
Prompt-centric operators exploit the contextual understanding capabilities of LLMs to guide the sampling process at the language level. By enriching few-shot prompts with additional information (such as reflections on past samples or summaries of search trajectories), these operators steer the LLM toward generating more promising candidate algorithms. Existing LES methods primarily focus on this strategy 7 g].
EC-inspired operators emulate established evolutionary mechanisms to maintain diversity and ensure global search capabilities. These operators use the niche structure to select parent individuals and apply evolutionary strategies both within and across niches. This design enables efficient resource allocation and promotes the discovery of diverse, high-quality algorithms. The integration of niche-aware EC operators core contribution of this work.
Importantly, the proposed framework is highly flexible and extensible. It allows integration with existing LES methods or mature EC techniques to enhance intra-subregion sampling, refine LLM prompt engineering, and improve resource allocation across subregions. Additionally, any feature
3

---

and clustering method can be utilized. In this work, we introduce PartEvo, specific implementation that demonstrates the effectiveness of search space partitioning automated algorithm discovery.
3.2 PartEvo
We instantiate the general framework with Partition to Evolve (PartEvo), an LES method that integrates niche-based collaborative search with advanced prompt design for automated algorithm discovery. Given a problem specification as input, PartEvo outputs high-quality algorithms tailored to the task. The overall architecture of PartEvo is shown in Fig. with pseudocode and concrete example provided Appendix
Reflection-bawee evalution
Crosuver betw ben niches
Filuee- benel-
Fliness- base
Selection
Nicho Salection
& Geteeratooy
Knnneledge
Rersower
Selection
Filuess beardl
Feature Space
Initialization
Reflection
Generator
Genterotor
Summary-based evolution
Local and global guided evolution
Feulure-asisted niche
Sorted External Archive
Niche Best Solation
Summary
Global Best Solution
Nithe-hased evalution
Surmtaatizr
Simess hased
Feature Space
Population management
Selection
Generetor
Geberator
Niche
A Igorithms
Niche
Niche
Figure I: Overview of the PartEvo framework. The left section illustrates the ideal distribution and clustering results of the algorithms within the feature space. The central part highlights PartEvo's main workflow. The right section provides a detailed illustration of PartEvo's evolutionary operators.
Individual representation. PartEvo performs algorithm discovery through an evolutionary process, individual. Each individual consists of both code
representing each candidate algorithm and its corresponding "thoughts", following the design introduced representation is adopted for two reasons: it leverages the LLM's strong language understanding and broad knowledge base, and it enables feature projection from both the algorithm's 's code and its natural language rationale. This flexibility beneficial for subsequent feature-assisted niche construction. Concrete examples of individuals are provided in Appendix H
EoH [7]. This dual-format
Population initialization. PartEvo initializes individuals in batches, iteratively constructing the initial population to reach a target size N. During the generation of each batch, the "thoughts" " of all previously generated individuals (if any) embedded into the few-shot prompt guide the LLM, helping avoid redundant outputs. This approach encourages diversity and promotes broad coverage of the search space. In addition, PartEvo maintains external archive of individuals, ranked by performance, with a fixed size E. This archive stores search trajectories and provides valuable contextual information that helps the LLM develop more refined understanding of the search space.
Feature-assisted niche construction PartEvo employs K K-means clustering [1] to group initial individuals based on their feature representations, thereby constructing K niches. The number of niches acts as a control knob, allowing users to adjust the granularity of partitioning. For fixed population size N, different values of K correspond to varying degrees of sampling resource concentration, thus balancing exploration and exploitation in the search space. We explore two feature representations as illustrative examples:
1.
Code Similarity Vector: Each individual is represented by vector capturing its similarity to all others in the population. This captures global relationships between individuals, ensuring that similar individuals are placed close to each other.
Thought Embedding: The "thoughts" of individuals are embedded into a vector space using word embeddings, providing semantic representation of their design rationale.

---

We adopt K K-means to present our main experimental results due to its simplicity and transparency, ensuring that observed performance improvements primarily reflect the effects of niche-based evo- lution rather than complex partitioning. Additional experiments with other clustering methods are provided in Appendix B.S Further details on feature projection are provided in Appendix E
Evolutionary operators. As illustrated in the right section of Fig. I PartEvo employs two types of operators within each niche pote--prooooottttiitiee and EC-inspired operators operators-detalledd as follows:
(I) Reflection- Evolution (RE): A subset of m individuals is selected, and the Reviewer generates reflective feedback for each. These individuals and their corresponding feedback are provided as few- shot prompts to the Generator, which produces m new individuals. This operator drives improvement by guiding individuals reflect on and revise their past designs.
(II) Summary-bart basse Evolution (SE): The Summarizer analyzes the external archive to assess the overall algorithmic landscape, identifying both promising and unproductive subregions. The resulting summary, combined with selected subset of m individuals, used to construct few-shot prompts that guide the Generator in producing m new individuals. This operator facilitates informed exploration by incorporating global knowledge of the search space. draws inspiration from classical ideas such as information sharing and tabu search 42].
(III) Crossover between Niches (CN): niches are randomly selected, and one representative in- dividual is drawn from each. These k individuals are paired with m individuals from the current niche to form m pairs. Each pair then prompts the Generator to produce new individual, promoting cross-niche knowledge transfer and enhancing population diversity.
(IV) Local and Global guided Evolution (LGE): Each individual is paired with both the best- performing individual in its niche and the global best in the population. These triplets are then formatted as few-shot prompts to generate new individuals. This design ensures that all individuals benefit from guidance toward perefhrmfoniig regions, thus promoting faster convergence.
Operators RE and SE are classified prompi-centric: they employ auxiliary LLMs to provide "verbal gradients" that steer generation semantically meaningful directions. In contrast, CN and [GE are EC-inspired: they incorporate the principles of niche collaboration and evolutionary heuristics to improve search efficiency. Detailed prompts for these operators are provided in Appendix
Resource allocation strategy In PartEvo, sampling resources (e.g., queries to LLMs) are distributed evenly across niches. Within each niche, a finer- aner-grained resource allocation is implemented through piiteerreerring probabilistic selection. The best individual in each niche is always chosen parent, while the remaining parents (when m>>1) are selected probabilistically based on their performance. This strategy not only encourages the exploitation of figh-performing candidates within each niche but also ensures the exploration less-explored subregions. Unlike greedy parent selection, which concentrates resources solely on the rest-performing individuals, PartEvo's 's niche-based evolution allows dynamic control over resource allocation, effectively balancing exploration and exploitation. This is also the rationale behind the name Partition to Evolve.
Population management. In each iteration, the niches independently execute the four operators, collectively generating N+3+m* K individuals. Each niche selects individuals using replacement roulette wheel mechanism, ensuring the total population size remains constant.
Experiments
This section presents PartEvo's performance algorithm discovery tasks. Additionally, conduct thorough ablation studies to assess the contributions of niching, feature-based niche construction, and the proposed operators. The results demonstrate that PartEvo, which integrates advanced prompting strategies with niche-based EC techniques, achieves significant improvements in search efficiency.
4.1 Experimental settings
using four benchmarks. each case, the LES method tasked with generating algorithms to solve the given problem. With the same sample budget, the effectiveness of the LES method is proportional to the
Benchmarks. We assess PartEvo's ability to design meta-heuristic algorithms 43
5

---

quality of the solutions produced by the algorithms it designs. A brief overview of each benchmark is provided below, with detailed descriptions in Appendix G
(I) Unimodal optimization problems (P1): This benchmark assesses the LES method's ability to design algorithms for problems with single global optimum and no local optima. It evaluates the efficiency of the generated search strategies in simple optimization landscapes.
(II) Multimodal optimization problems (P2): This benchmark focuses on problems with multiple local optima, aiming to evaluate whether the LES-designed algorithms can avoid premature convergence and maintain sufficient exploration of the search space.
(III) Task offloading in mobile edge computing systems (P3): This real-world benchmark involves solving task offloading problems under constraints such as execution time, energy consumption, and bandwidth [44. evaluates the LES method's ability to generate algorithms for mixed-integer nonlinear programming (MINLP) problems with multiple real-world constraints.
(IV) Machine-level scheduling for heterogeneous plants (P4): This problem requires designing algorithms to handle scheduling tasks across machines with different capabilities and managing transportation of products 45 tests the LES method's ability to design algorithms that solve high-dimensional nonlinear integer programming problems subject to complex industrial constraints.
Baselines. We compare PartEvo against three LES methods and three human demanndesigned meta- heuristic algorithms. The peer LES methods include Funsearch, EoH, and ReEvo.
Funsearch [6] uses island-based evolutionary approach iteratively improve function quality, with each island acting as a niche formed through random grouping. Our comparison with Funsearch highlights that crature-assisted niche construction effectively balances exploration and exploitation.
explotta
EoH models algorithm discovery as a search problem in the dual space algorithms, rep- resented both as code and as thought. Direct comparison with EoH demonstrates the superior search efficiency of PartEvo, which is attributed to the integration of advanced prompt design and niche-bases EC techniques.
ReEvo 8 introduces "verbal gradients" into the LES paradigm via genetic cues in natural language form, enabling more focused searches within the language- based search space. A direct comparison with ReEvo highlights the superior search efficiency achieved through the integration of advanced EC techniques.
The human-designed meta-heuristic baselines include enhanced variants based on Genetic Algorithm, Differential Evolution, and Particle Swarm Optimization [46H48]. For simplicity, we refer to them as GA, DE, and PSO in the experimental settings, with all variants tuned to provide strong human- designed baselines. A more detailed description of these algorithms is provided in Appendix F By comparing PartEvo against numan-designed baselines, we provide compelling evidence of its effectiveness in designing algorithms for both synthetic and real-world optimization problems. Such comparisons essential to assess whether LES methods can surpass established human expertise and offer practical value as automated design tools.
Implementation details. We frame all benchmarks as black-box optimization tasks and provide them to LES methods. Each LES method receives only performance feedback for the algorithms it generates, without access to the mathematical models or instance-specifee details. This prevents the LLM from exploiting explicit knowledge of the problem to overfit or cheat. Each benchmark includes multiple instances, split into training and testing sets. LES methods develop algorithms on the training instances, and their performance is evaluated on the testing set. This setup allows us to assess whether the LES methods overfit training instances, thereby mitigating potential biases arising from fixed-pattern solutions that could inflate performance.
For all LES methods, the number of candidate algorithms sampled by LLMs is set to 500. Each benchmark is independently solved over four runs to reduce the effect of randomness. All LES methods utilize the pre-trained (PT-4o-min model. The resulting meta- heut-shtisss are constrained to 30,000 candidate solution evaluations and maximum runtime of 180 seconds. EoH and PartEvo both use a population size of 16. Funsearch, EoH, and ReEvo follow their respective default configurations provided by the L.MMADD platform [49] Unless otherwise explicitly stated, all results reported for PartEvo in this paper utilize Code Similarity the feature for its niche construction process.

---

4.2 Comparative evaluations
We evaluate the ability of PartEvo, Funsearch, EoH, and ReEvo to design meta-heuristic algorithms for identical instances. The generated algorithms are also compared against human- desiandd baselines. Table Table[]report reports the best results across four runs on training and testing sets, Lower objective values indicate better performance, reflecting how efficiently each designed meta-heuristic utilizes the fixed budget of 30,000 evaluations. For reference, the Optimal column lists the known optima, representing the minimum achievable objective values for each benchmark. Additional statistics, including mean and standard error of the mean across runs, are provided in Table 2 and illustrated in Fig.
Table 1: Best results from multiple runs on training and testing instances for each benchmark
Benchmark
Instances
GA
PSO
Funseareh
EoH
ReEvo
PartEvo
Optimal
Training
20.293
15.569
0.052
0.418
0,000
0.000
0.000
0,000
PI
Testing
x.6
0.000
0.000)
0.604
0.000
2.253
0.000
0,000
Training
1772.61
890.99
881.44
800.25
800,00
801.32
800,00
800,00
P2
Testing
585.58
421.76
386.50
376.36
462.08
375.57
344.62
310.00
P3
Training Testing
31447.05
8805,00 09
6541.56
10503:67
7003.53
6589.08
6471.31
75279.85
58704.47
60892.12
58327.20
59061.28
56876.40
Training
5.29
30980.3
25088.2
20425.0
3697.6
10630.3
2792.1
P4
Testing
x14
x.37x10'
1.49x10*
34891.9
x.34x
14396.1
As shown in Table T PartEvo consistently outperforms both peer LES methods and human- designed baselines across all benchmarks. On complex problems such as P3 and P4, it demonstrates clear advantage over Funsearch, EoH, and ReEvo. Notably, on P3, it the only LES method that surpasses the DE baseline. On the most challenging task, P4, it achieves 90.1 reduction in manufacturing cost compared to the best-performing DE variant.
Table 2: Average results and standard error of the mean from multiple runs on training instances
Benchmark
Funsearch
EoH
ReEvo
PvrtEvo
PI
6.30(+2.111)
(.43(+1.40)
1.64(+1.113
0.00(++0000)
P2
810.88(+5.26))
800.36(10..3))
804.52(#1.60)
800.01(10.01)
P3
2301.9(4+221.60))
7074.5(#
6583.8( 77..74)
P4
30094.6(+3385.37) 17679.9(+3082.57)
15345.5(#2381.9)
5171.6(+2189.14)
Table presents the mean and standard еrror of the mean (SEM) of results across multiple runs on training instances, complementing the data shown in Table PartEvo consistently delivers the best average performance with relatively small SEM values, indicating strong algorithm design and stability. Overall, PartEvo outperforms peer methods across all four benchmarks.
Fig. illustrates the convergence behavior of the best objective values for PartEvo, Funsearch, EoH, and ReEvo across four runs, showing their performance over 500 samples, which reflects the algorithm design efficiency. Colored curves show mean performance across runs, and shaded areas denote variance. Dashed horizontal lines mark numan-designed baselines. PartEvo not only reaches superior solutions with fewer samples but also maintains the smallest variance, indicating strong robustness and consistency. Compared to Funsearch, which also uses subpopulation, PartEvo continues to improve throughout the search, suggesting feature-assisted niche construction more effectively avoids local optima. When compared to ReEvo, which uses the concept of verbal gradient but relies a basic EC framework. PartEvo shows higher search efficiency. This highlights that advanced EC techniques can significantly enhance search performance and should be given equal emphasis prompt engineering in the design of the LES paradigm.
In summary, PartEvo achieves the fastest convergence, the lowest final objective values, and the highest stability across runs. These results demonstrate the effectiveness of combining based evolutionary search with LLM guidance for domain-specific algorithm discovery. To control for the influence of initialization, additional experiments using identical initial populations are conducted (see Appendix B.2). Further results on additional benchmarks are provided Appendix B.3 Comparative evaluations of these LES methods with 2000-sample budget presented in Appendix B.6
7

---

PSD-vadari
GA-valsut
100
GA vatarr
DE-vatani
Olpecive Furddia
Fusaouth
Ealt
Fundigg
880
Hunzenth
Refiro
PartEvo
640
820
:00
200
500
330
600
300
Sampling
Sampling
(a) P1
(b) P2
-10*
PSO varann
11000
DE-vomtent
Objectre Furdios
10030
Eoit
Function
Futseerch
EoH
+Rebro
8030
7030
100
320
400
500
Sampling Count
Sampling Count
(c) P3
(d) P4
Figure 2: Convergence LES methods across four runs on benchmarks (PI-P4).
4.3
Sampling similarity analysis
We assess PartEvo's 's ability to balance exploration and exploitation by analyzing the similarity of its sampled results. T reports the similarity scores for PartEvo, Funsearch, EoH, and ReEvo, measured over the first and last 50 samples out of 500 total sampling steps. The score is measured using CodeBLEU [50], metric that quantifies the similarity of abstract syntax trees among sampled individuals. The results reveal that each LES method exhibits distinctive and consistent sampling patterns across different problem settings.
Table Similarity scores of sampling results for PartEvo, Funsearch, EoH, and ReEvo
First 50 samples ReEvo
Last 50 samples
Benchmark
Funsearch
EoH
PartEvo
Funsearch
EoH
ReEvo
PartEvo
P1
0.64
0.54
0.65
0.36
0.57
0.58
0.71
0.70
P2
0.60
0.55
0.65
0.38
0.56
0.60
0.71
0.67
P3 0.62
0.54
0.66
0.41
0.57
0.58
0.69
0.60
P4
0.61
0.54
0.63
0.41
0.58
0.60
0.60
0.60
PartEvo exhibits substantially lower similarity scores in the early stage compared to the other LES methods, indicating broader coverage of diverse subregions at the beginning of the search. As the search proceeds, PartEvo's similarity scores rise, reflecting natural shift toward convergence, where niches begin to share more effective and practical techniques. This dynamic adjustment highlights PartEvvo's ability to transition from wide exploration focused exploitation.
By contrast, Funsearch's island-based evolution encourages exploration across multiple subpopula- tions, but the lack of ceature-assisted niche construction results in less targeted resource allocation. EoH, which relies solely on tuness-bbssd parent selection, maintains consistently high similarity scores throughout the process, suggesting bias toward exploitation with limited exploration. ReEvo, by excessively adhering to accumulated long- and short-term experience, also produces persistently high similarity, leading reduced diversity.
The superior performance of PartEvo, as discussed in Subsection 4.2, underscores benefits its batch initialization strategy and explicit feature-assisted niche construction. By effectively balancing exploration and exploitation, PartEvo aligns with the ideal search paradigm: wide exploration in the early stages followed by concentrated exploitation the search converges.

---

4.4 Ablation study
We conduct a comprehensive ablation study assess the contributions of key components in PartEvo.
Effectiveness of Niching, reports the performance of PartEvo when dividing population of size 16 into 1, 2,4, and 6 niches. The number of niches, denoted as K, controls the degree of resource distribution during the sampling process. Specifically, K indicates no partitioning, where the entire population is treated as a single niche. When K is small, resources are concentrated on individuals with lower objective values, enabling more intensive exploitation of specific regions of the search space. Conversely, when K is large, resources are more dispersed across the search space. Such over-dispersion reduces the sampling intensity in promising regions, leading to diminished search efficiency due to excessive exploration. The results in Table 4 empirically confirm this trade-off.
Table 4: Effect of niching granularity on PartEvo performance
Benchmark
K=l
K=2
K=4
K=6
P1 9.379
4.544
0.000
8.241
P2 800.00
800.01
800.00
800.11
P3 6474.07
6549.09
6471.31
6553.08
P4
13418.7
10572.0
4539.1
12427.2
For population of size 16, setting = achieves the best balance between exploration and ex- ploitation, significantly improving the algorithm discovery process. When K=1, PartEvo reduces to strategy that selects parents solely based fitness values, similar to EoH. While this approach works well for simple problems (e.g., P2), where iterative refinements on any baseline algorithm can easily reach the optimal solution, it struggles with more complex problems. In such cases, the lack of exploration causes the search to stagnate in local optima, limiting the discovery of high-quality algorithms. When K=6, the excessive dispersion of sampling resources results in performance degradation, as promising subspaces fail to receive sufficient attention. These findings emphasize the importance of niching in improving search efficiency. However, the granularity of niching must be carefully tuned to strike an optimal balance between exploration and exploitation. Based on our experiments, find that for a population size of 16, K=4 achieves the best performance.
Table 5: Effect of feature usage on PartEvo performance
Benchmark
Random
Code Similarity
Thought Embedding
Pl
3.974
0.074
0.000
P2
803.92
800.42
800.00
P3
6607.99
6539.24
6.588.15
P4
17872.1
6424.7
13213.8
Importance of feature-assisted niche construction. 5able:ccompare the performance of PartEvo using feature-assisted partitioning versus random partitioning for niche construction. Results are averaged over three independent runs, each initialized with the same population. The best performance for each benchmark highlighted in bold, with the second-best underlined. The results demonstrate that PartEvo consistently achieves the best or second-best performance across all benchmarks when using Code Similarity and Thought Embedding. This suggests that these features effectively capture the structural properties of the language space, leading to more informed partitioning. In contrast, random partitioning, which lacks such structural guidance, performs significantly worse. These findings emphasize the importance of meaningful features in informing partitioning strategies. A more detailed analysis of the convergence behavior under different features provided in Appendix B.4
Effectiveness of prompt-centric and EC-inspired operators. We conducted an ablation study by systematically disabling specific operators to evaluate the effectiveness of prompt-centric and BC- inspired operators. Table 6 reports the performance degradation (APerf) relative to the full PartEvo. PvrtEvo serves as a baseline version, retaining only crossover and mutation operations, excluding advanced prompt and niche-based strategies. The results clearly indicate that disabling advanced

---

prompt engineering or niche- based collaborative search leads to significant performance drops. with the most severe degradation occurring when both are removed simultaneously. These findings highlight the crucial role of integrating EC-based techniques into the language space to enhance PartEvo's 's performance. Detailed ablation results for each operator are provided in Appendix B.5
Table 6: Performance degradation in ablation study of PartEvo
Method
P1
P2
P3
P4
Perrf (%)
Full PartEvo
0.074
800.00
6539.24
6424.7
0.00
w/o prompt-centris operators
1.585
800.00
6946.69
15259.2
553.74
w/o EC-inspired operators
1.288
807.37
7651.28
14324.7
451.31
PvrtEvo
7.248
807.58
8524.38
17924.0
2509.76
Discussions and limitations
Dependency on large language models The performance of PartEvo is inherently tied to the capabilities of the underlying LLMs. Specifically, prompt-centric operators, such as RE (Reflection) and SE (Summarization) depend on the LLM's 's ability to perform sophisticated contextual reasoning and leverage expert knowledge in algorithm design. As LLMs continue to evolve, we anticipate improvements in PartEvo's ' algorithm discovery capabilities, a trend also evidenced by comparative results across different LLMs (Appendix B.7]. However, is important to highlight that the integration of EC-inspired operators, such as CN and LGE, helps mitigate some of this dependency. These operators enable more refined parent selection processes and use compact prompts that capitalize on the LLM's core strengths [51] reducing reliance complex summarization abilities.
Understanding the language search space PartEvo indirectly analyzes and partitions the language search space through ceature-assisted niche construction. It treats algorithm discovery as a black-box optimization task, iteratively sampling candidate algorithms within the language space using an evolutionary process. A further pursuit is to enhance the understanding of the structure and landscape of this language space. Achieving this requires more sophisticated feature mapping techniques to better analyze and partition the space (PartEvo currently utilizes code similarity and thought embeddings as examples). is worth noting that PartEvo can serve as a valuable platform for validating novel language space mapping techniques. New algorithm similarity or landscape analysis methods can be easily integrated into the framework. As discussed in Appendix [B.4] improvements in niche partitioning (i.c., the ability to partition the language search space effectively) will likely lead to more stable and improved performance on specific algorithm design tasks.
Additional discussions and extended analyses are provided in Appendix A.2
Conclusion
This work introduces a general Large I Language Model-assisted Evolutionary Search (LFS) framework with abstract search space partitioning, enabling LES methods to effectively integrate niche- Evolutionary Computation (EC) techniques within language spaces. Building this framework, we present PartEvo, which combines advanced prompting strategies with both local and global search methods. Applying PartEvo to four optimization benchmarks demonstrates that it yields competitive meta-heuristic algorithms that outperform numan-designee ones. Furthermore, surpasses prior LES methods, particularly in crete example, proves the feasibility and significant benefits of integrating mature EC techniques into the LES paradigm. Future work will foeus on refining abstract partitioning techniques and developing more specialized EC search strategies to further enhance the LES method's 's capabilities in automating algorithm discovery. The source code can be found in https//github.com//igg wtw.m///igg./000///rrrii///
settings. Crucially, this work, using niching as con-
Acknowledgments and Disclosure of Funding
This work was supported by the General Research Fund (GRF) of the Research Grants Council (RGC) of Hong Kong (Project No. CityU 11217325).
10

---

References
[M] C. Huang, Y. Li, and X. Yao, "A survey of automatic parameter tuning methods for metaheuristics." JEEE Transactions on Evolutionary Computation, vol. 24, no. 2. pp. 201-216, 2019.
[2] F. Zhang, Y. Mei, S. Nguyen, and M. Zhang, "Survey on genetic programming and machine learning techniques for heuristic design in job shop scheduling," IEEE Transactions on Evolutionary Computation, vol. 28, no. 1. pp. 147-167, 2023.
[3] Q Zhao, T. Liu, B. Yan, Q. Duan, J. Yang. and Y. Shi, "Automated metaheuristic algorithm design with autoregressive learning." IEEE Transactions on Evolutionary Computation, pp. 1-1, 2024.
[4] W. Yi, R. Qu, L. Jiao, and B. Niu, "Automated design of metaheuristics using reinforcement learning within novel general search framework," IEEE Transactions on Evolutionary Compulation, vol. 27, no. pp. 1072-1084- 2023.
[5] C. Yang, X. Wang, L. H. Liu, Q. V.e,, D. Zhou, and X. Chen, "ILarge language models as optimizers," in The Twelfth International Conference Learning Representations, 2024.
[6] B. Komera-Paredess. M. Barekatain, A. Novikov, M. Balog, M. P. Kumar, E. Dupont, F. J. Ruiz, J. S. Ellenberg. P. Wang, O. Fawzi, et al. "Mathematical discoveries from program search with large language models," Nature, vol. 625, no. 7995, pp. 468 475, 2024.
[7] F. Liu, T. Xialiang, M. Yuan, X. Lin, F. Luo, Z. Wang, Z. Lu, and Q Zhang, "Evolution of heuristies; Towards efficient automatic algorithm design using large language model," in Forty-first International Conference on Machine Learning, 2024.
[8] H. Ye, Wang, Z. Cao, F. Berto, C. Hua, H. Kim, J. Park, and G. Song, "Reevo: Large language models hyper-heuristics with reflective evolution," " arXiv preprint arXiv:2402.01 145, 2024.
[9] Y. J. Ma, W. Liang, G. Wang, D.-A. Huang, O. Bastani, D. Jayaraman, Y. Zhu, I., Fan, and A. Anandkumar, "Fureka: Human-leva reward design via coding large language models," in ICLR, 2024.
[10] J. Liang, L. Zhang, K. Yu, B. Qu, F. Shang, and K. Qiao, "Interactive niching-basee based two-stage evolutionary algorithm for constrained multiobjective optimization." Swarm and Evolutionary Computation, vol. 83, p. 101402, 2023.
[11] L. Wang, R. Fonseca, and Y. Tian, "Learning scarch space partition for black-box optimization using monte carlo tree search," in Advances in Neural Information Processing Systems (H. Larochelle, M. Ranzato, R. Hadsell, M. Balcan, and H. Lin, eds.). vol. 33, pp. 19511-19523 Curran Associates, Inc., 2020.
[12] E. Wang. F. Cassano. C. Wu, Y. Bai. W. Song. V. Nath, Z. Han, S. Hendtyx, S. Yue, and H. Zhang, "Planning in natural language improves Ilm search for code generation," arXiv preprint arXiv:2409.03733 2024.
[13] 7.-I. Wang. Z.-H. Zhan, Y. Lin, W.J. Ya, H. Wang. S. Kwong, and J. Zhang, "Automatic niching differential evolution with contour prediction approach for multimodal optimization problems," IEEE Transactions on Evolutionary Computation, vol. 24, no. 1. pp. 114-128, 2019.
[14] N. Pillay and R. Qu, Automated Design of Machine Learning and Search Algorithmss Springer, 2021.
[15] Q Zhao, Q Duan. B. Yan, S. Cheng, and Y. Shi, "Automated design of metaheuristic algorithms: A survey." arXiv preprint arXiv:2303. 065332, 2023.
[16] T. Akiba, S. Sano. T. Yanase, T. Ohta, and M. Koyama, "Optuna: A next- gent-genee hyperparameter optimization framework." in Proceedings of the 25th ACM SIGKDD international conference on knowledge discovery & data mining, pp. 2623-2631, 2019.
[17] S. K. Karmaker, M. M. Hassan, M. Smith, L. Xu, C. Zhai, and K. Veeramachaneni, "Automl to date and beyond: Challenges and opportunities," " ACM Computing Surveys (CSUR), vol. 54, no. 8. pp. 1-36, 2021.
[18] Y. Sun, B. Xue, M. Zhang, G. G. Yen, and J. Lv, "Automatically designing enn architectures using the genetic algorithm for image classification," JEEE Transactions Cybermetics, vol. 50, no. 9, pp. 3840- 3854, 2020.
[19] P. scheduling problems." " European Journal of Operational Research, vol. 282. 3, pp. 835-845, 2020.
R. Ruiz, F. Pagnozzi, and T. Stützle, "Automatic algorithm design for hybrid flowshop
[20] Y. Bengio, A. Lodi, and A. Prouvost, "Machine learning for combinatorial optimization: methodological tour d' l'borizon." European Journal of Operational Research, vol. 290, no. 2, pp. 405-40444 4211, 2021.
11

---

[21] X. He. K. Zhao, and X. Chu, "Automl: A survey of the state-of-the-art," Knowledge-based Systems. vol. 212, p. 106622, 2021.
[22] N. Li, I. Ma, G. Yu, B. Xue, M. Zhang, and Y. Jin, "Survey evolutionary deep learning: Principles, algorithms, applications, and open issues," ACM Compusing Surveys, vol. 56, no. pp. 1-34, 2023.
[23] X. Wu, Y. Zhong, J. Wu, B. Jiang, and K. C. Tan, "Large language model-enhanced algorithm selection: towards comprehensive algorithm representation," " arXiv preprint arXi::2311.13/84 2023.
[24] X. Jiang, Y. Dong, L. Wang. 7. Fang, Q. Shang, G. Li, Z. Jin, and W. Jiao, "Self- "Seef-planning code generation with large language models," ACM Trans. Softw. Eng. Methodol., vol. 33, Sept. 2024.
[25] T. Bäck, D. B. Fogel, and Z. Michalewicz, "Handbook of evolutionary computation." Release, vol. 97,
no. p. B1, 1997.
[26] A. E. Eiben and J. E. Smith, Introduction evolutionary computing. Springer, 2015.
[27] R. Li, L. B. Allal, Y. Zi, N. Muennighoff. D. Kocetkov, C. Mou, M. Marone, C. Akiki, J. J. Chim, et al., "Starcnder: may the source be with you!." arXiv preprint 2305::2305.06 2023.
[28] E. Hemberg, S. Moskal, and U.-M. O' O'Reilly, "Evolving code with a large language model," Genetic Programming and Evolvable Machines, vol. 25, no. 2. P. 21, 2024.
[29] Q Guo, R. Wang. J. Guo, B. Li. K. Song. X. Tan, G. Liu, J. Bian, and Y. Yang. "Connecting large language models with evolutionary algorithms yields powerful prompt optimizers," arXiv preprint 209.52239. 2023.
[30] C. Xu, Q. Sun, K. Zheng. X. Geng, P. Zhao, J. Feng. C. Tao, and D. Jiang. "Wizardim: Empowering large language models to follow complex instructions." arXiv preprint 1rXi::3304.2244 2023.
[31] P. Shojaee, K. Meidani, S. Gupta, A. B. Farimani, and C. K. Reddy, "Lim-sr: Scientific equation discovery via programming with large language models," arXiv preprint arXiv:2404. 18400, 2024.
[32] F. Liu. X. Tong, M. Yuan, and Q Zhang. "Algorithm evolution using large language model," arXiv preprint arXiv:331.15249. 2023.
[33] V. Aglietti, I. Ktena, J. Schrouff, E. Sgouritsa, J. Ruiz, A. Malek, A. Bellot, and S. Chiappa, "Funbo: Dis- covering acquisition functions for bayesian optimization with funsearch," " arXiv preprint arXiv:2406.04822 2024.
[34] R. Hazra, A. Sygkounas, A. Persson, A. Loutfi, and P. D. Martires, "Revolve: Reward evolution with large language models using human feedback," arXiv preprint arXiv:2406.01300 2024.
[35] M. Nasir, Farle, J. Togelius, S. James, and C. Cleghorn, "Llmatic: Neural architecture search via large language models and quality diversity optimization," in Proceedings of the Genetic and Evolutionary Computation Conference, GECCO 24, (New York, NY, USA), p. 1110-1118, Association for Computing Machinery, 2024.
[36] W. Chen, S. Gao, C.-H. Chen, and L. Shi, "An optimal sample allocation strategy for partition baase random search," IEEE Transactions Automation Science and Engineering, vol. 11, no. 1. pp. 177-186, 2014.
[37] Y. Matanga, P. Owolawi, C. Du, and E. van Wyk. "Niching global optimisation: Systematic literature review," Algorithms, vol. 17, no. 10, 2024.
[38] M. Sazanovich, A. Nikolskaya, Y. Belousov, and A. Shpilman, "Solving black-box optimization challenge via learning search space partition for local bayesian optimization," Proceedings of the NeurIPS 2020 Competition and Demonstration Track (H. J. Escalante and K. Hofmann, eds.), vol. 133 of Proceedings of Machine Learning Research, pp- 77-85, PMLR, 06-12 Dec 2021.
[39] Y. Li, Y. Shen, W. Zhang. C. Zhang, and B. Cui, "Volcanoml: speeding up end-to-en end automl via scalable search space decomposition," The VLDB Journel, vol. 32, no. 2, pp. 389 413, 2023.
[40] H. Jin, X. Wang. S. Cheng, Y. Sun, M. Zhang. H. Lu, H. Wu, and Y. Shi, "Solving dynamic multimodal optimization problems via niching-based brain storm optimization with two archives algorithm." Swarm and Evolutionary Computation, vol. 89, p. 101649, 2024.
[41] M. Ahmed, R. Seraj, and S. M. S. Islam, "The k-means algorithm: A comprehensive survey and perfor- mance evaluation." Electronics, vol. 9, no. 8, p. 1295, 2020.
12

---

[42] F. Glover, "Tabu search paarch-parti i," ORSA Journal computing, vol. 1, no. 3, pp. 190-206, 1989.
[43] Z. Beheshti and S. M. H. Shamsuddin, "A review of population-based meta-heuristic algorithms," " Int. j. adv. soft comput. appl, vol. 5, no. 1, pp. 1-35, 2013.
[44] H. Yuan, Q. Hu, J. Bi, J. Lü, J. Zhang, and M. Zhou, "Profit-optimized computation offloading with autoencoder- evolution in large-scale mobile-edge computing," IEEE Internet of Things Journal, vol. 10, no. 13, pp. 1 1196-119999 19999, 2023.
[45] H. Yuan. Q. Hu. J. Bi, G. Gong, J. Zhang. and M. Zhou, "Machine-leve collaborative manufacturing and scheduling for heterogeneous plants," " IEEE Internet of Things Journal, vol. 11, no. 9, pp. 16591-16603, 2024.
[46] C. A. Coello, "An updated survey of ga- based multiobjective optimization techniques." ACM Computing Surveys (CSUR), vol. 32, no. 2. pp. 109 143, 2000.
[47] V. Kachitvichyanukul, "Comparison three evolutionary algorithms: Ga, pso, and de." Industrial Engi- neering and Management Systems, vol. 11, no. 3. pp. 215-223, 2012.
[48] M. Pant, H. Zaheer, L., Gerande-er,,dezz, A. Abraham, et al., "Differential evolution: A review more than two decades of research," Engineering Applications of Artificial Intelligence, vol. 90, p. 103479, 2020.
[49] F. Liu, R. Zhang, Z. Xie, R. Sun, K. Li, X. Lin, Z. Wang. Z. Lu, and Q. Zhang, "LIm4ad: A platform for algorithm design with large language model," arXiv preprint arXiv:2412.1
2024.
[50] S. Ren, D. Guo, S. Lu, I.. Zhou, S. Liu, D. Tang, N. Sundaresan, M. Zhou, A. Blanco, and S. Ma, "Codebleu: method for automatic evaluation of code synthesis," arXiv preprint 2r0i 10297, 2020.
[51] M. Levy, A. Jacoby, and Y. Goldberg, "Same task, more tokens: the impact of input length on the reasoning performance of large language models," arXiv preprint arXiv:2402. 14844888,, 2024.
[52] M. M. Bhatti, M. Maria, A. Zeeshan, and S. I. Abdelsalam. "Recent trends in computational fluid dynamics," " Frontiers in Physics, vol. 8, p. 593111, 2020.
[53] W. L. Jorgensen, "Efficient drug lead discovery and optimization," Accounts of chemical research, vol. 42, no. 6. pp. 724-733, 700-773320000 2009.
[54] P. Jiang, Q Zhou, X. Shao, P. Jiang, Q. Zhou, and X. Shao, Surrogate-model-boee design and optimization. Springer, 2020.
[55] Y. Mci, M. N. Omidvar, X. Li, and X. Yao, "A competitive divide-and-conquer algorithm for unconstrained large-scale black-box optimization," ACM Transactions on Mathematical Software (TOMS), vol. 42, no. 2, pp. 1-24, 2016.
[56] S. Nguyen, Y. Mei, B. Xue, and M. Zhang, "A hybrid genetic programming algorithm for automated design of dispatching rules," Evolutionary computation, vol. 27, no. 3. pp. 667-496,77 2019.
[57] K. Zheng, Y. Wang, F. Liu. Q. Zhang, and W. Song, "Cst-llm: Enhancing airfoil parameterization method with large language model," Aerospace Science and Technology, p. 1 110548, 2025.
[58] M. Liu, C.-H. Yu, W.-H. Lee, C.-W. Hung, Y.-. Chen, and S.HH. Sun, "Synthesizing programmatic rein- forcement learning policies with large language model guided search." arXiv preprint arX1v:2405.16450 2024.
[59] Y.-D. Kwon, Choo, B. Kim, I. Yoon, Y. Gwon, and S. Min, "Pomo: Policy optimization with multiple optima for reinforcement learning." Advances in Neural Information Processing Systems, vol. 33. pp. 2188 21 21198, 2020.
[60] X. Huang, W. Liu, X. Chen, X. Wang, H. Wang, D. Lian, Y. Wang, R. Tang, and E. Chen, "Understanding the planning of I1m agents: A survey," arXiv preprint arXiv:2402.02716, 2024.
[61] C. Gao, H. Jiang, D. Cai, S. Shi, and W. Lam, "StrategylIm: Large language models as strategy generators, executors, optimizers, and evaluators for problem solving," arXiv preprint arXiv: 1.80000, 2023.
[62] D. Huang, C. Yan, Q. Li, and X. Peng, "From large language models to large multimodal models: A literature review," Applied Sciences, vol. 14, no. 12, p. 5068, 2024.
[63] G. Jawahar, B. Sagot, and D. Seddah, "What does bert learn about the structure of language?," in ACL 20t9-57th Annual Meeting of the Association for Computational Linguistics, 2019.
13

---

[64] A. Islam, A. Debnath, M. Ghose, and S. Chakraborty, "A survey on task offloading in multi-access edge computingg" Journal of Systems Architecture, vol. 118. p. 102225, 2021.
[65] H. Yuan, Q. Hu, S. Wang, J. Bi, R. Buyya, J. Lü, J. Yang, J. Zhang, and M. Zhou, "Cost-optimized task offloading for dependent applications in collaborative edge and cloud computing." IEEE Internet of Things Journal, pp. 1-1, 2024.
14

---

A Discussion: Comparison, Limitation, and Impact Statement
A.1 Comparisons with Funsearch, EoH, and ReEvo
Funsearch [6] is of the earliest LLM-assisted Evolutionary Search (LES) methods. Prior to this, the Algorithm Evolution using Large Language Model (AEL) 32 explored the use of prompt engineering guide LLMs evolutionary operators for searching knowledge-rich language spaces.
EoH 7 keenly recognized that LLMs can assist in searching the code space through higher-level thoughts. Consequently, EoH represents individuals using both code and thoughts, exploring the cocvolution of thoughts and code within an evolutionary framework, which leads to superior perfor- mance while mitigating computational costs.
ReEvo 8 demonstrated that LLMs can provide verbal gradients through genetic cues, enabling more directed searches within the language search space. This approach effectively implements gradient-based search techniques in the language domain.
In terms of search strategies, Funsearch employs an island-based evolutionary approach to iteratively improve function quality by generating multiple islands in a random manner [6]. This island management promotes exploration and reduces the risk of premature convergence. In contrast, both EoH and ReEvo follow -ingll-population evolutionary framework, focusing on leveraging the capabilities of LLMs.
Our work introduces niche-based collaborative search mechanisms from evolutionary computation into LES, further enhancing search efficiency within language spaces. The proposed framework is general and can be easily integrated with existing LES approaches, with PartEvo serving specific instantiation of these core ideas. PartEvo first introduces feature-assisted niche construction to indirectly partition the language space into multiple subregions. It then defines prompt-centric operators (RE and SE) and EC-inspired operators (CN and LGE) for algorithm discovery processes. In the prompt-centric operators, LLMs are guided to generate promising algorithms through verbal gradients. The EC-inspired operators incorporate principles of niche collaboration and evolutionary heuristics to improve search efficiency. Experimental results demonstrate that the integration of niche-based techniques significantly improves algorithm discovery efficiency by over twofold, making LES more practical.
A.2 Additional discussion and limitation
Scenarios where PartEvo may not be suitable A key requirement for PartEvo is the continuous validation of the generated algorithms. This requires that the problems to be solved are formulated as mathematical models or situated within environments that can be simulated. For example, with sample budget of 500 and an evaluation budget of 30,000, a total of 15,000,000 evaluations would be required assess the objective value of the solutions. For some expensive optimization problems (e.g., parameter optimization in computational fluid dynamics (52) or drug design (53)) the extensive evaluations required by PartEvo could result prohibitive computational costs. A promising direction to address this is the use of surrogate models (54) to approximate objective values, which could accelerate the evaluation process and alleviate the computational burden. Alternatively. designing algorithms for similar, more computationally feasible problems and then transferring these algorithms to expensive optimization tasks may also provide an effective strategy. For large-scale problems, employing sidide-and-ooqueee approach [55] at a higher level to reduce problem complexity could enable PartEvo to focus on solving lower-level subproblems more efficiently.
Cross-paradigg discussion The LES is an emerging paradigm for automated algorithm discov- ery (AAD). This work primarily focuses on addressing the challenges of incorporating advanced evolutionary computation (EC) techniques into the LES paradigm, which complicated by its abstract, language- based search space. Using the integration of niching techniques as a case study, we demonstrate significant improvements in the performance, efficiency, and robustness of the LES paradigm for AAD when successfully applying these advanced EC methods. However, the field of AAD also includes other established paradigms, such as Genetic Programming (GP) (56), Deep Learning (DL.) R. and Reinforcement Learning (RL) 4. which merit a more systematic comparison with the LES paradigm. Incorporating insights from related works 77] 28 (77.5). we provide a qualitative, literatureinformeed comparison across several key aspects in Table highlighting these
15

---

distinctions. A comprehensive, systematic cross paradigm comparative study is a valuable direction for future research, and we are committed to pursuing it. Existing work has already begun to explore such comparisons [28]. and we plan to conduct a more thorough analysis to better understand the trade-offs between these distinct paradigms.
Table 7: A qualitative comparison of AAD paradigms
Feature
LES
GP
DL or RL
Manual Design Cost
Low. The framework automatically generates the algorithms based on a high-level task descrip- tion, requiring minimal human effort.
High. Requires signif- icant human effort to define problem-specific operators, functions, and grammar for the algorithm search space.
High. Requires exten- sive design of network architectures, training pipelines, and domain- specific datasets.
Algorithm De- sign Efficiency
High. A new task can solved in hours by re- lying iterative LLM inference, without the need for model training or fine-tuning.
High. The core pro- cess is based solely on evolutionary operations, which are computation- ally fast on per-step ba- sis, but can still require many generations to find a good algorithm.
Low. Requires exten- sive training periods, of- ten spanning days or weeks, for each specific task.
Resource requirements
High (requires LLM inference). However, LLM inference tasks can be offloaded third-party providers via API calls, reducing local hardware requirements.
Low. Runs on standard CPUs or GPUs, often without needing high- end hardware.
High. Requires substan- tial GPU resources and large-scale parallel com- putation, especially dur- ing the training phase.
Extensibility
High. Leverages the vast knowledge embed- ded in LLMs, allow- ing strong generaliza- tion to diverse algorithm design tasks with mini- mal adaptation.
Low. Generalizing to a new problem often re- quires extensive manual redesign of the grammar and operators to build the algorithm space for the target problem.
Low. Generalizing a new problem neces- sitates substantial data collection and lengthy retraining periods.
A.3 Extended applications
PartEvo excels at searching for feasible solutions within the language space. In this work, we utilize PartEvo to design meta-heuristic algorithms from the ground up. In fact, any design task at the language level can be executed using this framework, including planning [60], strategy formulation [1]] algorithm refinement, and more. Moreover, if LLMs are extended those with multimodal capabilities (2) feature-assisted niche construction can still be applied within image spaces. In this case, tasks such as industrial design and blueprint drafting could also be automated.
A.4 Impact statement
This paper advances the field of automated algorithm discovery by integrating EC-based evolutionary techniques with Large Language Models. The proposed method enhances search efficiency within language search spaces, offering significant potential for cost reduction and practical applications across diverse industries. Moreover, the algorithm evolution process generates valuable data, which can further accelerate research and development in this domain. By automating aspects of algorithm discovery, the role of human experts may transition from adapting algorithms for specific applications focusing on higher-level innovation. However, this shift raises important considerations, including potential changes in labor demands and ethical challenges related to transparency and faimess in
16

---

Al- Al-generated systems. We advocate for continued research to address these concerns and to promote the responsible and equitable deployment of this technology.
B Additional experiment results
B.1 Hyperparameters for PartEvo
Unless otherwise stated, we adopt the parameters in Table 8 for PartEvo runs. The entire experiment, including the implementation of PartEvo, EoH, and Funsearch for all four benchmarks, was developed in Python and executed on a single CPU (Intel 19-13980HX) with 32GB of RAM.
Table 8: Hyperparameter settings for PartEvo.
Hyperparameter
Parameter Description
Value
K
Number of niches to partition the population into
4
k
Number of cooperating niches selected in the CN operator
2
m
Number of parents selected each evolution step
N
Population size
16
E
Size of external archive
40
LLM
Version of L.LM used in the evolutionary operators
GPT-4o-mini
LLM temperature
Hyperparameter controlling the randomness of L.MM text generation
B.2
Comparative evaluations under identical initialization
To ensure a rigorous comparison the search behaviors of PartEvo, Funsearch, and BoH, we evaluate their performance using the same initial population. The initial population is iteratively generated by PartEvo and then shared across all other LES methods. Each method is independently run three times with the same initial population. The average results of these independent runs summarized in Table Even when eliminating the randomness introduced by initialization, PartEvo consistently outperforms the other methods across all cases. It is noteworthy that the performance of Funsearch, EoH, and ReEvo shows significant improvement when utilizing the initial population generated by ParEvvo. This finding highlights the effectiveness of the batch initialization strategy employed by PartEvo.
Table 9: Average results from multiple runs on training and testing instances for each optimization problem (Under identical initialization)
Problem Instances
Funsearch
EoH
ReEvo
PartEvo
Training 9.769
1.846
1.572
0.074
P1
Testing
0.879
1.646
1.013
0.023
Training 800.09
800.98
800.00
800.00
P2
Testing
439.31
368.95
364.19
365.20
Training Testing 57696.53
7042.21
7268.22
6564.44
6539.24
P3
61142.93
57455.34
56753.77
Training 24289.3
19152.4
11102.3
6424.7
P4
Testing 1.67x105
77134.3
79795.2
62731.4
Figure presents the convergence of the best objective function values over 500 samples for PartEvo, Funsearch, and EoH. Under the same initial population, the convergence behavior mirrors that observed in Subsection 4.2 Notably, PartEvo achieves superior results with fewer samples.
B.3 Comparative evaluations on Online Bin Packing Benchmark
In the main paper, we established four benchmarks (P1-4) showease PartEvo's 's design capabilities for heurisee algorithms. To provide more comprehensive evaluation, we further assess Pvo
17

---

020
Objertive Pumetoon Valus
Objective Function Value
DE-variant
Furzearti
-Eab ReEvo
Eah
RolEzo
PartEvv
- Partiivo
20
800
000
100
250
300
400
430
100
300
500
Sampling Count
Sampling Count
P1
(b) P2
x10
10000
PSO-variant
PSD-vanant
Otyectivn Function Value
Dbjective Functton Value
GA mariaai
D DE-vartant
5000
Punsearch
Funsearch
Coh ReEivo
Eoh
65000
Purtvoo
ReEvo
8000
PartiIvo
7000
6600
100
200
300
100
500
500
Sampling Count
Sumpling Count
(c) P3
(d)
Convergence behavior of different LES methods using the same seed across four indepen- dent runs on Problems P1, P2, P3, and
Figure 3:
alongside peer LES methods the Online Bin Packing Benchmark for heuristic design, utilized in EoH and Funsearch studies. Additionally, we compare the recent ReEvo approach on this task.
For consistency, sample budget of 500 is applied across all methods, with population size of 16, conducting four independent runs for each algorithm. Table 10 presents the fraction excess bins (as percentage) relative to the lower bound (with lower values indicating better performance) for various bin packing heuristics applied to Weibull instances.
Table 10: Average results from multiple runs the Bin Package Benchmark (Lower is Better)
Method
lk_ C100
1k Ik_C500
5k_C100
5k C500
10k_ C100
10k_C500
First fit
5.32
4.97
4.40
4.27
4.44
4.28
Best fit
4.87
4.50
4.08
3.91
4.09
3.95
EoH
4.76
4.38
2.32
2.18
2.04
1.97
Funsearch
4.07
4.69
3.19
3.56
3.16
3.44
ReEvo 4.24
3.94
2.13
2.07
2.07
1.96
PartEvo 3.51
3.23
1.14
1.11
0.84
0.79
First Fit and Best Fit are two human-designe heuristics. All LES methods surpass these baselines. The performance of all LES methods on the Bin Package Benchmark is consistent with the results from the four benchmarks presented the main paper. PartEvo achieves best average performance, proving its high efficiency in solving AAD tasks under limited budget constraints.
Notably, PartEvo and ReEvo outperform EoH and Funsearch, underscoring the advantages of Re- flection and Summary. However, due to its unique EC-inspired operators (CN and LGE), PartEvo surpasses ReEvo. This highlights that both "verbal gradient" and advanced EC techniques can enhance search efficiency and should be equally emphasized.
B.4 Detailed feature analysis
Figure presents the convergence behavior of the best objective values achieved by PvrtEvo when utilizing different feature sets problems PI and P2. Each feature set is evaluated through three
18

---

PSO-vatiatt E-vartant
Objectr Punction Valus
Obgetree Function Vluur
Golverian
810
DE vamaantt
Randon
Random
20
Thought Embedking
T Thought Embodding
Code Similarit
800
Codd Similarity
830
10
810
800
50
100
100
200
250
300
350
400
4.0
300
100
200
300
100
500
Sampling Count
Sampling Count
P1
(b) P2
Figure 4: Impact of partitioning features on convergence in PartEvo on P1 and P2
independent runs, all initialized with the same population. As observed, PartEvo, when augmented with the Code Similarity and Thought Embedding features, consistently converges to high-quality algorithms. In contrast, the use of random features (depicted by the blue curves and shaded regions) leads to broader range between the upper and lower envelopes, indicating greater instability in algorithm design.
These results emphasize the critical role of feature-assisted niche construction in enhancing the efficiency of automated algorithm discovery. Even with random partitioning, there is a chance that the resulting niches may coincidentally align way that facilitates the discovery of high-performing algorithms. This suggests that certain niche structures inherently lead to better outcomes. Notably, Code Similarity and Thought Embedding, as employed in this work, are two viable feature mapping strategies that markedly increase the reliability and efficiency of generating powerful algorithms with PartEvo.
B.5 Detailed ablation study
To better understand the contributions of the four evolutionary operators in PartEvo, we conducted comprehensive ablation study. The average performance across multiple runs for each ablation setting is summarized in Table 1T Variants of PartEvo with specific operators removed are denoted using the prefix "w/o" (e.g.. w/o RE indicates the removal of the RE operator). For clarity, we restate the definitions of the key variants here, consistent with Subsection 4.4 in the main text:
PartEvol: Removes the RE and SE operators, which utilize additional LLMs to enhance prompts through reflection and summarization.
PartEvol: Removes the CN and LGE operators, which implement subspace based collaborative search.
PartEvo : A baseline version that retains only crossover and mutation operations, excluding advanced prompt and subspace-bard based search strategies.
Table 11: Ablation study results on four benchmark problems.
Method
P1
P2
P3
P4
APerf (%)
Full PartEvo
0.074
800.00
6539.24
6424.7
0.00
w/o RE
0.389
800.00
6915.27
8433.5
117.74
w/o SE
1.493
800.01
6610.28
13761.8
515.12
P
1.585
800.00
6946.69
15259.2
553.74
w/o CN
0.723
800.00
6547.89
11357.3
241.82
w/o LGE
0.873
800.04
6849.81
10491.9
290.98
!
1,288
807.37
7651.28
14324.7
451.31
PartEvo
7.248
807.58
8524.38
17924.0
2509.76
19

---

The metric A quantifies the performance degradation relative to the full PartEvo, with larger values indicating greater declines. As shown Table each operator critical to the overall efficiency of PartEvo.
Removing either the reflection or summarization enhancement operators (RE or SE) results in significant performance drops (APerf=117774% and 515.12%; respectively). The impact even more pronounced when both are removed (APcrf=553.44%)
Similarly, removing the subspace- based collaborative search operators (CN or LGE) leads to substantial degradations (APerf=241.88 1.22% and 290.98% respectively). The effect worsens when both are excluded (APerf=451.31 %)..
The baseline version (PartEvo*) ) which excludes all advanced operators and relies solely on basic evolutionary operations, experiences the most severe performance degradation (APerf= 2509.76%)
These results underscore the importance of both prompt-centric and EC-inspired operators. By integrating these two aspects, PartEvo effectively combines the strengths of LLMs and niche-based EC techniques, demonstrating its potential as robust LES method for automated algorithm discovery.
B.6 Comparative evaluations with an expanded sample budget
PartEvo demonstrates superior efficiency in algorithm discovery under limited sample budget of 500 LLM queries. To further assess its potential, we expand the budget to 2000 queries and evaluate performance benchmarks P3 and P4. Benchmarks P1 and P2 excluded from this analysis, as prior results indicate that 500 queries already suffice for PartEvo, ReEvo, and EoH to design meta-heuristics that approach optimal solutions. All four LES methods are initialized with the same random seed for fairness, as detailed in Appendix [B.2] This experiment aims to: (i) examine whether PartEvo's is early-stage efficiency induces premature convergence that limits long-term performance; and (ii) investigate whether Funsearch, EoH, and ReEvo can exploit the larger budget to substantially improve their outcomes.
Table 12: Average results from three runs on training instances for benchmark P3 and P4
Benchmark
Budget
Funsearch
EoH
ReEvo
ParEvo
500*
7042.20
7268.21
6564.44
6539.23
P3
2000
6656.7
6749.5
6506.51
6479.49
500*
24289.3
19152.4
11 11102.32
6424.6
P4
2000
15646.1
14235.5
6038.45
5489.0
Average Improvement
20.53%
16.40%
23.25%
7.74%
Each method is independently executed three times under the 2000-sample budget. Table 12 reports the average performance, where lower values indicate better algorithm performance. Results marked with 500* are drawn from Table
As shown in Table 12 with an expanded budget of 2000 samples, PartEvo maintains its lead on both benchmarks P3 and P4, consistently outperforming Funsearch, EoH, and ReEvo. On P3, its performance improves modestly from 6539.23 to 6479.49 (0.92%), while the more challenging P4, achieves significant gain from 6424.6 to 5489.0 (14.4%). These results demonstrate that while PartEvo is highly efficient at discovering effective algorithms early on, it does not suffer from premature convergence. Instead, it continues to refine its search and achieve better results as more computational budget is allocated.
Meanwhile, Funsearch, EoH, and ReEvo exhibit substantial improvements with a larger budget. ReEvo benefits most, with an average improvement of 23.25%, followed by Funsearch (20.53%) and EoH (16.40%). This suggests that additional queries provide these methods with much-needed exploration capacity. Nevertheless, their performance remains far behind PartEvo. Notably, even with 2000 queries, Funsearch (15646. and EoH (14235.5) perform worse than PartEvo with only 500 queries (6424.6). This finding highlights PartEvo's 's remarkable efficiency in algorithm discovery, as it achieves up to 3-4x higher effectiveness in complex problem settings with a fraction of the computational effort.
20

---

3000
PSO-varami DEvvastnnt Purosarch
P50-variath
Obeetrr Furzioon Valun
Oltjective Funçtinn Value
GA-vatiat DEvvarant Funseorch
8000
ReBio H PartEvo
7500
7000
500
1000
1500
2000
200
400
630
800
1000
1200
1400
1600
1800
2000
Sampling Count
Sampling Count
(a) P3
(b)
Figure 5: Convergence behaviors of three LES methods across three runs benchmark P3 and P4.
Figure 5 further illustrates the convergence dynamics. On P3, PartEvo surpasses the human-designed DE variant baseline with only 600 queries and continues to improve, while ReEvo catches up after about 1000 queries. Funsearch and EoH converge more slowly, indicating lower search efficiency. On P4, PartEvo again dominates the entire trajectory, discovering strong meta-heuristics as early as 500 queries and steadily improving thereafter, demonstrating resilience against local optima. ReEvo progresses more gradually. only approaching PartEvo's 's 600-query performance after nearly 1200 queries, while Funsearch and EoH show limited late-stage gains.
By the 500-query mark, PartEvo has already identified highly effective algorithms, leaving limited room for further a pattern that explains its relatively modest 7.7% gain when the budget increases to 2000. Importantly, this efficiency is not transient but a fundamental property of PartEvo's search process, making consistently robust choice across both constrained and generous budgets.
Overall, these results highlight two key findings: (i) PartEvo achieves remarkable efficiency without premature stagnation, and (ii) integrating mature EC principles into LES crucial for sustaining effective algorithm discovery under varying computational budgets.
B.7 Comparative evaluations with different LLMs
To investigate the influence of LLM capability the overall performance of algorithm discovery, we evaluate PartEvo with three different models: GPT-3. 5PT-3.5-turll 5utboo, PrT-4o-miii. and GPT-4o. Experiments conducted the P4 benchmark with identical initial populations, fixed budget of 500 samples, and three independent runs for each model. For clarity of presentation, the results the three runs sorted in ascending order of performance. The outcomes summarized in Table 13
Table 13: Performance comparison of PartEvo with different LLMs on the P4 benchmark.
LLM
Run
Run
Run 3
Average
SEM
GPT-3.5-turbo
4924.5
9200.2
13261.9
9128.9
1965.360
GPT-4o-mini
4539.1
4679.5
10055.4
6424.7
1482.610
GPT-4o
3304.6
6028.0
8187.5
5840.0
1153.466
Overall, PartEvo demonstrates the ability to generate nigh-performing algorithms across all tested LLMs. As shown Table [3] stronger LLMs tend to yield better and more stable performance. These findings support our discussion in Appendix[A.. suggesting that as LLMs continue to advance, we can expect further improvements in PartEvo's 's algorithm discovery capability.
B.8 Comparative evaluations with various clustering methods
In PartEvo, clustering serves as a for grouping individuals within the feature space. As we outlined in Section 3.1 PartEvo is designed to be compatible with variety of clustering methods. We initially chose K-Means for simplicity, transparency, and widespread use. This deliberate choice allows us to isolate and confirm that the observed performance improvements primarily due to
21

---

the effects of niche-basee evolution, rather than influenced by the complexities of more advanced clustering methods.
To further demonstrate PartEvo's 's flexibility and compatibility, we conducted additional experiments using three distinct clustering methods: K-Means, Gaussian Mixture Model (GMM), and Spectral Clustering (SC). Each version of PartEvo was initialized with the same feature mapping and initial population ensure fair comparison. Each method was run three times independently ensure the robustness of the results. The experimental results, comparing the performance of these PartEvo variants against the baseline method EoH, are presented in Table 14
Table 14: Performance comparison of PartEvo with different clustering methods
Methods
P1
P2
P3
P4
EoH
1,846
800.98
7268.22
19152.4
PartEvo-K-means
0.074
800.42
6539.24
6424.7
PartEyo-GMM
0.048
800.10
6580.82
5423.5
PartEvo-SC 0.075
800.01
6543.54
6714.2
As the results show, all PartEvo variants consistently and significantly outperform EoH across all four benchmarks, regardless the specific clustering method used. On problems P1, P2, and P3, the performance metrics for PartEvo-Ke K-maans, SarrEvo-GMMM. and PartEvo-SC are remarkably similar, all achieving superior results to EoH. On the P4 problem, PartEvo-GMM shows a slight edge over the other two PartEvo variants, but this minor difference does not change the overarching conclusion that PartEvo, as a framework, fundamentally superior to the baseline method.
These findings strongly support our claim that the PartEvo framework is compatible with various clustering algorithms. Furthermore, they provide compelling evidence that the core advantage of our approach lies in its niche-based evolutionary mechanism, rather than the specific complexities of any single clustering method. This adaptability allows practitioners select clustering algorithm based on the specific requirements of their application.
C An example of PartEvo in practice
To facilitate better understanding of how PartEvo operates in practice, this section presents both the pseudocode and concrete application example.
C.1 Pseudocode of PartEvo
The pseudocode in A outlines description of the algorithm design problem and set evaluation instances, PartEvo autonomously generates customized algorithms.
Dottiines the overall procedure of PartEvo. Given natural language
Algorithm Pseudocode PartEvo
Require: Task description, training instances
1: Initialize the algorithm population
2: Construct algorithm niches via feature-assisted niche construction
3: while termination condition not met do
4:
for each niche in parallel do for each evolutionary operator do Select parent algorithms
5:
6:
7:
Construct few-shot prompt (parents task description operator-specific instructions)
8:
Generate offisring algorithm via LLM
9:
Evaluate offspring on training instances
10:
Manage offspring at the niche level
end for
12:
end for
end while
14:
return Best designed algorithm so far
22

---

In Line the initial algorithm population is generated either using batch initialization strategy with LLMs or by loading pre-existing algorithms. In Line algorithm niches are constructed based on the initial population using, feature-assisted niche construction. The process then enters an iterative evolutionary loop (Lines 3 which continues until a predefined termination condition met, such as reaching a maximum number of generations or achieving target fitness threshold. Within this loop, evolution occurs in parallel across each niche (Line(4). In each niche, designed evolutionary operators are applied sequentially or in random order to generate new algorithms (Lines 10). Each algorithm generation step includes selecting parent algorithms, constructing prompts, generating offspring algorithms using LLMs, evaluating the offspring, and managing the individuals at the niche level. Finally, the best- perforrie algorithm returned at Line 14
C.2 concrete example on benchmark P1
As an example, consider Benchmark PI, where the goal is to discover meta-heuristic algorithms for unimodal optimization problems. PartEvo initiates the process by defining the task description and preparing set of training instances. The task description specifies the algorithm design problem, guiding PartEvo generating relevant algorithms, as depicted in Figure The training instances serve practice problems, providing a consistent basis for evaluating and ranking candidate algo- rithms during the evolutionary process. These instances are representative of the problem domain and help refine the algorithms' effectiveness on the task. In the case of PI, the goal is to solve unimodal optimization problems, for which standard unimodal optimization functions are chosen as the target functions.
Subsequently, PartEvo proceeds through iterative evolutionary loops. In each loop iteration, parent algorithms are selected, prompts are constructed, new offspring algorithms are generated by the LLM, and these offspring algorithms are evaluated and managed. The prompt construction follows the template described in Appendix(D [D] Through repeated iterations, the algorithm population gradu- ally improves, with the best-performing algorithms emerging as specialized solutions to unimodal optimization problems.
This workflow generalizes beyond unimodal optimization. For any benchmark real-world applica- tion, one only needs to specify the problem and provide representative training instances; PartEvo will then evolve tailored algorithms accordingly.
D Prompts
This section describes the prompts employed in PartEvo. In all displayed prompts, the black text remains fixed, while the red placeholders correspond to task-specific elements, and the blue placeholders change throughout the evolutionary process.
Figures 6 and show the prompts used by the summarizer and reviewer LLMs. The generator LLMs, which utilize different prompts for each of the four evolutionary operators (e.g.. RE, SE, CN, LGE), detailed in Figures to These prompts applicable across all four optimization problem types.
Task description prompts for the four problem types are shown in Figure 14 Specifically, all problems are presented as black-box optimization tasks to the LLMs. This approach ensures that PartEvo is effective across wide range of optimization problems.
E Feature projection
The general LES framework employs an abstract search space partitioning mechanism that projects sampled individuals into feature space. In this feature space, clustering is performed based on the distances between the mapped representations, which indirectly partitions the abstract search space into multiple subregions. PartEvo leverages two primary methods for feature mapping:
E.1
Code similarity vector
Similar individuals should exhibit comparable relative distances to all other individuals. Thus, construct Code Similarity Vector (CSV) for each individual to represent its similarity to all
23

---

You are the person responsible for recording the progress of the experts' research. Some experts are working the following task:
"[Task Description Placeholder]"
Based previous design methods, have following summary:
"[Previous Summary Placeholder]"
Experts
explored additional [Number Placeholder] algorithms this problem, ranging
from No. to No. [Number Placeholder), with pertormance gradually declining (the smaller the objective function value, Algorithm No. has objective function value of [Value Placeholder], and its idea is: "[Thought Placeholder]"
performance).
Algorithm No. 2 has an objective function value of [Value Placeholder], and its idea is: "[Thought Placeholder]"
Please review the previous experiences and the current algorithms, analyzing which techniques within
algorithms effective in solving this problem and which are not. Finally, summarize both the effective and ineffective techniques, and update previous summmary accordingly. Please ensure that the summaayy you provide is enclosed in braces.
Figure 6: Prompt of summarizer.
An intelligent agent currently executing the following design task:
"[Task Description Placeholder]" The agent has designed an algorithm with the following ideas and code: "[Thought Placeholder]"
"[Code Placeholder]"
Based on your understanding and knowledge of this design task, please provide targeted suggestions
the current algorithm to guide the agent in improving it. Please return your final suggestions in curly braces so that the agent can interpret your advice.
Figure 7: Prompt of reviewer.
group experts are working on "[Task Description Placeholder]"
Experts have proposed [Number Placeholder] algorithms to this problem. The ideas for these algorithms as follows:
algorithm's 's idea "[Thought Placeholder]"
No. algorithm's m idea "[Thought Placeholder]"
Please create new algorithm that differs from the existing algorithms by at least [Value Placeholder] % First, describe your concept for the new algorithm and its main steps in as few words as possible while ensuring clarity. The description must enclosed braces. Next, implement in Python runnable function named "[Function Signature Placeholder]". This function should accept "[Value Placeholder]" input(s): "[Input Placeholder]". The function should return "[Value Placeholder]" output(s): "[Output Placeholder]". Do not include any comments in the code.
Figure 8: Prompt of generator for initiation.
others. As result, the distances between CSVs of similar individuals should be small. Since code is generated by LLMs, there may be many implementations of functionally equivalent code. In constructing the CSV, focus both the syntactic and dataflow similarities between codes. The process of constructing the CSV is as follows:
Pairwise similarity calculation: For population of size N, we compute the pairwise similarity between all individuals and store the results in similarity matrix S. Specifically, use the

---

You are an algorithm design expert. An intelligent agen! currently executing the following design lask: "[Task Description Placeholder]"
The agent has designed an algorithm with the following ideas and code: "[Thought Placeholder]"
"[Code Placeholder]"
decide whether incorporate experts feedback, and then create new algorithm that differs from the given one but motivated by it. The suggestion "[Reflection Placeholder]"
expert provided some suggestions algorithm.
First, describe concept for the new algorithm and its main steps in as few words possible while ensuring clarity. description must enclosed braces. Next, implement Pylhon a function named "[Function Signature Placeholder]". This function should accept "[Value Placeholder]" input(s): "[Input Placeholder]".
function should return "[Value Placeholder]" output(s): "[Output Placeholder]". You can understand the inputs and outputs based the current algorithm's code. Do not include any comments the code.
Figure 9: Prompt of generator in RE.
are an algorithm design expert. intelligent agent currently executing the following design lask: "[Task Description Placeholder]"
agent designed an algorithm following ideas
"[Thought Placeholder]"
"[Code Placeholder]"
Please help me create new algorithm that different from the given one but motivated by it. First, describe your concept the new algorithm and its main steps as few words as possible while ensuring clarity. The description must be enclosed in braces. Next, implement in Python runnable function named "[Function Signature Placeholder]". This function should accept "[Value Placeholder]" input(s): "[Input Placeholder)". The function should return "[Value Placeholder]" output(s): "[Output Placeholder]". You can understand the inputs and outputs based on the current algorithm's code. Do not include any comments in the code.
Figure 10: Prompt of generator in Mutation.
You are algorithm design expert, currently collaborating with other experts on the following task:
"[Task Description Placeholder]"
Experts have designed "[Value Placeholder]" slgorithms with their corresponding codes. The No. algorithm and the corresponding code are:
"[Thought Placeholder]"
"[Code Placeholder]"
The No. algorithm and the corresponding code are:
"[Thought Placeholder]"
"[Code Placeholder]"
Please Algorithm No. the main framework and try incorporate the characteristics other algorithms into to create better algorithm. First, describe your concept for the new algorithm and its main steps words as possible while ensuring clarity. description must enclosed implement in Python as runnable function named "[Function Signature Placeholder]". This function should accept "[Value Placeholder]" input(s): "[Input Placeholder]". The function should "[Value Placeholderj" output(s): "[Output Placeholder]". You can understand the inputs and outputs based on current algorithm's not include any comments code.
Next,
Figure 11: Prompt of generator in CN.
CodeBLEU metric [50] to evaluate the similarity between the code of two individuals. CodeBLEU
25

---

You are an algorithm design experi, currently collaborating with other experts on the following task: "[Task Description Placeholder]"
Currently. "[Value Placeholder]" algorithms have been explored for this problem, with their effectiveness decreasing No. "[Value Placeholder]" The concepts for these methods follows. No. algorithm is: "[Thought Placeholder]"
algorithm "[Thought Placeholder]"
Please analyze summary then modify the following algorithm for this problem. The thoughts and code for the algorithm to be modilied are as follows: "[Thought Placeholder]"
promising solution
"[Code Placeholder]"
First, describe your concept the new algorithm and its main steps few words as possible while ensuring clarity. The description must be enclosed in braces. Next. implement it in Python runnable function named "[Function Signature Placeholder]". This function should accept "[Value Placeholder]" input(s): "[Input Placeholder]". The function should return "[Value Placeholder]" output(s): "[Output Placeholder]". You can understand the inputs and outputs based on the current algorithm's code. Do not include comments in the code.
Figure 12: Prompt of generator in SE.
You are an algorithm design expert, currently collaborating with other experts on the following task: "[Task Description Placeholder]"
Experts are divided into several groups, with each group responsible for the development of specific algorithm cluster. Each incorporates different techniques while maintaining its own framework to explore diverse algorithms.
On your algorithm cluster, after several llerations, the current algorithm (idea corresponding code) is:
"[Thought Placeholder]"
"[Code Placeholder]"
During the iterations your cluster, better-perfonee penter-performnggg algorithm appeared, its idea code are as
follows:
"[Thought Placeholder]"
"[Code Placeholder]"
In addition, among all the algorithms tested (Including those from other clusters), the best-performing algorithm's Idea and code are as follows:
"[Thought Placeholder]"
"[Code Placeholder]"
Using the above information and adhering to the core framework of the current algorithm, please suggest potential improvements enhance performance solving problem. First, briefly describe your concept for the new algorithm and its main steps. The description must be enclosed in braces. Next. implement Python function named "[Function Signature Placeholder]". This function should accept "[Value Placeholder]" input(s): "[Input Placeholder]". The function should return "[Value Placeholder]" output(s): "[Output Placeholder]". can understand inputs and outputs based the current algorithm's code. Do not include any comments in the code.
Figure 13: Prompt of generator in LGE.
proposes weighted n-gram match and syntactic abstract syntax match measure grammatical correctness, and introduces semantic data-flow match to calculate logic correctness. In this work, the similarity score between two individuals and j is computed as:
Stj = 0.5 st,j - 0.5 di.j
where 8ty and d:,, denote the syntax match score and dataflow match score, respectively. This formulation ensures balanced evaluation of both syntactic structure and semantie behavior.
26

---

You are lasked with solving black box optimization Bebllem/iiee--inteee Nonlinear Programming (MINLP) problem/integer nonlinear programming (INLP) using meta-heuristic optimization algorithm. The goal is to minimize the objective value. Please design metaheuristic algorithm that can effectively solve this optimization problem and
optimal solution.
Figure 14: Prompt of specifie problem.
2. Symmetrization CodeBLEU scores not inherently symmetric, i.e., St, / Sj.á. which may lead to inconsistencies the similarity matrix. address this, we enforce symmetry by averaging the scores:
S 2
This step ensures that the similarity relationship between any two individuals consistent and interpretable.
Feature representation: For each individual, the corresponding row the similarity matrix is used its CSV, capturing its relative relationship with all other individuals in the population.
E.2 Thought embedding
In PartEvo, individuals are encoded into two components: code and thought. For the thought component, expressed in natural language, we leverage techniques from natural language processing to perform feature projection. Specifically, we utilize pre-trained language models, such as BERT [63. to extract high-dimensional semantic representations of the algorithmic descriptions provided by individuals. The similarity between individuals is then determined based on the distances between their embedding vectors. The process is as follows:
1. Text tokenization:
The algorithmic description of each individual is tokenized using the BERT tokenizer, ensuring truncation and padding to a maximum sequence length 512 tokens.
2. Embedding extraction: The tokenized input is fed into pre-trained BERT model, and the hidden state of the token from the final layer extracted as the embedding. This embedding is fixed-length vector capturing the semantic meaning of the input text.
Feature representation: The extracted embeddings are directly used as feature vectors for the
individuals.
E.3 Random features for comparison
To evaluate the effectiveness of the adopted feature projections, we also introduce random feature vectors baseline. For each individual, random vector is generated by sampling from Gaussian distribution with a dimensionality consistent with the other feature representations. As shown in Table 5] the comparison with random features demonstrates that the proposed feature projections significantly enhance the performance of PartEvo. This improvement one of the key reasons why PartEivo achieves higher search efficiency compared Funsearch, which employs an island model.
F Human-designed meta-heuristic algorithms
In this work, we use human-designed meta-heuristic algorithms baselines. Specifically, we adopt enhanced variants of Genetic Algorithm (GA), Differential Evolution (DF), and Particle Swarm Optimization (PSO). Detailed descriptions of these variants are provided below.
F1 GA-variant
GA is classical meta-heuristic optimization algorithm inspired by the process of natural selection. It iteratively evolves population of candidate solutions through genetic operations such selection, crossover, and mutation. The GA-variant used in this work incorporates several enhancements to improve its performance and convergence behavior. These enhancements include:
27

---

A hybrid selection mechanism combining Roulette Wheel Selection (RWS) and Tournament Selection (TS) is employed. TS is used the primary selection method, with tournament size of 3, while RWS ensures diversity preservation.
Selection:
Simulated Binary Crossover (SBX) is adopted the primary crossover operator. SBX generates offssping by simulating the distribution of genes in natural reproduction, with crossover rate of 0.9 and distribution index rj Single-point crossover also supported as a fallback.
Crossover:
Gaussian Mutation and Uniform Mutation used to introduce diversity into the population. Gaussian Mutation adds normally distributed noise to selected genes, with a mutation rate of 0.1.
Mutation:
Fitness normalization: mitigate the effects of extreme fitness values, normalization step applied to the fitness scores before selection.
Elitism: The best solution in the current generation preserved and directly passed to the next generation, ensuring that the global best solution is not lost.
An early stopping mechanism is implemented to terminate the algorithm if no significant improvement in fitness is observed for 500 consecutive generations. The convergence tolerance is set to 10
Early stopping:
The GA variant is designed balance exploration and exploitation effectively. The use of hybrid selection and advanced crossover/mutation operators ensures that the algorithm can explore the search space thoroughly, while elitism and early stopping mechanisms help accelerate convergence and avoid stagnation.
F.2 DE-variant
DE is population-based optimization algorithm that relies on mutation, crossover, and selection operators to iteratively improve candidate solutions. The DE variant used in this work incorporates several enhancements to improve its convergence speed, robustness, and ability escape local optima. These enhancements include:
Adaptive mutation factor and crossover probability: The mutation factor F and crossover probability CR are dynamically adjusted during the optimization process. Specifically, and CR are sampled from uniform distributions in the ranges [0.5, 0.8) and [0.5, 0.9). respectively, to balance exploration and exploitation.
Global best participation: The global best solution is incorporated into the mutation process to guide the search towards promising regions of the solution space. This modification improves convergence by leveraging the knowledge of the current best solution.
Diversity maintenance: To avoid premature convergence, random perturbations are introduced when population diversity is low. This helps the algorithm escape local optima and explore new regions of the search space.
Boundary handling: A reflection boundary handling strategy is used prevent solutions from being stuck the boundaries of the search space. This ensures that mutated solutions remain within valid bounds while preserving diversity.
Early stopping: An early stopping mechanism is implemented to terminate the algorithm when no significant improvement fitness is observed for 500 consecutive generations. The convergence tolerance is set to 10
Parallel evaluation: For computational efficiency, the evaluation of the objective function is parallelized when possible, particularly for computationally expensive objective functions.
The DE variant is designed to enhance the standard DE algorithm's 's performance in solving complex optimization problems. By incorporating adaptive parameters, diversity maintenance, and boundary handling, it achieves a good balance between exploration and exploitation. The early stopping mechanism further improves computational efficiency by avoiding unnecessary evaluations when the algorithm has converged.
28

---

F.3 PSO-variant
Particle Swarm Optimization (PSO) is population-basee optimization algorithm inspired by the social behavior of bird flocks and fish schools. In PSO, particles explore the search space by updating their positions and velocities based on their own best-known positions and the global best-known position. The PSO variant used in this work integrates several advanced mechanisms to enhance its convergence performance and robustness. These enhancements include:
Dynamic inertia weight: The inertia weight w is linearly decreased from Wmax 0.95 to W'mi 0.4 over the course of the optimization balance global exploration and local exploitation.
Simulated Annealing (SA): simulated annealing mechanism is incorporated to probabilisti- cally accept worse solutions during the optimization process. This helps the algorithm escape local optima, with the acceptance probability controlled by a temperature parameter that de- creases exponentially.
Genetic Algorithm (GA) operators: Genetic dggorithm-inspiree operators, including crossover and mutation, are integrated to further enhance diversity. Specifically:
Crossover: Offssrrng particles generated by combining the local best positions and the global best position using a weighted random combination.
Mutation: A small mutation probability (pm. 0.02) is used to introduce random perturba- tions to particles.
reflection boundary handling strategy is applied to ensure particles remain within the feasible search space. This prevents particles from being stuck the boundaries while maintaining diversity.
Boundary handling:
The velocity of each particle is clamped to prevent it from exceeding the maximum allowable velocity, which is proportional to the search space bounds.
Velocity clamping:
Hybrid fitness evaluation: The fitness of each particle is evaluated using a combination PSO updates, GA operators, and simulated annealing, ensuring robust exploration of the search space.
The PSO variant combines elements from PSO, GA, and SA, making it hybrid algorithm that leverages the strengths of each method. The dynamic inertia weight helps balance exploration and exploitation, while the simulated annealing and genetic operators enhance the algorithm's 's ability to escape local optima. These modifications make the PSO variant well-suited for solving complex optimization problems.
F.4 Comparison and analysis
To directly compare the performance of the standard and enhanced versions of GA, DE, and PSO, evaluated all methods on four benchmark problems (PI-P4), as summarized in Table [5] In this table, GA, DE, and PSO refer to the standard versions, while GA-variant, DE-variant and PSO-varian represent the enhanced versions discussed in the main text. Each algorithm was limited to a maximum of 30,000 solution evaluations. The best results are highlighted in bold, while the second-best results are underlined.
Table 15: Best results from multiple runs on training and testing instances for each optimization benchmark (Lower values indicate better performance)
Problem
Instances
GA-variant
DE
DE-variant
PSO
PSO-variant
PartEvo
Training Testing
5.944 x 106
523.182
15.849
0.004
118.973
16.241
0.000
P1
6.358 x 10
13.145
0.680
0.000
50.000
0.000
0.000
Training Testing
2485.72
2087.63
1139.32
868.40
1467.15
939.02
800.00
P2
1.05 x 10
542.48
516.96
365.60
2.11 x 10
411.80
344.62
Training Testing
15661.66
52850.88
21379.04
6541.55
156817.83
10036.17
6471.31
P3
78080.20
85629.55
77851.57
61298.43
2.89 x 10
60464.14
56876.40
Training
2.60 x 10"
1.48 x 107
91784.6
25736.4
82953.9
32594.0
2792.1
P4
Testing
5.13 x 107
8.48 x 107
3.78 x 10F
1.72 1
6.31 x 10
2.28 x 10
14396.1
29

---

From the table, several key observations can be made. First, the performance of different algorithms varies significantly for each problem. highlighting the distinct behaviors of each algorithm within the search space, which allows them to adapt to different landscapes.
Second, within each family of algorithms, the enhanced variants typically outperform the standard versions, except for the GA variants on problems P3 and P4. This suggests that targeted algorithmic improvements enable more effective navigation of the solution landscape. However, improper upgrades can lead to performance degradation, underscoring the importance of thoughtful algorithm design.
Most notably, the algorithms generated by PartEvo achieve superior performance across all bench- mark instances. This is because PartEvo continuously explores techniques that are well- to the problem during the evolutionary process, allowing the designed algorithms to efficiently search the landscape. This demonstrates that PartEvo plays crucial role in automating and enhanc- ing the algorithm design process, highlighting its potential to assist human experts in developing high- nigh-performance optimization algorithms.
G
Benchmark
This section introduces the four optimization benchmarks and their specific instances used to evaluate PartEvo. These include symthetic problems, such as unimodal and multimodal optimization problems, as well as real-world scenarios, including task offloading in mobile edge computing systems and machine-level scheduling for heterogeneous plants.
G.1 Unimodal optimization problem
Unimodal optimization problems feature single global optimum with no local optima, making it an ideal benchmark for testing basic search capabilities. Algorithms yielded by the LES framework are evaluated on their ability to locate the global optimum in this simple scenario efficiently. This benchmark serves baseline verify PartEvo's ability to generate algorithms with effective core search mechanisms.
For this study, we utilize various unimodal objective functions. For some of these functions, we introduce a shift term s. This shift increases the difficulty of optimization, preventing the LES framework from generating algorithms with fixed outputs that could bypass the optimization process. In the experiments, the parameter configurations for the training and testing instances are shown in Table 16 The mathematical formulations and key characteristics of these ten functions are outlined as follows:
Table 16: Unimodal instance setups for the experiments
Formula
Variable Range
Optimal Function Value
(x(
[-20, 20]
Training Instance
f2(x)
[-00, 20]
fa(x,
[-20, 20]
fa(x, 2.5)
[-10, 10]
f2(x, 5)
[-10, 10]
fa(x, = 0)
[-10, 10]
Testing Instance
fa(x, s 2.5)
[]
fa(x, = 1)
301
O
fx(x)
[-100, 100]
fo(x)
[-100, 100]
Ro 0
Sphere function. The Sphere function is a well-known unimodal optimization benchmark, character- ized by global minimum located at the origin. It is mathematically defined as:
m
f1(x)
(3)
30

---

-21,22,...... 22,... ,2,...,, E I" represents the decision variable in an n-dimensional space. The function exhibits a smooth, symmetric parabolic shape, with the global minimum located at x This simplicity makes it a fundamental benchmark for evaluating optimization algorithms.
where x = [x
Shifted sphere function. The Shifted Sphere function is modified version of the Sphere function, where the global minimum is shifted to predefined point in the search space. This shift increases the difficulty of the optimization process, preventing trivial solutions. mathematically expressed as:
f2(x, s) = T s;)²
where s = [51, 82, 82,,.., ..] E IR" represents the shift vector, and 8, denotes the shift applied to the i-th dimension.
Unimodal sinusoidal function. The Unimodal Sinusoidal function is a mangle-peaked benchmark, characterized by the use of squared sine terms. It is mathematically defined as:
fa(x, Σ sin (x: 8:)
This function is particularly useful for testing optimization algorithms on periodic landscapes, as its sinusoidal nature introduces oscillatory behavior while maintaining single global minimum, making it valuable benchmark for evaluating algorithm robustness.
Unimodal Gaussian function. The Unimodal Gaussian function is smooth and symmetric bench- mark function, characterized by a single peak. It is mathematically expressed as:
fa(x, 8) = Σ exp (-(x; (=x=)))3))) );)
(6)
here, s = [81, 88,,82 .., 8n] E Rn is the shift vector that determines the location of the global minimum. This function is often used in optimization benchmarks due to its smooth, unimodal nature and its ability to test the precision of optimization algorithms.
Unimodal logistic function. The Unimodal Logistic function smooth, monotonic benchmark function, commonly used to evaluate optimization algorithms in continuous domains. It is mathemat- ically defined
fs(x)
1 + exp(-x;)'
i=1
This function particularly useful for testing algorithms in scenarios involving smooth, non-linear transformations.
Unimodal exponential function. The Unimodal Exponential function is monotonic benchmark function characterized by exponential decay. It is mathematically expressed as:
fa(x)= Σ exp( exp((xx:).
The global minimum of this function is asymptotically approached as each x; tends towards positive infinity, making a useful benchmark for testing optimization algorithms in decaying landscapes.
G.2 Multimodal optimization problem
Multimodal optimization problems introduce multiple local optima in the search space, making them significantly more challenging. These problems evaluate the designed algorithms' ability to balance exploration and exploitation. Practical algorithms must avoid premature convergence to suboptimal solutions while maintaining robust search strategy. This benchmark tests PartEvo's ability to address complex optimization landscapes.
We utilize ten multimodal objective functions in study. Similar to the unimodal optimization benchmarks, we introduce shift term in certain functions to increase the difficulty of optimization. The parameter configurations for the training and testing instances are summarized in Table [7
31

---

Specifically, the decision variable dimensionality for the training instances is set to 40, while that for the testing instances is set 50. This setup allows us evaluate the generalization capability the metaheuristic algorithms generated by LESs. The mathematical formulations and key characteristics of these functions are outlined below.
Table 17: Multimodal instance setups for the experiments
Formula
Variable Range
Optimal Function Value
fr(x,s = 2)
[-5.12, 5.12]
100
Training Instance
folx.s 50)
[-600, 600]
200
Sin(x, 5)
[-10, 10]
500
f 10)
[-10, 10]
50
f1)(x)
[-100, 100]
300
Testing Instance
(12(x)
[-0.5, 0.5]
[1a(x)
[-10, 10]
0
f14(x)
[0. m]
-50
Modified Rastigin function. The Modified Rastrigin function is multimodal benchmark with numerous local minima, making it a popular choice for evaluating optimization algorithms. is mathematically defined as:
(x; si)2 A cos((2(s)))] + 100,
(9)
where A 10, and the search range is 24 12,1.12,..111 5.12). The global minimum located x xs= with an optimal value of fxx") = 100.
Modified Ackley function. The Modified Ackley function is widely used multimodal benchmark with a complex landscape. is mathematically expressed as:
I Σ 1=1
[ cos 2T (x: 8:]) 20 + 50, cos((2
s)( = ex0
-.2
[1 8.)2
exp
(10)
where is Euler's constant, and the search range is E [- 1-32.768, 32.768). The global minimum is shifted to x*=s, with an optimal value f(x*) 50.
Modified Griewank function. The Modified Griewank function features a smooth landscape with numerous local minima. It mathematically defined as:
fo(x,s) = +
1
I 51)
II
cos
2j
+ 200.
(11)
4000
Vi
1=1
i=1
The search range is r; [0000, 600). The global minimum located at x*=s, with an optimal value of f(x*) 200.
Modified Levy function. The Modified Levy function known for its rugged landscape and numerous local minima. It is mathematically expressed as:
fio(x, s) = sin? (Tw) ) Σ [w,, 1)2 (1 10 ( +
(1 + +sn (2Tw,) +500,
(12)
where Wi 1 #1 and the search range is x; E [[100, 10). The global minimum is located at x"=s;, with an optimal value f(xx) ) = 500.
Modified Schaffer function N.2. The Modified Schaffer Function N.2 is a two-dimensional multi- modal benchmark. is mathematically expressed as:
sin? (II 50)² (x2 + 50) - 0.5
fx(() 0.5 + [1+ 0.001 ((x1 50)2 + (x2 + 50) + 300.
(13)
32

---

The search range is r; E [-100, 100), and the global minimum is located at x" = [50,-50] -50], with an optimal value of f(x*) 300.
Modified Weierstrass function. The Modified Weierstrass function is characterized its fractal-like landscape. It is mathematically expressed
Kan
fx2(x)
cos (2 (z+ - 0.5))
a cos (2mb²* 0.5) + 10, (14)
2=1 R=0
A===
where a 0.5, 3, kmax 20, and the search range is m, E 0.5, 0.5). The global minimum is located at x o.... 0J.
with optimal value of f(x*) 10.
Modified Alpine function. The Modified Alpine function is multimodal benchmark, defined as:
f1a(x) Σ ay (x;) + 0.1v.
(15)
The search range is x; E [-10, 10), and the global minimum is located at x" = [0, 0,.0,.... .......,............ OJ. with an optimal value of f(x*)
Modified Michalewiez function. The Modified Michalewicz function is highly multimodal, defined
sin i
2m
f1((x)
Σ sin(2;)
(16)
where m = 10, and the search range is I; E 10,z|. k.. The global minimum depends on the dimension- ality, with approximate optimal value of ((x))==>
G.3 Task offloading in mobile edge computing systems
Task offloading is well-established optimization problem the domains cloud edge comput- ing [4]. A representative system architecture is illustrated in Fig. [5] In such systems, numerous smart mobile devices are required to exccute computation- and data-intent inte-inteesive applications. However, due to physical limitations, such restricted device size, these devices often lack sufficient computa- tional resources and energy to process tasks independently. Task offloading addresses this limitation by partitioning computational tasks and offloading them to high-performance servers, either the edge or in the cloud, for efficient processing.
Claud data
Subtask SMD
Subtask SMD
Subtask SMD
W vreless connection
Wired connection
Connection among ENs
EN1
SBs
sns
SMD S
Edge secver
SMD
EN $85
Figure 15: System architecture of cloud-edge collaborative computing framework
The task offloading problem is inherently constrained by various practical factors, including the characteristics application tasks, the availability of communication channels, and the computational capacity of edge servers (64] Consequently, task offloading is frequently formulated as constrained mixed-integer non-linear programming (MINLP) problem. The objective typically involves minimiz- ing the total system cost, maximizing data throughput, reducing energy consumption, all while
33

---

adhering to the imposed constraints. To tackle these complex problems, meta-heuristic algorithms are widely employed in the literature.
In this study, we treat task offloading as a real-world optimization problem to evaluate the algorithmic capabilities of PartEvo. The performance of the algorithms designed within LESs this benchmark reflects their ability to address real-world optimization problems with intricate constraints and real-time requirements.
To this end, we implement a task offloading system model inspired by prior studies optimization objective is to minimize the total system cost. To simulate diverse real-world scenarios, we design multiple instances by varying the number users, user request types, and user distributions. The configurations of the instances used for training and testing are summarized in Table 18
The
Table 18: Configurations of task offloading instances for experiments
Instance Index
Number of Users
Average Task Size (GB)
15
150
Training Instances
2
30
150
5
50
15
300
45
10
Testing Instances
20
7
30
200
1
100
G.4 Machine-Level scheduling for heterogeneous plants
Scheduling and production planning are among the most critical challenges in industrial manufac- turing, as they directly impact the efficiency and cost-effectiveness of operations. production supply chain, from the moment a customer places order, every stage ragging from raw material procurement to the production of semi-finished components and the final assembly of diverse products-requine meticulous scheduling and coordination. Under the paradigm of industrial Internet, the entire supply chain can be modeled as a collaborative manufacturing and scheduling (CMS) system, composed of multiple heterogeneous factories. The architecture of such system is illustrated Figure 16
complete
Optimization production
scheduling
Enterprise
Orders
Customers
Allocating production tasks
Finished orders
m
m
m
Factory
Factory 2,
Factory P
Transporting products among Factories
Machine
Machine
Products
Delivering products to customers
Figure 16: Architecture of a collaborative manufacturing and scheduling system.
34

---

The CMS framework involves four primary entities: enterprises, customers, a transportation system, and multiple factories. Within the system, there are [P] factories, and each factory p (1=><IP) contains ", machines. Raw materials, semi-finished products, and finished goods are transported among factories and delivered to customers via the transportation system. The workflow proceeds as follows: customer places an order with the enterprise, which then formulates a manufacturing and scheduling plan based on the order's requirements, including product types, quantities, and delivery deadlines. Subsequently, the enterprise assigns production tasks to machines across different factories and coordinates the transportation of products. Finally, the transportation system delivers the completed products directly to customers, fulfilling the order. To minimize the overall cost of the CMS system, it is essential to optimize the collaboration among factories, encompassing manufacturing, transportation, and sales processes.
In this context, we use the term "product" to broadly refer to all production components, including raw materials, semi-finished products, and finished goods. The production scheduling process consists of several interdependent plans: (1) production plans for different factories, (2) substitution plans for interchangeable products, (3) transportation plans for products between factories, and (4) delivery plans specifying the number of products shipped from each factory to customers, Notably, the factories within the CMS system exhibit heterogeneous characteristics, which introduce additional complexities:
1. Heterogeneous manufacturing capacities: Different factories have varying production capabilities for different products, resulting in diverse time and material costs for manufacturing the same product.
2. Variable transportation costs: Due to differing distances between factories and between factories and customers, transportation costs vary significantly across the system.
Complex product relationships: Certain products can substitute for others (e.g., a large resistor can be replaced by two smaller resistors in series), while others may be consumed as intermediate inputs in the production of other products. Properly managing these dependencies is critical for cost reduction.
The objective of the scheduling process is to allocate production tasks to machines and coordinate the inter-factory transportation of intermediate products. The goal is to minimize the combined production and transportation costs while ensuring that all customer orders are completed within their respective deadlines. This problem represents a large-scale combinatorial optimization challenge, often formulated constrained nonlinear integer programming problem. Numerous studies have explored the use of meta-heuristic algorithms to address such production scheduling problems.
In this study, we evaluate the performance of our proposed framework, PartEvo, on this industrial scheduling benchmark. PartEvo designed to generate meta-heuristics capable of addressing large- scale, high-dimensional optimization problems with complex interdependencies. By benchmarking PartEvo on this real-world problem, we aim to assess its ability develop efficient and effective scheduling strategies.
Table 19: Configurations of machine-level scheduling instances for experiments
Instance Index
Product Types
Factory Number
Planning Horizon
Training Instances
3 7 days
2
9
3
7 days
Testing Instances
3
9
5
7 days
10
3
7 days
To this end, we implement a CMS system model inspired by prior work (45). The optimization objective is to minimize the total cost of production and transportation. To simulate diverse scenarios, we construct multiple problem instances by varying key parameters, including the number of product types, the number of factories, the number of production lines per factory, and the planning horizon. The configurations of these instances, used for both training and testing, are summarized in Table 19
35

---

H
sicceefittess up. ap.array arraay ([obj(ctiveeeeccc (wapjeecctreeffuuttiaaaaaaaa function(ind) for ind in populatioa)) [:nun_ Generated meta-heuristic algorithms
This section presents the
meta-heuristic algorithm generated by PartEvo across all benchmark problems. These codes illustrate that PartEvo can design highly complex yet well- structured algorithms. These empirically validated solutions can be readily deployed address analogous optimization challenges, demonstrating significant practical utility.
Meta-heuristic represents the most effective algorithm generated by PartEvo for benchmark Pl, achieving an optimal fitness value on both the training and testing sets. Thought of Meta- heuristic is follows: "The proposed algorithm is an Enhanced Hybrid Differential Evolution (EHDE) approach. It combines principles from differential evolution and adaptive mechanisms while incorporating niche-like exploration. The algorithm leverages multiple mutation strategies to diversify the search space. It also includes a local refining search that periodically improves the best individual. The key steps are: initialization of the population, evaluation of fitness, application of multiple mutation strategies, a dynamic adaptive learning rate for perturbation, selection of the best solutions, and local search refinement step. The evaluation process monitors the fitness convergence to terminate early if necessary." Detailed code can be found in Fig. 17
import numpy
from scipy optimize import ninimize
daf algo ppppplattalllppplaattrr individual.upper andividual_leen objective_funetion)) eox_evaluatione 30000
evaluations -
iun_individuale. nun_dimensions initial_population..oo..
pepulation initial.population copy
best_aoluuion population [up (rgpin((ooeectiveeeeeeeett
for ind in populationl)
evalustions nss_evaluatione:
cict_niche_iindcae ap argsort
for in range (num, individuals):
autation.strateg, up rp.dem choice (['best', 'rand!])
if iuratiin,attateen -- "beat';
vutation_vector
rp.rendon uniform 0.5,
run_dimensions)
indices uppraanorn permutation (nun, individuals) [:3]
xi, x2, x3 population [indices]
autation_vector
0.5 (x2 x3)
autation ap.clig(autatiion
individual. individuel_upper
crossover mask randon. (nun
< 0.6
trial_vector up shara mask, matatien_ vector, population [1])
vector up (lppttrllll individual_ individual.upper
let_fitness (ijjectiveffncctin(ttiiii evaluations +-
if nex fet_eit < #iche_fitness[1]]
population [i] triel_vector
if nev_sitnese < objective__unction (baetiioobesttsollttiio best_solution trisl_
for idx in best_niche_indicee loral_search. candidate
(abjective_functiin) population [idx],
#list (xip (indiviiidda Lindividual_loeer loxer (ettvvidaaluupper) new nev_ fitness objective_fuucee woocaiionooooal.......... seachh candidatte evaluations
law_losal_ffte fitness abjactive, abjjctiie__uunttinn (beat_solutionn
best_aolution
if (ttchttancnceiiiinnnnn 16-5:
bresk
return best cost_solution
Figure 17: The best PartEvo-generann Meta-heuristic for PI.
36

---

Meta-heuristic 2 represents the most effective algorithm generated by PartEvo for benchmark P2. achieves an optimal fitness value of 800 on the training sets and outperforms the algorithms generated by the peer LES method on the testing set. Thought of Meta-heuristic 2 is as follows: This algorithm, named Enhanced Adaptive Chaotic Differential Evolution with Dynamic Local Search and Diversity Management (EACDE-DLSDMM) reñines the existing EACDE framework by incorporating enhanced chaotic initialization, strategic adjustment of mutation rates, adaptive local search techniques, and more robust diversity maintenance strategy. By optimizing the balance between exploration and exploitation, the algorithm systematically updates the population based on performance feedback, dynamically alters the local search radius based on stagnation detection, and integrates both structured and random local searches for improved solution refinement. This approach aims effectively minimize the objective function while ensuring diverse and high-quality solutions are retained throughout the optimization process." Detailed code can be found in Fig. T8
Meta-heuristic 3 represents the most effective algorithm generated by PartEvo for benchmark P3. It outperforms widely used DE-based hybrid algorithms. To ensure execution efficiency, employs multi-processing for computation. Thought of Meta-heuristic 3 is as follows: "The modified algorithm, named Enhanced Adaptive Hybrid Differential Evolution with Self-Adjusting Local Search (EAHDE-SALS), incorporates several key improvements over the original AHDE-SALS framework. It utilizes advanced adaptive mechanisms that dynamically adjust both mutation and crossover rates based population diversity and solution success. The local search component has been enhanced with a hybrid mechanism that combines perturbative refinement with gradient- awareness to optimally exploit promising regions of the solution space. The memory management system is refined to retain not just elite solutions but also a diverse set of high-performing candidates, ensuring that exploration of the solution space remains effective and robust. The algorithm proceeds through initial population evaluation, global search via differential evolution, localized searches, dynamic parameter adjustments, and concludes once the evaluation limit is reached." Detailed implementation can be found in Fig. 19
Meta-heuristic& 4 represents the most effective algorithm generated by PartEvo for benchmark P4. It has achieved impressive results on P4, reducing costs by 90.1% on real-world scheduling compared to human-designed DE-based algorithms. Thought of Meta-heuristic 4 is as follows: "The proposed algorithm, named "Enhanced Adaptive Hybrid Enhanced Memory, Dynamic Local Search, and Novel Perturbation Scheme," aims to strike balance between exploration and exploitation more effectively. Key enhancements include: 1) Implementing novel approach for elite solution selection that emphasizes diversity, 2) Utilizing perturbation strategy that introduces variations to the population, drawn from both elite solutions and randomly generated candidates, 3) Enhancing the local search process to explore the neighborhood of elite solutions more aggressively. using a combined memory that recalls previous successful perturbations, and 4) Integrating cooling temperature mechanism to modulate mutation strength dynamically. The iterative process will continue until the evaluation limit is reached, ensuring the best-found solution is returned at the end." Detailed implementation can be found in Fig. 20
-ifferi-Diifer Evolution with
adaptive
I Licenses for used assets
Since Funsearch is not officially open-source, we conducted our experiments using the version of Funsearch provided on the LLM4AD platform.
Table 20: Asset Usage and License Information
Type
Asset
License
Usage
Funsearch [6
MIT License
Evaluation
Code
EoH 77]
MIT License
Evaluation
ReEvo
g
MIT License
Evaluation
Dataset
L.M4ADD
49
MIT License
Testing
37

---

def Logietic_naple r-3.99):
plgo(Initiall.ppuultiooo individual_upper,
Individual_lover, objective.
population initial peettal_oppiattion.cpp.
beat_aolutien popolation
best_value ojjeettie.fucciioottts.......
evaluatione (en(popalation)
nax_evaluatiens, stagnat:io__coutt stagnation_ 30000, 0, 1500
tocal_eearc__raduus 0.1 (individual__pp _appaaapaappprrrr.. individual_
nutation_base divereification_raat
0.15
shile avaluations mar._valluation::
staguatien,.ouut 2- staguation,linit
divereification.cour int (len (ppppullpll)))
divereification_raa
for in sagge(divrrsffiiarooooooom randon_solution ap.randon. www.:m/.iiinmmmmimmmm vtndforllidddvdddal___.
individual_
eize-population.ttun [1])
(ppulation[pp...
wtt.pp.eedda..rand..e.. (populstion chape [0])] randon
*= 0.5
stagnation
chaos_nap up zoroo (pppulatttippiii
for in range (popalatiin. [h] [0]):
chaos_map [i] logistic_nap (np.aiianaapppadadda....aaa rad
chave_map [i] chaos_nap[i]
(individual.uppe individusl_
individual_
population (population chaos_nap)
for in range paagoppoollaiic...dppee
current_solution population
current_value objective_ ceurrotttttttr...ttio
if current_valus < best_
best_value current_value
beat_solution current_eelution.cop
ctagnation_count
clss:
adappi stagmation_
for sdaptive.uutation.fe
autation_base (L-Deet.vaalue//beet.aalll 1e-10)) in range wtwpoppaaiio..sspeeeeeeeeeeeeeee
selocted,indices up ra.do.... (population. shape [0], size-3, replace-
False)
parenti, parent2 solected [:2]
selected__nndcce[[1]]] objeetive_
[ill) elss salected_iiniiea [1:3]
donor denor_vector up. sl.p www.ecppppplaaiooppprrrreo (parenti)) adaptive_mutes (aattti factor=( population [ (opulsstion individual_lowe
individual_ugped
trial_solution oppcllapddooor.vactor up. randon. (scale=0. 1tmp std
population), aize=population.co ahape [1]), individual_
individual_upper
abjjeciioe_funttoo (tritttlootrrr__soololo
evaluations +-
trial_value beet_value:
best_value trial_value
beat_solution trial_.oolsttonccag//
stagmation_count
for in ppggepooplaatinn..aap
local_neighbort population randon. uniform www..mmmooocall.earrd....
soccl_,earch_redduue size-population shape [1])
nocah_nei population [i] ap random. uniform (-1ocal_seerch_raa
0.6, local__earch_radeeeeooeee 0.5, dize-pooplation..aaae
local_netghbort (p.cl1p(local_neighh
individual_ individual_upper
local_neighbor?
individual_loees individual_upper
best best_local_sod solcutino local neillareih if objective Dejeetive_functioniiee (ooiilonhooellneigbboroo
objective_functiiii (local locaa__eeigbber2) noigh_neighbrr2 neighbor_valus = abjectiie_functtiiee (baatttt
evaluations
if neighbor_valle best_value:
beat_value beighbor_value
best_solution gest_local,eolutiin stagnation_count = evaluations
seral_ *= 1.05
returu best, best_iolution
Figure 18: The best ParlEvo-generated Meta-heuristic for P2.
38

---

import aumpy se aр from concurrent futures import ProeessPoolBxecuse
vyllatte.olluiionooolliioo objective__fucce function);
return abgective_tunctinnfflltttoo
def Lggeibaitiell.ppplat..ooo individusl_upui individual_lover (ojeetiiv_function))
population inivial.populattoo... copy
sun_individuals, aun_variables population. shape
avaluations
heat_objactive float()inf.)
nemory [] 0.5
autation_factor 0.5
0.5
alite_size int (0.1 num_individualss)
30000:
for in (mug. individuals);
indices (ist (Lai(rrrannnnnniiivvvvddee33
indices.renoven
a, b, c up. ppp.clipppppllactonnn mutation_ factor (population [b]
3, replace=Falze)
population (c)), individusl_ individual,uppe.
trial up cppy cpp.coyy (ppplaatooo))
for in range (aun, variablea):
if renden rand() crossover_
trial[j] nutant
wwllute..eluttionntiiall objective.
evaluations +-
if fstrial objeectve: best_objective f_trisl
best_solution trial
[_current evaluate, pppiuic..oltttoocppppulaallll objeetivee
if f_trisl f_current:
population
success_rata + if f_trial best_ best_obbective: nanory (triad))
diversity up www.udl(vallatt..luuuiaaaaaa population))
([lective_ffuction)
mutation_facto: ap. cstpcsscccess_raate nun_indswiduals, 0.1, 1.0) crossover_rats ap.clip(anccesa_ratt nun_individuala 0.5, 0.1, 0.9) success_rate
if len (nemory) > 10:
unique_menory _enory (map (tuple, nemory))
nemory sorted (listttitt uuiiuue_neooryl koy=lanbda sol: ovaluate_tolutiin bejectivv_funcctoon) [:10]
local_search_stze cp.clip(0.1 (1 diversity), 0.01, 0.1)
with ProcessPoolExecutor(( as executor:
list map (lanbda sea_solution: evaluate_colutaet solutionn (np. clip men_solution randon. (aun_dddaaaaablllo 2 1)
local_search_sten individual__over, individual.upper). Dojeetive_fuuctionn memory))
for index, f_local in emneraata(reault.
if f_local < bbeet_objecttve:
best_objective =_local
beat beat_solation = www.lpppnnorylnnnaaaa (ap. randon. rand socal_ individual_
2 individual.uphh upper)
indiviiduissssse.........iiisiid sorted (population, key=lanbda evaluate_solution (ind, Dnllettivefuacttion relite_size
population sizte_ olite_individeale
returu best_ solution
Figure 19: Meta-heuristic 3: The best PartEvo-generated Meta-heuristic for P3.
39

---

def pepulation mp.
individusl_upper
individual_lower objective_function
copy (pt.ccaattalllpppllatiioo
popolation_aize. dimensions popalation..hape eval_count, bolttin, best_fitmess aax_evaluationss caa.aattinn.aggaatton.ccmm
(loat('inf')
max_stagnation 30000, 0, 100
alite_count: max(1, population.size 10)
initisl_tenp cooling.rete, elite_nemory 1.0, 0.85, 0] def
dimensions, lover, upper):
intervals up. sg.linepace (lover, upper, size
points пp. rand (aize, dimensions)
sample intervals return ap.clip(sample, laver, upper)
(intervals[1:] intervals [:-1]] points
population =Leii_.bpporreeeeeeaap... htt..aannngnngppolltiooo...eeeeo dimensions, individual. individusl.ugper))
fitness_values up. ap. (braayy(bbjscciirofffttttt. (ind) for ind in popolation))
population_ates
beat_aolution
best_fitmess wp..iia(fiieee..vlllll.
elite_memory. aapnory..ppend (beat_aolution)
while eval_count nex.evaluatione
solected_indices (f.arrgor [:elite_couat]
parents popilttiollsclccted..ddi offapring
temp - initiel_ (cooling.rate, ** (eval_count // population,eize))
for in range (pouuuaattati
parentl, paremtt,ppremt22pareetee crossover_point aaraaaa.aaaaaaa.aaadaaaaa randint (1, dimensiene)
rww.oradd................... (prreeaattaa....aa........ (paleeepaaatttaaaaaapas.. [0], replace-Falee)]
childi-mp. cot.:/ppppccccctttttttccaaac crossorrrrtrrrrooooooooorr
crossover_point wtt.-pppottppaaeeee... parent2 [crossover_point:]]]
child2=np. concatanate ((paatt crossover p...nt.....annttt. parenti
p [erossover.
child [child1, child2]:
if ap random.rand() < 0.5:
autation_idx ap.randon randint (0, dimensions)
child(nutation. idx] cpp randon. wnwforn (Indmmmidddiiddlllloe....
mutation_idzi andividual_uppe (mutation_idx)) temp,
individual_lower [mutation_iex]], individual_upper [nutation_iix] apppring-appand(np... (chppcci (child, individual_ individual_uppar)
offepring, local sera up array (offspriag),
for ind in offepring:
for range(loaal.caarc.tttts]]]
parturb_ids ap.random. randint (dimensions)
trisl_colution ap. copy
(pp. random.rado.....mom uniform (individual. (idiiial looer
perturb_idx], individual [perturb_ idx]), individual_lover if ind[:] trial_solution
perturb_idx], undividuxl__pper (perturb_xx)]
< snrectiv__fucctioniin
ritness. fitness fitness_ wtw.:a__vauussss.pappp.p. (ttaaaa htt.:r/aaaaaaaaaooooooaeeeteeeee cubjective_tuncttonnnie (Ind) for ind in offspring])
eval_count affapring.shape
conbined, combined.populer population ap. vataca (steca((poolaattio) offspring))
appulese
combined_fitnees fitmees_ argeort [:population_
population combined_poppllationnll lcc.ooiieddiitneeeeeee
fitness_values fitness_valles [ct.baaalluccccntteeeeettttaa
burrent__eett_fitnee -p..niiffitaess._aall
if currant_best_ffte current_beat_fttnee..
current.
best_fitnecd
best_ population ang.a
elite_memozy append (best_oouutio))) stagnation_couat
staguation_count
if stagnation_count >= max_stagnation:
perturb.idx randon randint (0, population_eize
(ppulatio(pperuubb.uk - capp (ap.pandos. randon, randon.rand rand (dieeniiios
individual_upper indiwidual [over) individual_Lover. individual_loser andiridual_upper?
if elite_nemory:
population population(perturb_kkk (perturb_ idx] cp. (elttee___ee (up randon choice (len(
elite_nenory))]]
staguation_count
olite_mezorr list (set (nap(tuple, elite_nemory!))
(eeeetttte_eemrr))) elite_count:
elita_nemory elite_nemory [elntte_count:]
retura best_ bolution
Figure 20: The best ParlEvo-generated Meta-heuristic for P4,
40

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
The main claims are that integrating niche-based EC techniques into Large Lan- guage Model-assisted Evolutionary Search (LES) can improve the efficiency of automated algorithm discovery. We propose a LES framework that incorporates abstract search space partitioning enable structured niche construction. We instantiate PartEvo and conduct comprehensive experiments to validate our claims. These are accurately conveyed in both the abstract and introduction.
Justification:
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results
be expected to generalize other settings.
is fine to include aspirational goals as motivation as long is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification:
The Limitation of the PartEvo is performed in Appendix A.2
Guidelines:
The answer NA means that the paper has limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g.. independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low or images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed not penalize honesty concerning limitations.
3. Theory assumptions and proofs
41

---

Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [NA]
Justification: The paper does not include theoretical results. Instead, it demonstrates through
experiments that the integration of niche-basee techniques into the LES method enhances the efficiency of algorithm discovery.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Answer: [Yes]
The technical details of the PartEvo framework are provided in Section Sec- tion 3 The information required to reproduce the experiments is detailed in Appendix [.1
Justification:
The prompts involved in PartEvo are thoroughly presented in Appendix D
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution a specific model and empirical evaluation, may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, other means that appropriate to the research performed.
also be provided via detailed
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) the contribution is primarily a new algorithm, the paper should make clear how to reproduce that algorithm.
(b) If the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) the contribution a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
42

---

(d)
We recognize that reproducibility may be tricky in some cases, in which case authors are welcome describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [Yes]
Justification:
The source code be found in httpss//gihhbbboom////ggg//////p
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://nips... cc/
for more details.
While we encourage the release of code and data, we understand that this might be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https (oodeSummissiooPoa
/ //nips. httiss.cccpubblcguuudeeo
for more details.
The authors should provide instructions data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permirted.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary understand the results?
Answer: [Yes]
Justification: The experimental setting, including benchmarks, baselines, and implementa- tion details, is described in Section The details of the four benchmarks are provided in The hyperparameters used in the experiments listed in Appendix B.1
Appendix G
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, or as supplemental material.
7. Experiment statistical significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [Yes]
Justification: The average performance and standard deviation of four independent runs on all benchmarks are provided Section 4.2
Guidelines:
43

---

The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by enror bars, confi-
dence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
should be clear whether the error bar is the standard deviation or the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that out of range (e.g. negative етror rates).
If error bars are reported in tables or plots. The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information the com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification: The information the computer resources needed to reproduce the experi- ments provided in Appendix A.2 and B.T
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).
9. Code of ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics httpp::/neurips: http:///cc/c.ttttttttctiiiaaaeee
Answer: [Yes]
Justification:
This paper fully complies with the NeurIPS Code of Ethies.
Guidelines:
The answer NA means that the authors have reviewed the NeurIPS Code of Ethics.
the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make to preserve anonymity (e.g., if there is special consid- eration due to laws or regulations their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive socictal impacts and negative socictal impacts of the work performed?
Answer: [Yes]
44

---

Justification:
The societal impacts are discussed in Appendix A.4
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of MI.).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (c.g., pretrained language models, image generators, scraped datasets)?
Question:
Answer: [NA]
Justification:
The paper poses no such risks
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
recognize that providing effective safeguards challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper. properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification:
The licenses for the existing assets detailed in Appendix
Guidelines:
The answer NA means that the paper does not existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset used and, possible, include URL.
45

---

The name of the license (e.g., CC-BY 4.0) should be included for each asset. For scraped data from a particular source (c.g., website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the palaaets.ppperssttt paperssi thcode. com con/datases package should be provided. For popular datasets,
has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
If this information is available online, the authors are encouraged to reach out to the asset's s creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification: No new assets are introduced at this time.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create anonymized URL or include an anonymized zip file.
14. Crowdsourcing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?
Answer: [NA]
Justification:
The paper does not involve crowdsourcing nor research with human subjects
Guidelines:
The answer NA means that the paper does involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According the NeurlPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals equivalent for research with human subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification:
The paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsoureing nor research with human subjects.
46

---

Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if' applicable), such as the institution conducting the review.
16. Declaration of LLM usage
Does the paper describe the usage of LLMs if is an important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing, editing, or formatting purposes and does not impact the core methodology, scientific rigorousness, originality of the research, declaration is not required.
Question:
Answer: [Yes]
This paper focuses on improving the efficiency of the LLM-assisted evolution- ary search method for automated algorithm discovery. LLMs play core role in algorithm generation in this study, and we describe their usage in detail in Section 3.2 Additionally, detailed LLM prompts are presented in Appendix
Justification:
Guidelines:
The answer NA means that the core method development in this research does not involve L.MMs as any important, original, non non non-standard components.
Please refer to our LLM policy (https: /tttps://eeuips... wwtperrrppspp.oooeeeeeeeeeiieeee ccCoonffreencss/2225555 for what should or should not be described.
47


Reviews:
[{"review_id": "zRWwBoeryT", "reviewer": "Reviewer_pyQe", "summary": "This paper discuss a popular topic in recent optimization community: using large language models for automated optimization algorithm design. The authors propose a refined procedure termed as PartEvo to improve existing works such as EoH and Funsearch. Inspired by the success of niching strategy in traditional evolutionary algorithms, the authors first propose adding niching-based searching process into EoH framework. To this end, the authors first address the challenge of measuring similarity in abstract program space, where two metrics are proposed: code similarity and thought similarity. The former is based on the BLEU score of the generated algorithm sourcecodes and the latter is based on the hidden feature of the thought text (as those in EoH) through a pretrained BERT model. Given the similarity metric, the authors further add two additional searching strategies into the algorithm evolution process, which provide effective program search within similar algorithm individuals and balance the exploration and exploitation tradeoff in the same time. The experimental results, including the comparison test and ablation study, demonstrates this work shows better performance than EoH and Funsearch, and its proposed niching-related technique is effective.", "questions": "See Strengths And Weaknesses above.", "limitations": "I strongly suggest the authors discuss the limitations of this paper. Using LLMs for algorithm design is promising. However, the true potential, the efficiency and the generalization boundary are very aspects that deserve in-depth analysis.", "rating": 3, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 2, "originality": 2, "strengths_and_weaknesses": "Pros:\n\n1. The writing of this paper is generally clear. \n2. The motivation of this paper is properly backed up.\n3. The methodology, including the similarity metrics and the corresponding niching-based searching enhancement, is well-organized and technically rational.\n4. Considering existing LLM-assisted optimization framework, this work presents certain novelty and performance improvement. The potential audience of this work is hence moderate.\n\nCons:\n\n1. Overall, I think the contribution of this paper is incremental. Applying evolution-like thoughts for algorithm design is novel, such as EoH. Employing high-level evolutionary technique such niching, however, is incremental.\n2. I still think the authors overpromise the significance of this work, since they claim that “Importantly, the proposed framework is highly flexible and extensible.” (line 134), and “ Additionally, any feature and clustering method can be utilized” (lines 136-137). For a framework, at least two or more scenarios should be validated. I suggest the authors add more experimental analysis to validate the general usage of the proposed framework.\n3. There might be an unclear methodology issue. Given the two similarity metrics the authors have proposed, how to use them in the proposed framework? I can not find any detail of how to combine these two metrics, this further lead to my confusion on the experimental results in Table 1, are the results of PartEvo obtained by aggregating the results of PartEvo with CSV and PartEvo with TE? If so, the robustness of PartEvo would be highly dependent on the metric selection, which is somewhat not so ideal.\n4. Although in experiments, four optimization scenarios are validated, the experimental setting is still too simple. For example, in P1 and P2, the training instances and testing instances are “small size with simple structure (sphere function)”, which makes me doubt the generalization ability of the learned algorithm. Adding more diversity in your testset to validate the true potential of the generated optimizer.\n5. The experiments still lack broader comparison with existing automated algorithm design methods such as those using Genetic Programming and Reinforcement Learning. The training efficiency, the resource requirement and the final performance should be fairly discussed.  \n6. Review on the related works is quite limited. For “Automated algorithm discovery” part, many valuable papers should be recognized and carefully reviewed to respect the scientific integrity and historical evolution path in this field, such as\nvan Stein, N., & Bäck, T. (2024). Llamea: A large language model evolutionary algorithm for automatically generating metaheuristics. IEEE Transactions on Evolutionary Computation.\nYu, H., & Liu, J. (2024). Deep Insights into Automated Optimization with Large Language Models and Evolutionary Algorithms. arXiv preprint arXiv:2410.20848.\nMa, Z., Guo, H., Gong, Y. J., Zhang, J., & Tan, K. C. (2025). Toward automated algorithm design: A survey and practical guide to meta-black-box-optimization. IEEE Transactions on Evolutionary Computation.", "was_revised": true, "final_justification": "I am maintaining my original recommendation that this paper is not ready for acceptance at NeurIPS. While I appreciate the authors' detailed, multi-stage rebuttal, the revisions and arguments presented do not sufficiently address the core weaknesses of the manuscript.\n\nFirst, as I indicated, the primary contribution of this work—integrating a niching technique into the LES framework—is a useful incremental improvement. However, it does not strike me as a groundbreaking or sufficiently novel contribution to meet the high standards of this venue. Given the limited theoretical novelty, the burden of proof falls heavily on the experimental section to demonstrate significant and insightful results. Unfortunately, the experimental validation remains a weak part of this paper.\n\n**The new experiment hastily added to compare against AM, POMO, and LEHD is problematic.** First, the major experiments conducted in the paper  focused on numeric optimization, yet the authors chose to compare against methods that are primarily designed for combinatorial problems like the TSP, seemingly for the sake of a quick comparison, rather than selecting from the many existing learning-for-optimization algorithms relevant to their domain. Second, I have strong reservations about the conclusion that LES can outperform a strong baseline like LEHD. This claim is likely biased, as it is based on tests conducted on a very small and insufficient sample of only six instances.\n\nBesides, **the lack of standardized benchmarking problems persisted.** The paper's numerical experiments are still not evaluated on any widely-used, standard benchmark test sets, such as the COCO or CEC suites. Instead, the authors rely on their own newly constructed dataset, which makes it difficult to compare the method's performance and position it within the broader field.", "rebuttal": "Dear Reviewer and Area Chairs:\n\nThank you for your thoughtful review and constructive suggestions. We appreciate your insights and respond to each of your concerns in detail below.\n\n> ### Q1: The contribution of this paper seems incremental?\n\nWe would like to clarify the significance of our work. Integrating large language models (LLMs) with evolutionary computation (EC) for algorithm design is still an emerging research field. While some initial methods have explored the basic combination of the two using simple evolutionary mechanisms such as crossover and mutation, EC itself has decades of research focused on advanced mechanisms designed to improve performance, efficiency, generalization, and more. These mature EC techniques hold the potential for driving progress in this emerging LES paradigm. In this context, we introduce **niching** as an example to demonstrate how EC techniques can contribute to more sustainable progress.\n\nImportantly, our work goes beyond simply applying niching. We uncover the challenges faced when applying it, particularly when the search space is extended to the implicitly shaped language space. To address this, we introduce *feature-assisted niche construction* to support the implementation of niching techniques within the LES framework. Our experimental results show that niching significantly enhances the performance, efficiency, and stability of the LES framework.\n\nOur work is expected to deepen the community's understanding of integrating LLMs with EC and inspire further innovative thinking. Although our contribution may seem incremental, it is a necessary step that lays the groundwork for more significant advancements in this field. We are confident that these improvements will lead to profound breakthroughs in the future.\n\n> ### Q2: Overpromising the flexibility and general validation of the PartEvo?\n\nWe would like to clarify our claim of the framework's flexibility and extensibility. When we state \"flexible and extensible (line 134),\" we mean that our niching-enhanced framework is compatible with the core concepts from works like EoH and ReEvo (e.g., PartEvo incorporates concepts such as \"Thought\" from EoH and \"verbal gradients\" from ReEvo). As future advancements are made, the niching-enhanced framework can continue to evolve and integrate new techniques.\n\nRegarding the general validation concern:\n\n(1) Features:  We have proposed two feasible features (CSV and TE) to validate our framework, demonstrating their value compared to random features, as shown in **Table 4**.\n   \n(2) Clustering Method: To intuitively demonstrate PartEvo's compatibility with various clustering methods, we conduct additional experiments considering three clustering approaches: K-means, Gaussian Mixture Model (GMM), and Spectral Clustering (SC). In this experiment, each version of PartEvo uses CSV features and the same initial population, with three independent runs for each method. The average results are presented below:\n\n|Methods|P1| P2 |P3 | P4 |\n|:-|:-:|:-:|:-:|:-:|\n|EoH | 1.846 | 800.98 | 7268.22 |19152.4|\n|PartEvo-K-means| 0.074 | 800.42 | 6539.24 | 6424.7|\n|PartEvo-GMM | 0.048 | 800.10 | 6580.82 | 5423.5 |\n|PartEvo-SC | 0.075 | 800.01 | 6543.54 |6714.2 |\n\nAs the table illustrates, PartEvo, regardless of the clustering method employed, consistently outperformed EoH across all tested scenarios. These results strongly confirm our claims regarding the framework's compatibility with various clustering methods.\n\nThese experimental validations indeed demonstrate that our framework is flexible and extensible, supporting our original claims.\n\n> ### Q3: Unclear statement regarding the use of two similarity metrics.\n\nWe sincerely apologize for any confusion caused by the unclear explanation. To clarify, all results presented in the main text are based on PartEvo using **CSV** as the feature, except for Table 4, which examines the impact of different features. We will provide a more explicit clarification of this in the Implementation Details section.\n\nThank you for bringing this to our attention. We hope this resolves any confusion. Additionally, as shown in **Appendix B.5** (lines 625-639), both CSV and TE versions of PartEvo exhibit excellent robustness.\n\n> ### Q4: Concerns about the benchmark setting and generalization ability of the learned algorithm.\n\nWe appreciate the reviewer’s concerns and would like to clarify our benchmarking efforts. We designed benchmarks with varying levels of difficulty to thoroughly evaluate PartEvo. Benchmarks P1 and P2 involve standard unimodal and multimodal test functions that are commonly used in optimization research, while P3 and P4 introduce more complex, real-world industrial optimization problems.\n\nAs detailed in **Appendix F**, we set different training and testing instances for each benchmark. For example, in P1 (see Table 13), the training instances consist of simple sphere functions, whereas the testing instances include more complex functions such as the logistic and Gaussian functions that are outside the training data. To further increase the diversity of the testing set, we also introduced shift terms. Additionally, to better assess the scalability of the learned algorithm, we incorporated a variety of complex test instances in P3 and P4. As shown in **Table 15**, these instances vary in terms of dimensions, scenarios, and optimization complexity.\n\nIt is important to note that **Table 1** shows that PartEvo’s algorithms perform well on the testing instances, indicating a certain level of generalization ability. We hope this clarification addresses your concerns. Furthermore, we fully agree with the reviewer on the importance of generalization and provide a more detailed response to this issue in **Q7**.\n\n> ### Q5: Broader discussion about comparisons between LES and Genetic Programming (GP) or Reinforcement Learning (RL).\n\nThank you to the reviewer for the insightful suggestion. Our main contribution lies in integrating EC techniques to enhance the LES paradigm, which is why our experiments focused on comparisons with existing LES methods. However, we welcome further discussion regarding the comparisons between LES and GP or RL.\n\nReference [1] provides a detailed comparison between GP and LES, showing that GP generally saves time and requires fewer resources in a single code generation step, as it does not involve interactions with LLMs. However, LLMs help overcome structural limitations and leverage external knowledge, improving code quality in a generation step and reducing the total number of generation steps, thereby accelerating problem-solving. Recent studies [2],[3] also emphasize that LLMs enhance the flexibility and intelligence of methods, decreasing reliance on human experts and improving performance, which goes beyond traditional resource concerns. Regarding RL, there are currently no known studies providing a direct comparison between RL and LES. However, we believe the LES paradigm's advantage lies in avoiding the need to retrain neural networks for each task, along with an interpretable algorithm design process.\n\nOverall, we agree that rethinking the relationship between GP, RL, and LES is valuable and merits future exploration. Our intention is not to compare them in a competitive manner but rather to acknowledge the significant potential of new tools like LLMs. These tools hold great promise and require further research, such as this paper, to optimize their utilization.\n\n[1]Hemberg, Erik, et al. \"Evolving code with a large language model.\" Genetic Programming and Evolvable Machines 25.2 (2024): 21.\n\n[2]Zheng, Kefeng, et al. \"CST-LLM: Enhancing Airfoil Parameterization Method with Large Language Model.\" Aerospace Science and Technology (2025): 110548.\n\n[3]Liu, Max, et al. \"Synthesizing programmatic reinforcement learning policies with large language model guided search.\" arXiv preprint arXiv:2405.16450 (2024).\n\n> ### Q6: Suggestions for expanding the review of related works in automated algorithm discovery.\n\nThank you for your insightful feedback. We fully agree that a more comprehensive and up-to-date review of the rapidly evolving field of automated algorithm discovery is essential. We will make sure to include and carefully examine the works you mentioned, along with other key references. This will provide a broader context and enable us to better position our approach within the existing literature.\n\nOnce again, thank you for your constructive suggestions. We are confident that these revisions will significantly strengthen the manuscript.\n\n> ### Q7: Expanding the discussion on limitations, including generalization boundaries.\n\nThank you for your constructive suggestion. We would like to clarify that we have already discussed several key limitations in the \"Discussion and Limitations\" section of Appendix A (lines 540-574), including the efficiency aspects in lines 562-574. We are also happy to further discuss other aspects, such as generalization boundaries.\n\nIn the LES paradigm, the automatic discovery process of algorithms relies on evaluations over training instances. Similar to supervised learning, training on fixed data may lead to overfitting. Currently, the algorithms learned by PartEvo perform well within the distribution of training instances but may not generalize equally well to instances that are outside this distribution. We recognize this limitation and are actively researching multi-distribution cooperative evolution to improve generalization. Additionally, many techniques developed in machine learning to address generalization issues can be adapted and applied within the LES paradigm.\n\nWe will include a more detailed discussion of these aspects in the revised manuscript. Once again, thank you for your insightful and professional suggestions. We believe that these clarifications will enhance the manuscript and provide a more comprehensive understanding of the work's limitations and future directions."}, {"review_id": "j7WNAnS7F4", "reviewer": "Reviewer_h8jf", "summary": "The paper proposes an LLM-assisted evolutionary search (LES) framework for Automated Algorithm Discovery (AAD) that incorporates niching-based collaborative search in the LES abstract language space. The proposed approach (PartEvo) combines verbal gradients with search strategies, leading to high performing algorithms for the AAD task. The method is evaluated on both synthetic and real data, showing its advantage over human-designed metaheuristic algorithms.", "questions": "1. Why did you consider k-Means algorithm and not other clustering methods?\n2. How internal parameters of PartEvo were selected? How sensitive is the method to their choice?\n3. Could you compare your approach with the one proposed in [1] (see Weaknesses)?", "limitations": "Yes", "rating": 3, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 2, "significance": 2, "originality": 3, "strengths_and_weaknesses": "Strengths:\n1. An interesting extension of LES algorithm through incorporation of EC-niching method.\n2. In experimental evaluation on 4 benchmark problems the methods performs comparably or better than the considered baselines.\n\n\nWeaknesses:\n1. The proposed approach lacks deeper investigation of parameter settings. For instance, what is the relation between $k$ and $m$, and how should their values be selected for a given problem at hand? \n2. The experimental evaluation is not sufficient. More diverse benchmarks (especially related to real-life problems) should be considered for a thorough assessment of PartEvo. Furthermore, mean and standard deviation results should be presented within Table 1 in the main paper.\n3. Besides standard population-based metaheuristics, PartEvo is evaluated only against two specialised methods, which does not allow for making definitive conclusions regarding its strengths and weaknesses. Other methods, e.g. [1], should be considered in the experimental assessment of the method. \n4. The contribution of the paper is not precisely stated. To my understanding it is an extension of LES approach by means of adding a niching algorithm, however the idea of niching in EC is clearly not novel. I understand that the novelty lies in the application of niching to a specific search space, however, this space specificity and the consequent difficulty of designing the niching method is not clearly explained in the paper.\n\n[1] ReEvo: Large Language Models as Hyper-Heuristics with Reflective Evolution, [NeurIPS 2024]", "was_revised": true, "final_justification": "My concerns are mostly addressed. Still, since the paper does not provide strong theoretical analysis, I would expect more thorough experimental evaluation. Also, the question of how to choose the algorithm’s parameters is not fully addressed. I’m raising my score to 3: Borderline reject.", "rebuttal": "Dear Reviewer and Area Chairs, \n\nThank you for your thorough review and constructive suggestions. We appreciate your insights and have carefully addressed all the concerns raised in the following five key points.\n\n> ## Q1：Why K-Means? Can PartEvo be compatible with other clustering methods? (Question 1)\n\nIn PartEvo, clustering serves as a tool for grouping individuals within the feature space. As we mentioned in lines 136-137, PartEvo is compatible with various clustering methods.\n\nWe specifically chose K-Means due to its simplicity and transparency, as noted in lines 169-170. This deliberate choice allows us to isolate and confirm that the observed performance improvements are primarily due to the effects of niche-based evolution, rather than influenced by the complexities of more advanced clustering methods.\n\nTo further demonstrate PartEvo's flexibility and compatibility, we conduct additional experiments considering three clustering methods: K-means, Gaussian Mixture Model (GMM), and Spectral Clustering (SC). Each version of PartEvo used the same feature mapping and initial population, with three independent runs per method. The experimental results are presented in the table below:\n\n|Methods|P1| P2 |P3 | P4 |\n|:-|:-:|:-:|:-:|:-:|\n|EoH | 1.846 | 800.98 | 7268.22 |19152.4|\n|PartEvo-K-means| 0.074 | 800.42 | 6539.24 | 6424.7|\n|PartEvo-GMM | 0.048 | 800.10 | 6580.82 | 5423.5 |\n|PartEvo-SC | 0.075 | 800.01 | 6543.54 |6714.2 |\n\nAs shown in the table, PartEvo consistently outperforms EoH across all benchmarks, regardless of the clustering method employed. These results strongly support our claims that PartEvo is compatible with various clustering algorithms.\n\nIn summary, (1) K-Means was chosen for its simplicity, transparency, and broad familiarity within the research community; (2) PartEvo can seamlessly integrate with other clustering methods, as demonstrated by the results presented above.\n\n> ## Q2: Internal Parameter Selection and Sensitivity (Question 2 and W1)\n\nWe appreciate your inquiry about the internal parameter settings and their sensitivity.\n\nFor parameter $m$, we adopted the settings established in the EoH paper to ensure a consistent baseline. As for parameter $k$, which determines the number of parents selected from other niches and influences the length of the prompts given to the LLMs, we set it to 2 based on empirical insights and practical considerations.  This choice helps avoid excessively long prompts, as longer inputs can reduce the performance of LLMs and potentially lead to hallucinations, as noted by Levy et al.[1].\n\nFurthermore, we conducted a detailed investigation into the effect of varying $K$ value under a fixed population size, as outlined in lines 313-332. The value of $K$ is crucial for balancing exploration and exploitation within the search process. As shown in Table 3, the trends for $K$ are consistent across both easy and hard problems. This suggests a practical approach: $K$ can be effectively tuned on simpler problems (where computational costs are lower) and then applied to more complex tasks, thereby enhancing the applicability of PartEvo.\n\nGiven the exploratory nature of this paper at the framework level, we have demonstrated the viability of the niche-enhanced framework. However, we acknowledge that a comprehensive analysis of the interactions between parameters has not yet been conducted at this stage. Despite this, our current parameter settings have yielded promising results on problems P1-P4 and the Online Bin Packing problem, demonstrating their applicability in supporting algorithm design for various tasks. Moving forward, we are committed to extending this analysis and providing a more comprehensive evaluation on a publicly accessible platform. Thank you for your constructive suggestion!\n\n[1] Levy, Mosh, Alon Jacoby, and Yoav Goldberg. \"Same task, more tokens: the impact of input length on the reasoning performance of large language models.\" arXiv preprint arXiv:2402.14848 (2024).\n\n> ## Q3: Concern about the sufficiency of experimental evaluation (Weakness 2)\n\nWe would like to clarify that we have used **five** distinct benchmarks with varying levels of difficulty to evaluate PartEvo, which we believe provides a robust and comprehensive evaluation. \n\nSpecifically, P1 and P2 are standard unimodal and multimodal test functions widely used in optimization research. **P3 and P4 represent more complex, real-world industrial scheduling problems**. Additionally, we conducted extensive experiments on the Online Bin Packing benchmark, which is detailed in Appendix B.4, further broadening the scope of our evaluation.\n\nRegarding the mean and standard deviation results, they are presented in Appendix B.2. We understand that greater accessibility is desirable, and in response to your suggestion, we will incorporate these detailed statistical results into the main paper in the revised version, making them more readily available to readers.\n\n> ## Q4: Comparison with ReEvo (Question 3 and W3)\n\nWe highly value the reviewer's suggestion to compare PartEvo with ReEvo. In response, we have included ReEvo as one of our primary experimental baselines for comparison. While ReEvo was previously incorporated as a baseline comparison for the Online Bin Packing (OBP) problem in Appendix B.4, its integration in the main experiments for P1-P4 significantly enhances the rigor of our study, offering a more comprehensive evaluation of PartEvo's capabilities.\n\nFor a fair comparison, we use the same initial population for ReEvo as for other LES methods and run it with the same sample budget to discover the algorithms for benchmark P1-P4. The average results from three independent runs are presented below (this is an expanded version of the original Table 8):\n\n| Problem | Instances | Funsearch | EoH    | ReEvo  | PartEvo  |\n|-|-|-|-|-|-|\n| **P1**  | Training  | 9.769     | 1.846  | 1.572  | **0.074** |\n|         | Testing   | 0.879     | 1.646  | 1.013  | **0.023** |\n| **P2**  | Training  | 800.09    | 800.98 | 800.00 | **800.00**|\n|         | Testing   | 439.31    | 368.95 | **364.19** | 365.20  |\n| **P3**  | Training  | 7042.21   | 7268.22| 6564.44| **6539.24** |\n|         | Testing   | 57696.53  | 61142.93| 57455.34 | **56753.77** |\n| **P4**  | Training  | 24289.3   | 19152.4| 11102.3 | **6424.7**  |\n|         | Testing   | 1.67${\\times}10^5$ | 77134.3 | 79795.2 | **62731.4** |\n\nConsistent with our previous findings in the OBP problem (lines 621-624), both PartEvo and ReEvo consistently outperform EoH and Funsearch across all benchmarks. This highlights the benefits of leveraging the \"verbal gradient\" concept, specifically by applying \"Reflection\" and \"Summary\" for enhanced algorithm discovery. Importantly, thanks to PartEvo's unique EC-inspired operators (such as CN and LGE), it outperforms ReEvo. This demonstrates that both \"verbal gradient\" and advanced EC techniques can enhance search efficiency and should be equally emphasized in future LES research.\n\n> ## Q5: Further Clarification on Paper's Contribution(Weakness 4)\n\nAs noted by other reviewers, we have indeed made efforts to clearly state our contributions. Here, we are pleased to offer further clarification for your consideration.\n\nOur primary aim is to introduce the niching as an example to demonstrate that established EC techniques can drive more sustainable progress in the LES paradigm. Importantly, our work goes beyond simply applying niching. We uncover the challenges faced when applying it, particularly when the search space is extended beyond traditional numerical or manually designed discrete spaces into abstract language spaces. To address this, we introduce feature-assisted niche construction to support the implementation of niching techniques within the LES framework. The experimental results clearly show that niching enhances the performance, efficiency, and stability of the LES framework. We hope that our work will encourage the research community to recognize that techniques refined over decades in EC can significantly contribute to the evolution of this emerging LES paradigm.\n\nFor your reference, we briefly discuss the abstract nature of the LES search space in lines 36-42. Furthermore, we explicitly describe the space specificity and the consequent difficulty of introducing niching in lines 101-106, with our proposed solution outlined in lines 107-110. We hope this clarification helps in understanding the contributions of our work.\n\n---\n\nIn summary, we hope that PartEvo serves as a significant step in bridging mature EC techniques with the emerging LES paradigm, offering valuable insights for the research community. We sincerely appreciate your constructive feedback and the time you dedicated to our work. We hope these responses address your concerns, and we look forward to your reconsideration of the rating."}, {"review_id": "kr6fD3dDCV", "reviewer": "Reviewer_DVie", "summary": "This paper presents PartEvo, a novel Language-based Evolutionary Search (LES) method that improves algorithm discovery efficiency. It introduces abstract search space partitioning to enable structured niche construction for algorithm discovery, integrating evolutionary computation techniques to enhance sampling resource allocation. PartEvo combines verbal gradients with local and global search strategies to discover high-performing algorithms, demonstrating significant efficiency gains through the fusion of evolutionary computation with LES. The method shows superior performance on algorithm automatic discovery tasks across both synthetic and real-world optimization problems, outperforming human-designed meta-heuristic baselines and proving the value of feature-assisted niche construction, advanced prompting, and niche-based sampling. PartEvo particularly excels in scenarios with limited sampling budgets by efficiently partitioning language search spaces and constructing niches that enable more effective allocation of LLMs queries.", "questions": "- ReEvo is also mentioned as an AAD approach. Why wasn't it included as a baseline in your experiment?\n- How does PartEvo perform regarding time and cost/token consumption?\n- Line 267, 280: \"EoH\" vs. \"Eoh\"—please be consistent.", "limitations": "Yes, in the appendix", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "**Strengthen**\n\n- Propose an approach integrating niche-based EC techniques that largely improve algorithm automatic discovery efficiency.\n- Comprehensive experiment results demonstrate that it out-performs existing baselines.\n\n**Weakness**\n\n- It would be helpful if the author could include experimental results on classical optimization problems such as TSP, job scheduling, etc.\n- Evaluation with LLMs of various capabilities would be necessary to determine how sensitive the approach is to different LLMs.", "was_revised": true, "final_justification": "Rebuttal of the authors mainly addressed my concerns, and I decide to keep the original positive score.", "rebuttal": "Dear Reviewer and Area Chair,\n\nThank you for your comprehensive review and valuable feedback. We appreciate your acknowledgment of the contribution our PartEvo framework makes to automatic algorithm discovery. Your insights have been instrumental in enhancing our paper. Below, we provide our responses to your questions and suggestions.\n\n> ## Question 1: ReEvo as a baseline\n\nWe sincerely appreciate the Reviewer's suggestion to include ReEvo as a baseline. As recommended, we have integrated ReEvo as a primary baseline for comparison in the main experiments for benchmarks P1-P4. While ReEvo was previously used for the Online Bin Packing (OBP) problem in Appendix B.4, its inclusion in the core experiments enhances the rigor of our study, providing a more comprehensive evaluation of PartEvo's capabilities.\n\nTo ensure a fair comparison, we use the same initial population for ReEvo as we did for other LES methods and run it with the same sample budget to discover the algorithms for benchmark P1-P4. The average results from three independent runs are presented in the table below (an expanded version of the original Table 8):\n\n| Problem | Instances | Funsearch | EoH    | ReEvo  | PartEvo  |\n|-|-|-|-|-|-|\n| **P1**  | Training  | 9.769     | 1.846  | 1.572  | **0.074** |\n|         | Testing   | 0.879     | 1.646  | 1.013  | **0.023** |\n| **P2**  | Training  | 800.09    | 800.98 | 800.00 | **800.00**|\n|         | Testing   | 439.31    | 368.95 | **364.19** | 365.20  |\n| **P3**  | Training  | 7042.21   | 7268.22| 6564.44| **6539.24** |\n|         | Testing   | 57696.53  | 61142.93| 57455.34 | **56753.77** |\n| **P4**  | Training  | 24289.3   | 19152.4| 11102.3 | **6424.7**  |\n|         | Testing   | 1.67${\\times}10^5$ | 77134.3 | 79795.2 | **62731.4** |\n\nConsistent with our previous findings in the OBP problem (lines 621-624), both PartEvo and ReEvo consistently outperform EoH and Funsearch across all benchmarks. This highlights the benefits of leveraging the \"verbal gradient\" concept, specifically through the application of \"Reflection\" and \"Summary\" for enhanced algorithm discovery. Importantly, thanks to PartEvo's unique EC-inspired operators (such as CN and LGE), it outperforms ReEvo. This demonstrates that both \"verbal gradient\" and advanced EC techniques can enhance search efficiency and should be equally emphasized in future LES research.\n\n\n> ## Question 2: Time and cost/token consumption\n\nIn terms of time consumption, the total time required for PartEvo to design algorithms can be broadly divided into two main components: LLM-driven algorithm generation and the evaluation of the generated algorithms. Typically, for a given number of queries, the evaluation time constitutes the majority of the total time, especially as the complexity of problem evaluation increases. For example, with a budget of 500 samples, the P1 benchmark, which involves three 20-dimensional standard test functions, requires approximately 1 hour. In contrast, the P3 benchmark, which includes two real-world scheduling instances, requires 2 to 3 hours. It is important to note that this still represents a significant efficiency gain compared to human experts, and the system can operate tirelessly.\n\nCost consumption is directly related to token usage and the specific LLM employed. Compared to EoH, PartEvo's \"Reflection\" and \"Summary\" operations (e.g., RE and SE operators) lead to greater token consumption than simpler crossover and mutation operations. For instance, the SE operator requires including the \"thoughts\" of all algorithms stored in the Sorted External Archive as part of the prompt for summarization by the LLM. When we set the archive size to 40, token consumption increases by 4 to 5 times compared to simpler operations like Crossover. (e.g., it requires about 1,500 tokens with the Crossover operator vs. 7,500 tokens with the SE operator).\n\nOverall, for real-world problems such as P3 and P4, a single run of PartEvo with 500 samples using GPT-4o-mini costs approximately 0.14 USD, which we consider acceptable. Furthermore, with anticipated advancements in computational power, this cost is expected to decrease in the future.\n\n> ## Question 3: Consistency in writing of EoH\n\nThank you for your careful observation. We will ensure consistency with the notation used in the EoH paper. We have corrected this issue and ensured that all relevant sections of the paper are now consistent.\n\n> ## Suggestion 1: Include experimental results on classical optimization problems\n\nWe sincerely appreciate this valuable suggestion. We agree that expanding our experimental results to cover a broader range of classical optimization problems would better demonstrate the universality of the PartEvo framework. We intend to include these problems in our future experiments and will share the findings on our open-source website soon. We believe that these additional experiments will further validate the applicability and advantages of the PartEvo, thereby boosting the community's confidence in the use of EC techniques within the LES paradigm.\n\n> ## Suggestion 2: Evaluation with LLMs of various capabilities\n\nThank you for your suggestion. We also recognize the impact of LLM capabilities on the overall algorithm discovery performance of PartEvo, as discussed in Appendix A.3. In response, we have added experiments using GPT-4o and GPT-3.5-turbo on the P4 benchmark, with an identical initial population, a 500-sample budget, and three independent runs. To improve readability, we have sorted the results of each model's three runs by performance for display. Below are the results for each run, the average, and the Standard Error of the Mean (SEM):\n\n| LLM  | Run 1  | Run 2  | Run 3  | Average | SEM |\n|:------------:|:------:|:------:|:------:|:-------:|:--------------:|\n| GPT-3.5-turbo      |  4924.5 |  9200.2 | 13261.9 |  9128.9 |       1965.360 |\n| GPT-4o-mini  |  4539.1 |  4679.5 | 10055.4 |  6424.7 |       1482.610 |\n| GPT-4o       |  3304.6 |  6028.0 |  8187.5 |  5840.0 |       1153.466 |\n\n\nAs shown in the table, we observe that the higher the LLM's capability, the better and more stable the performance of PartEvo. As mentioned in lines 541-545, as LLMs continue to evolve, we anticipate improvements in PartEvo's algorithm discovery capabilities.\n\n---\n\nOnce again, we sincerely thank the Reviewer for acknowledging our work and offering such valuable guidance. With these improvements, we believe our paper will be more widely recognized."}, {"review_id": "NqTlVakljU", "reviewer": "Reviewer_S91M", "summary": "The paper proposes \"PartEvo\", a novel framework integrating niche-based evolutionary computation (EC) techniques with Large Language Model-assisted Evolutionary Search (LES) for Automated Algorithm Discovery (AAD). Recognizing the unique challenges posed by abstract language search spaces in LES, the authors introduce a feature-assisted niche construction method to efficiently partition and explore these spaces. The contributions include a general LES framework, the PartEvo methodology, and comprehensive evaluations demonstrating its superiority over existing human-designed meta-heuristics and other LES methods (e.g., Funsearch and Eoh) on both synthetic and real-world optimization tasks.", "questions": "1. I have read the prompts in the appendix and seen the demonstration benchmark functions, but I am still unclear about how these elements work together in practice. Could you please provide a concrete example showing exactly how the prompts are combined with the benchmark tasks in your pipeline?\n2. Is it possible to extend this framework towards training a dedicated model, instead of using only a frozen LLM with prompting? For instance, could future work explore jointly fine-tuning or adapting the LLM during the evolutionary search?", "limitations": "Yes.", "rating": 4, "confidence": 1, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 2, "significance": 2, "originality": 3, "strengths_and_weaknesses": "**Strengths**\n\n1. Novel Integration: The paper uniquely integrates niche-based EC methods with LES, addressing effectively the complexity and abstractness of language-based search spaces.\n2. Superior Performance: Experimental results consistently demonstrate PartEvo's superior performance across multiple benchmark tasks, significantly outperforming existing human-designed and LES-based methods.\n3. Robustness: PartEvo demonstrates strong stability and robustness, achieving consistent high-quality outcomes across varied experimental setups and diverse optimization tasks.\n\n**Weaknesses**\n\n1. The authors could provide more specific examples to better illustrate the proposed method and its advantages clearly.\n2. The absence of publicly available resources or clear reproducibility guidelines may limit validation and adoption by the broader research community.", "was_revised": false, "final_justification": "", "rebuttal": "Dear Reviewer and Area Chair,\n\nThank you for your valuable feedback and insightful comments. We genuinely appreciate your review and are pleased to address your questions and suggestions in detail below.\n\n> ### 1.  A Concrete example of how PartEvo works (Question 1 & Weakness 1)\n\nThank you for requesting a clearer illustration of PartEvo's process. To help readers quickly grasp our methodology, we've now included easy-to-understand pseudocode for PartEvo:\n\n| Line | Description |\n|:-:|:-|\n| | **Input**: Task Description, Training Instances |\n| 1:| Initialize the algorithm population (generated directly by LLM or load existing algorithms) |\n| 2:| Construct algorithm *niches* through feature-assisted niche construction |\n| 3:| &nbsp;&nbsp;&nbsp;&nbsp;**For each $niche$ in *niches*:** |\n| 4:| &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;**For each *operator* in *operators*:** |\n| 5:| &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;Select parent algorithms from *niches* |\n| 6:| &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;Construct prompt (including parent algorithms, task description, and specific instructions related to operator) |\n| 7:| &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;LLM designs offspring algorithm based on the prompt |\n| 8:| &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;Evaluate the offspring algorithm on the training instances |\n| 9 :| &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;Manage the offspring algorithms  based on evaluation results |\n| 10 :| &nbsp;&nbsp;&nbsp;&nbsp;Exit if termination condition is met |\n| 11 :| **Output**: The Designed Algorithm |\n\nTo clarify the process, let’s take Benchmark P1 as an example. Our goal here is to discover powerful algorithms that can efficiently solve unimodal optimization problems.\n\nAs illustrated in the pseudocode, we start by defining the task (unimodal optimization) and preparing a set of training instances. These instances serve as \"practice questions\" for PartEvo, enabling us to evaluate whether the generated algorithms are improving. The training instances can be representative unimodal problems or all standard unimodal functions. They provide a consistent way to rank and manage candidate algorithms as they evolve. To assess the generalization ability of the generated algorithms, we use separate training and testing sets. For our experiments, we employ three functions as training instances.\n\nPartEvo continuously optimizes algorithms through an evolutionary process. Each evolutionary loop involves selecting parent algorithms, constructing prompts, generating offspring algorithms, and then evaluating and managing these offspring. Specifically, the generation of offspring algorithms is driven by an LLM, guided by the prompts. The prompt construction follows the template detailed in Appendix C. As PartEvo iterates, the algorithms progressively improve, ultimately leading to high-performing algorithms tailored for unimodal optimization problems.\n\nThis process is applied to all our other benchmarks and even real-world applications: simply define the problem and provide relevant training instances, and PartEvo can design a tailored algorithm. We hope this explanation clarifies how PartEvo works in practice.\n\n> ### 2. Extending PartEvo towards training a dedicated model, instead of using only a frozen LLM with prompting (Question 2)\n\nThis is a remarkably insightful and forward-thinking suggestion! As mentioned in our paper (lines 579-580), the automated algorithm discovery within PartEvo generates a lot of valuable data on algorithm design, including step-by-step creation processes and various solution attempts for specific problems. This rich data can be used offline to fine-tune LLMs, further enhancing their capabilities in algorithm design.\n\nAn exciting future direction is the concept of training the LLM \"on the fly\" as the evolutionary search progresses. Recent research, such as [1], has begun exploring this idea, and we also plan to investigate this in our future research. We believe this approach could enable us to create a smaller, specialized language model that, even with a reduced scale, could achieve powerful algorithm discovery abilities when combined with our evolutionary framework. This would be a significant step toward reducing the computational demands for LES-based algorithm design, making it more accessible and valuable to the wider community.\n\n[1] Huang, Ziyao, et al. \"Calm: Co-evolution of algorithms and language model for automatic heuristic design.\" arXiv preprint arXiv:2505.12285 (2025).\n\n> ### 3. Publicly available resources （Weaknesses 2）\n\nThank you for this valuable suggestion. We fully agree that making our work accessible is essential. We have already organized PartEvo's code, along with the benchmarks and scripts needed to reproduce our comparison experiments.\n\nWe plan to make our GitHub repository public as soon as the NeurIPS decisions are announced. This will enable other researchers to easily verify our findings, gain a detailed understanding of our methods, and build upon our work. Additionally, we intend to expand our experiments to cover a wider range of problems, providing more comprehensive insights. We genuinely hope that our work inspires further research and serves as a valuable resource for the community.\n\n---\n\nIn summary, we deeply appreciate the reviewer's constructive feedback, which has helped us significantly enhance the clarity and impact of our paper. We are committed to incorporating all the revisions and clarifications detailed above into our final revised version."}]

Author rebuttal:

Meta-review: ## Summary

This paper studies the use of large language models for automated algorithm design through evolutionary search. The authors introduce a framework that extends language model-assisted evolutionary search with feature-based niche construction, allowing methods from evolutionary computation to be applied in abstract program spaces. Based on this framework, they propose PartEvo, which integrates collaborative niche search with specialized prompting strategies. The method is evaluated on synthetic benchmarks and real-world optimization tasks. Results show that PartEvo improves upon earlier approaches such as EoH and Funsearch, and it achieves strong gains on resource scheduling problems, where it produces competitive meta-heuristics at low design cost. The findings indicate that niche-based strategies can enhance the effectiveness and practicality of evolutionary search with language models.

## Decision

This paper addresses the timely and important topic of automated algorithm discovery using evolutionary algorithms and large language models (LLMs). The problem itself is both significant and impactful. The paper is well-written, easy to follow, and the motivation is clear, supported by strong experimental evidence. The proposed approach is novel, and the results consistently demonstrate PartEvo’s superior performance across multiple benchmark tasks, substantially outperforming both human-designed baselines and prior LES-based methods. While the reviewers raised several important concerns during the rebuttal stage, the authors responded effectively and addressed them convincingly. I recommend this paper for acceptance, with the suggestion that the authors incorporate the additional experiments discussed during the rebuttal into the final version of the paper.
Decision: Accept (poster)