Generalizable Heuristic Generation Through LLMs with Meta-Optimization
Authors: Yiding Shi
Jianan Zhou
Wen Song
Jieyi Bi
Yaoxin Wu
Zhiguang Cao
Jie Zhang
Venue: ICLR 2026 Poster (2026)
Keywords: Combinatorial Optimization
Large Language Models
Heuristic Generation
Primary area: optimization

Abstract: Heuristic design with large language models (LLMs) has emerged as a promising approach for tackling combinatorial optimization problems (COPs). However, existing approaches often rely on manually predefined evolutionary computation (EC) heuristic-optimizers and single-task training schemes, which may constrain the exploration of diverse heuristic algorithms and hinder the generalization of the resulting heuristics. To address these issues, we propose Meta-Optimization of Heuristics (MoH), a novel framework that operates at the optimizer level, discovering effective heuristic-optimizers through the principle of meta-learning. Specifically, MoH leverages LLMs to iteratively refine a meta-optimizer that autonomously constructs diverse heuristic-optimizers through (self-)invocation, thereby eliminating the reliance on a predefined EC heuristic-optimizer. These constructed heuristic-optimizers subsequently evolve heuristics for downstream tasks, enabling broader heuristic exploration. Moreover, MoH employs a multi-task training scheme to promote its generalization capability. Experiments on classic COPs demonstrate that MoH constructs an effective and interpretable meta-optimizer, achieving state-of-the-art performance across various downstream tasks, particularly in cross-size settings. Our code is available at: \url{https://github.com/yiding-s/MoH}.

Published as a conference paper at ICLR 2026
GENERALIZABLE HEURISTIC GENERATION THROUGH LLMS WITH META- OETAAOPPIIINATION
Yiding Shi', Jianan Zhou'r, Wen Song" Jieyi Bi', Yaoxin Wu', Zhiguang Cao Jie Zhang' Nanyang Technological University *Shandong University @Singapore Management University *Eindhoven University of Technology yiding002@e. ntu. wtw..du..u........ edu.sg sg jianan004@e edw..du.... sg wensongeemail. sdu. adu.edu.com en jieyi y wu2@tue. nl zgcao smu, edu.sg zhangj@ntu. edu.
wtw. eud
ABSTRACT
Heuristic design with large language models (LLMs) has emerged promising approach for tackling combinatorial optimization problems (COPs). However, ex- isting approaches often rely on manually predefined evolutionary computation (EC) heuristic-optimizer and single-task training schemes, which may constrain the ex- ploration of diverse heuristic algorithms and hinder the generalization of the result- ing heuristics. To address these issues, we propose Meta- Optiimiztiiiiinnn of Heuristics (MoH), a novel framework that operates at the optimizer level, discovering effective heuristic-optimizers through the principle of meta-learning. Specifically, MoH leverages LLMs to iteratively refine meta-optimizer that autonomously con- structs diverse heuristic-optimizzr the reliance on a predefined EC heuristic-optmmizer These constructed heuristic- optimizers subsequently evolve heuristics for downstream tasks, enabling broader heuristic exploration. Moreover, MoH employs a multi-task training scheme to promote its generalization capability. Experiments classic COPs demonstrate that MoH constructs an effective and interpretable meta-optimizer, achieving state- of-the-art performance across various downstream tasks, particularly in cross-size settings. Our code is available at: https: / //gittutbbb// com/ viding-a s MMoH.
through (self- jinvocation, thereby eliminating
INTRODUCTION
Heuristies have long been integral to solving combinatorial optimization problems (COPs), offering practical and efficient approaches when exact methods become computationally intractable due to their exponential time complexity. Over the past few decades, substantial progress has been achieved in human-designed heuristics. Notable examples include the Lin- Lerrennig Heuristic (LKH) (Lin Kernighan, 1973) for the Traveling Salesman Problem (TSP) and the Best Fit heuristic (Johnson et al., 1974) for the Bin Packing Problem (BPP). However, developing effective heuristics for COPs typically requires an in-depth understanding of each problem's unique structure and the specialized expertise to craft suitable heuristic strategies. As result, the traditional approach to heuristic design is both time-intensive and significantly dependent on expert knowledge. This underscores the growing demand for more powerful approaches accelerate the development of effective heuristics for COPs.
With the explosive advancements in large language models (LLMs) in recent years, the landscape of heuristic design has undergone transformative shift (Jiang et al., 2024a; Liu et al., 2024b). A prominent trend involves leveraging LLMSs to generate effective heuristics aimed at solving NP-hard COPs. Specifically, these methods typically utilize in-context learning prompt L.MMs to produce heuristics, which subsequently become integral components of (meta-)heuristeee Jheuristic or learning- based solvers. Romera-Paredes et al. (2024) first demonstrated the feasibility of applying LLMs heuristic design in this domain. Building on this foundational work, recent approaches have increasingly integrated LLMs with evolutionary computation (EC), giving rise to LLM-EC frameworks (Liu et al., 2024a; Ye et al., 2024; Dat et al., 2024; Zheng et al., 2025). These methods enhance heuristic design by using LLMs to carry out evolutionary operations like crossover and mutation to evolve heuristics.
Corresponding authors.

---

Published as a conference paper at ICLR 2026
Despite achieving promising results, existing LLM-EC approaches face two limitations. First, their search space is constrained by manually designed, predefined EC heuristic-optimizer (e.g., a fixed workflow of crossover followed by mutation), which may restrict the exploration of diverse heuristics and ultimately hinder the discovery of more powerful heuristics (Dat et al., 2024). Second, their optimization process is only designed for single task (i.e., fixed-size COP), which may limit the generalization of the evolved heuristics. Fig. illustrates the generalization performance of EoH (Liu et al., 2024a), a representative LLM-EC approach, in optimizing improvement heuristics for TSP under various training settings. The results indicate significant generalization challenge, as performance gaps widen with increasing problem size. Although incorporating cross-size datasets during training can partially mitigate this issue, the overall performance remains suboptimal on large problem sizes (i.e., different tasks).
To overcome these inherent limitations, we intro- duce Meta-Optimizztion of Heuristics (MoH), novel framework leveraging the in-context reasoning and refinement capabilities of LLMs (Huang et al., 2022; Zelikman et al., 2024) to automate optimizer design. In this paper, we categorize optimizers into heuristic- optimizers and meta-optimizers based on their respec- tive roles. Heuristic-optimizers are algorithms, such as traditional EC frameworks, that are used to gen- erate or refine heuristics for COPs to improve solu- tion quality, whereas meta-optimizers are higher-level procedures that adapt and enhance these heuristic- optimizers. Technically, MoH implements an iterative meta-optimization module within multi-task frame- work to encourage both exploration and generalization. At each iteration, the meta-optimizer generates diverse population of candidate heuristic-optimizers through (self- jivvoatinooc...n... The most promising heuristic-opiimizer, evaluated by its effectiveness on downstream tasks in optimizing task-specific heuristics, is selected to become the meta-optimizer in the subsequent iteration. By doing so, the heuristic-optimizzr are improved to generate more effec- tive heuristics (see Fig. 2). With its innovative meta-optimization, MoH extends beyond traditional fixed EC optimization frameworks, facilitating broader exploration of the heuristic search space and operating higher abstraction level than existing approaches.
Trained TSP100
Fdil Trained on TSP300
Tralned TSP510
Kall Tratned on 18P1001
Cal 2 Onimma
Kall Trained TSP20-200
Mall Texised on TSP18-200
28
50
100
200
580
1080
Size of TSP
Figure 1: Generalization performance of the evolved improvement heuristics for TSP.
Our contributions are summarized as follows: 1) We propose MoH, a novel framework that highlights meta-optimization for producing effective COP heuristics. MoH enables broader heuristic exploration by autonomously discovering effective optimization strategies through an iterative meta-opuimization module, thereby addressing inherent limitations of existing LLM-EC approaches. 2) We position MoH within multi-task training framework to enhance its generalization capability to unseen tasks. 3) Extensive experiments across multiple heuristic algorithms and classical COPs demonstrate that MoH is able to generate effective and interpretable meta-optimizers that consistently outperform baselines. Notably, the resulting heuristics exhibit strong performance on large COP instances.
2 PRELIMINARIES
In this section, we first introduce three canonical COPs, TSP, BPP and CVRP, followed by an introduction to existing LLM-EC approaches and high-level comparison with our proposed MoH.
Traveling Salesman Problem. Traveling Salesman Problem (TSP) is a well-known COP (Applegate, 2006). A TSP instance is defined over complete graph G {VV &}. where V ((1 (Vi..... ......., Un. is the set of cities and E (e(v;, [0,], By E V, & j} is the set of edges, representing possible travel routes between cities. Each edge e(e,, v2) is associated with distance d,j. where d V x V - Rt defines the travel cost between any pair of cities. The objective is to find a Hamiltonian cycle (i.c., a permutation of V that starts and ends at the same city) with the minimum total travel cost, subject to the constraint that each city is visited exactly once before returning to the starting city.
Bin Packing Problem. The Bin Packing Problem (BPP) aims to pack set of items (i1, &n each with an associated weight wi, into a minimum number of bins of fixed capacity C, such that the total weight of items assigned to each bin does not exceed C. BPP can be categorized into online and
2

---

Published as a conference paper at ICLR 2026
Update Optimizer Population
Optistar Pepudaation
Nkerrstic-Opiimiiin M
Heurbdic- Optimioer
Heuristie-ppiiiiae
Heuristi Optiniterr I
Update Houristic Pepulacion for
Heuriati-- Optimber 2
Heurratic!
Okj-Ulllt
Trals
Meta- Optimizer
for Tank
Wingllal by 1.41
Heuristic Optimiaur
Houristle- Optimiter
Heuristie
0.48
fuility
8
Pronupt for Tark
:
Protpp thr Netacipitnter
sing Gdlvetng compaiaa. You sdlaton auudd out andia undeggaa make anatidyy
solss will belr socdiien sepoode Halp us de clans agoine but
Heuristic
Okj-Uuiity
be Secastu
dii yostiai)
skkrtta cemoom dook
kilo adliiaaionn
cOPTINIZATION STEPS]
Outer Loop
Inner Loop
Figure 2: Overview of MoH. In iteration f. the current meta-optimizer Ii-1 Ii-1 heuristic-optinn opprmizic-- in the outer loop. Each candidate cerrristc-optmizeertee is then evaluated through the inner loop, where it generates K heuristics that are applied to N downstream tasks. For each task, the best heuristic is selected, and its utility contributes to the overall utility of the heuristic-optimizzr After aggregating utility scores across all tasks, the heuristic-optimizer with the highest utility is selected as the new meta-optimizer Zí.
generates M candidate
offline variants depending on the availability of item information. In the online BPP (Seiden, 2002), items arrive sequentially in unknown order. For each arriving item, placement decision must be made immediately and irrevocably without knowledge of future items. The objective remains to minimize the total number of bins used while respecting capacity constraints. In contrast, the offline BPP (Coffman Jr et al., 1984) assumes complete prior knowledge of all item sizes and weights, enabling global optimization over the entire item set. This setting allows algorithms to exploit full information to search for higher-quality packing configurations.
Capacitated Vehicle Routing Problem. The Capacitated Vehicle Routing Problem (CVRP) is also classical NP-hard combinatorial optimization problem widely studied in the fields of logistics and operations research (Toth & Vigo, 2002). It builds upon the classic TSP by incorporating the crucial real-world constraint of limited vehicle capacity. Specifically, a CVRP instance can be defined over a complete graph 9 = (V U up, 3), where V (vI,..., Un denotes the set of customer nodes, to represents the depot, and & = {e(c;, Uy) Uj E VU co. i A j} is the edge set that includes all the possible travel routes between any two nodes, either customers or depot. Each edge e(v,, U2) is associated with non-negative travel cost or distance dij, where d : (V U oo) x (V vo) + R+ defines the travel cost between any pair of two nodes. Each customer v; E V has a demand q: > 0, while the depot has qo = O. The fleet consists of m vehicles, each with capacity Q. The objective is to determine a set of m' routes (usually m' < m) with minimized total travel cost across all routes, while satisfying the following constraints: 1) each route starts and ends at the depot, 2) each customer is visited exactly once by a single vehicle, and 3) the total demand on any route does not exceed Q.
Existing LLM-EC Approaches. In the domain of LLM-driven automatic design and generation of executable heuristics, early approaches (Liu et al., 2024a) leverage a fixed EC heuristic-opimmizer to discover effective heuristics through LLMs for solving COPs. Specifically, they maintain fixed- size population of heuristics tailored to a specific COP task. The EC heuristicoopimizzr refines this population by iteratively selecting promising candidates and applying crossover and mutation operations to generate improved heuristic variants. While this method can quickly converge to reasonably good heuristics, its performance may constrained by the insufficient exploration of the vast search space. Subsequent works propose various EC variants, such as incorporating a reflection mechanism (Ye et al., 2024) or integrating with Monte Carlo Tree Search (MCTS) (Zheng et al., 2025), however, these methods still suffer from limited exploration or high computational cost. In summary, existing approaches primarily focus on heuristic design using a fixed EC heuristic- opttiiiee whereas MoH targets optimizer design, going a step further by enabling automatic design of such high-level optimization frameworks themselves. This allows for more flexible and potentially more effective heuristic generation for downstream tasks, illustrated in Fig. 2. We believe our approach
3

---

Published as a conference paper at ICLR 2026
offers fresh insights into the field by introducing conceptually and methodologically meaningful advancement in the use of LLMs for combinatorial optimization, addressing both optimization framework discovery and heuristic evolution in unified and scalable manner.
3 METHODOLOGY
An overview of MoH is shown in Fig. 2, which features two-level optimization process: an outer doop for optimizer design and an inner loop for heuristic design. Inspired by recent advances in LLMs (Zhou et al., 2022; Zelikman capable of generating effective optimization strategies and improving heuristics on downstream tasks concurrently. Concretely, in the outer loop, the meta-optimizer produces diverse population of candidate heuristic-optimizers. Then, each generated heuristic-optimizer is leveraged in the inner loop to evolve task-specifie heuristics for downstream tasks. After evaluating the heuristics the validation dataset, the candidate heuristic-optimizer with the highest utility score selected as the new meta-optimizzr for the next iteration, enabling MoH to iteratively discover increasingly effective optimization strategies. Moreover, MoH is inherently suited for a multi-task training setting by maintaining diversity among tasks, thereby enhancing its ability explore a broader range of heuristics, leading to improved performance across diverse tasks. Examples of seed (or initial) and generated meta-optimizers are provided in Appendix In the following sections, we present the technical details of our proposed MoH framework.
al., 2024), MoH aims to construct meta-optimizer
3.1 PROBLEM FORMULATION
Suppose there are N downstream tasks, each corresponding to heuristic design task for a COP. For each task i, let he denote the heuristic found by the heuristic-optimizer I, DD the validation dataset, and U, D.) the heuristic utility function evaluating the performance of lif on D. The objective of heuristic design is discover the best heuristic hī using heuristic-optimizer I:
arg max U. D,), AFEW;
where H4; denotes the heuristic search space, comprising all possible heuristics for the task i. The heuristic utility function (h D,) is defined the negative of the solution optimality gap. Most studies so far integrate EC as the heuristic-optimizz While these LLM-EC approaches offer a certain degree of flexibility, they struggle effectively explore the vast heuristic search space due to the rigid structure of the fixed heuristic-optimizer I. Additionally, their heuristic design process necessitates separate training for each task i, making computationally expensive. An alternative is to incorporate diverse instances from N tasks into the training dataset. However, this simple mixture of data results in suboptimal performance, as a single COP heuristic usually struggles adapt effectively across different tasks, consistent with the No Free Lunch Theorem (Wolpert & Macready, 1997).
I within LLMs to perform heuristic design.
To address the limitations, MoH directly searches for optimizers rather than relying fixed one, ie., the optimizer design process. The objective of optimizer design is formally defined as:
I - arg max I
Σ w. U.(h7 D.), with h arg max (1 D.). h EH:
(2)
where I is the meta-optimizer, and uy is the task weight defined as Σラ with represent- ing the problem size of the i-th downstream task. In essence, MoH extends Eq. (1) by introducing an outer loop for meta-optimization. In this outer loop, the meta-optimizer I' produces a population of candidate heuristic-optimizers through (self-)invocation. The best heuristic-optimizer I, as evaluated by the optimizer utility function U(ī) = Σi Wi U, D:), is then selected serve as the new meta-optimizer I" in the next iteration.
3.2 OVERALI WORKFLOW
We summarize the MoH training workflow in Alg. I and detail each step as follows. We initialize each downstream heuristic design task with heuristic population H.. This is achieved by prompting
4

---

Published as a conference paper at ICLR 2026
Algorithm 1: MoH Training Workflow
Input: Number of downstream tasks N, Number of iterations I', Seed meta-optimizer Zo; Output: Meta-optimizer Z7., Heuristic populations H. across all tasks;
Function U (I):
<-0 0
for i ,,...,,N
,N do
b < I(H,, LLM, Prompt, "Task i"):
Steps for heuristic design with heuristic-optmmizer I:
a. Generate a group of heuristics using I - (kf,1..... k
b. Evaluate the utility score of each heuristic through U, Dz), Ak E [1,
c. Update H, = Top U ......... h k by utility and return the best heuristic h
2 f- u + W; U, Dj)
return u
Initialize heuristic and optimizer populations H (H,, P {Io}
for f = 1......,77 do
I < (P, U(., LLM, Prompt, "Optimizer"):
Steps for optimizer design with meta-optimizer I;-1:
a. Generate a group of heuristic- opttiize via (self- jinvocatinn of -7-1 - {II II. IM
b. Evaluate the utility score of each heuristic-optimzzer through U(I)), Vj [M
Update p TP U (I).... ..... IIM by utility and return the best heuristic-optmmizer I,
I +- I, return 17, H
LLMs to generate diverse heuristic ideas in natural language, accompanied by their corresponding code implementations. We also initialize an optimizer population using a given seed meta- optimizer (see Fig. 8), which serves as the starting point and reference baseline for subsequent iterations. Concretely, at iteration f, the current meta-optimizer I is used to generate set of candidate heuristic-opiimizers (Z:..... .. I.M of I using LLMs, with prompts constructed from the information in the optimizer population P. Then, each candidate coptistic-opiimizer Ił employs its own LLM-generated optimization strategy to evolve heuristics across all downstream tasks, resulting in updated heuristic populations for all
This generation accomplished via (self-) vivvociinoo
N downstream tasks (i. JJw After evaluation, the best heuristic for each task is collected, yielding (h ..... bN }. The utility score each candidate heuristic-optimizer is thus
calculated as U(I,) Σm wi (h, D.). The candidate heuristic-optimizer with the highest utility score is selected the meta-optimizer Zí for the next iteration. More details on population management can be found in Section 3.3. During inference, the meta-optimizer I can be deployed on new tasks that differ from those encountered during training, such as tasks with larger problem sizes. By performing several rounds of heuristic design using Tj., MoH can yield an effective heuristic tailored to this task.
3.3 DETAILED IMPLEMENTATION
As key subroutines of MoH, we further elaborate on the optimizer and heuristic generation (i.e., step (a) in the heuristic and optimizer design processes in Alg, 1). The main entities are as follows.
Individual and Population Structure. In the MoH framework. an individual is defined as structured entity comprising three components: 1) code implementation (String), 2) high-level natural language description of the core strategy (String), and 3) utility score reflecting its performance (Float). This unified individual format is adopted for both heuristics and optimizers. To ensure diversity and stability, we preserve a population with 10 individuals for both heuristic populations H. and optimizer population P.
5

---

Published as a conference paper at ICLR 2026
Optimizer Signature
def pptimize_algericee population: dict, utility: callablelldict), float], language_modell class 'ianquageModel" subtask_prompt: stz, subtask: str -> Tuple(str, str, float]:
This signature applies to meta- opttaizptiizers and heuristic-opiin oprriiie-.opimizerss.... Its detailed implementa- tion generated by LLMs, enabling recursive or iterative refinement of optimization strategies.
Figure 3:
Population Management, Individuals in the population are ranked by their utility scores. When a new candidate arrives, it is compared to the current worst-performing individual. If the candidate's s utility score is higher, it replaces that individual. After each insertion, the population re-sorted to maintain the utility-based ranking. This structure is efficiently managed using a heap.
Optimizer Signature. As shown in Fig. 3, the optimizer formatted as a callable function that takes the following inputs: 1) population: the population structure defined above. 2) utility: utility function that evaluates the performance of an individual and returns its utility score, 3) an LLM for generating heuristics heuristic-optimizens, 4) subtask_ pooopsskppppp::::-- task-specific prompt to guide the optimization, and 5) subtask: a string specifying the name of the task. The optimizer function returns the best individual discovered during the optimization process. Notably, it is designed to support recursive invocation, allowing it to take its own implementation as input through the population parameter in the first outer loop iteration.
language_mon
Optimizer Generation Procedure. The heuristic-opiimizer generation procedure in the outer loop iteration follows the standardized steps below: 1) Individual Selection: The current meta-optimizer Ii- uses its LLM 3LM-generated strategy to select promising candidate heuristic-optimizer from the optimizer population P. This step aims to balance the exploitation high-utility individuals with the exploration of diverse candidates. 2) Idea Generation: The iterative improvement of algorithms generated by LLMs critically depends on algorithmic reasoning articulated in natural language, as evidenced by (Wang al., 2024). Consequently, the meta-optimizer is encouraged to prompt LLMs to propose exploratory or refinement ideas based on the heuristic- neuristic-optimmzers selected the first step. 3) Implementation Generation: Guided by the generated ideas and task-specifer prompts, the LLM refines or generates new code implementations of selected optimizers through vivvociinoo of I producing a set of candidate heuristic-optimizers {T, ..., I.M). Each candidate is evaluated using the optimizer utility function U(.), and the best-performing heuristic-optimizer is selected as the new meta-optimizzer Σí for the next iteration. Note that our optimizer structure enables flexible exploration of various optimization strategies. While the specific behavior of a generated cearistti-opiimizze may vary depending on prior heuristic-oplin cppristic-ptimmizrss prompts, and LLM versions, their procedures generally adhere to the above three steps. Appendix E presents examples of LLM- generated meta-optimizers, some of which resemble traditional metaheuristics, while others exhibit hybrid or unconventional strategies.
I.M,,,IM)
Heuristic Generation Procedure. Given a heuristic-opiimizr T/ generated by the metaioopi Ti_1 in outer loop iteration I, the inner loop heuristic generation procedure for downstream task i follows the standardized steps below: 1) Individual Selection: The heuristic-optimizer I employs its LLM-ggnerrted strategy to select promising candidate heuristies from H. for evolution. 2) Idea Generation: The heuristic-optimiizr Z prompts LLMs to propose exploratory or refinement ideas based on the heuristics selected in the first step. 3) Implementation Generation: Guided by the generated ideas and task-specific prompts, the LLM generates or refines heuristic implementations,
resulting in a set of candidate heuristics (h;,i..... h,,kk Each candidate heuristic is evaluated using its corresponding heuristic utility function (.).), and the heuristic population H, is updated thereafter.
Despite their procedural similarities, the key differences between optimizer generation in the outer loop and heuristic generation in the inner loop are as follows: 1) Optimization Target: The outer loop focuses generating heuristic-optimizers using the meta-optimizer, whereas the inner loop

---

Published as a conference paper at ICLR 2026
Table 1: Results of constructive and improvement heuristics on TSP.
Train
Generalization
Methods
20
100
200
1000
Average Gap
Obj.
Cup
Ohjj..
Clap 1
(ijj 1
Gopp 1
Chij
Gipp1
Cip 1
(hi
Gip..
Concoed:
3,400
5.715
7.266
10,679
16,519
23.104
OR-Tools
3840
5.715
0.001S
7.772
10.944
2.478%
17.259
4.479%
24,362
5.011%
2.01U1
Nearest Neighbor
4.602
19.805%
7.055
23. 33.400%
9.626
24.07216
13.374
25.228%6
20,691
25.25216
28.990
25.474%
23.87396
Consrrctioon
Reariviic
Punsearch
4.261
11.000%
6.523
14.162%
9.018
16.109%
12.615
18.143%
19.531
18.242%
27.571
19.332%
16.165%
Boft
4.204
9.408%
6.402
12.007%
8.774
12.994%
12.233
14.548%
19:009
15.196%
26.
16.390%
13.420%
ReFvo
4117
9.250%
11.966%
8.756
13.139%
12.217
14,40%%
19,035
15.332%
36.818
16.0766
13.349%
HSEvo
4.108
6.897%
6.290
9,881%
8,705
12.102%
12.206
14.3211
19,550
18.349%
27.431
18.72756
13.37998
MCTS
4.107
6.882%
10.807%
8.735
12,49915
12.165
13,921%
190036
15.240%
26.514
12.568%
Moll (Ouns)
4.104
6.837%
6.280
9.893%
8.654
11.461%
12.100
13.307 %
18,869
14.224%
26.581
15,049%
11.792%
Improvmment
Haurisii
BaIl-GLS
3.800
0.000%
5.215
0.000%
7.268
0,024%
10.716
0.342%
16,714
1.176%
23.747
2,78116
0.721%
HSEve-GILS
3,400
0.001%
5.715
0.000%
7.768
0.078%
10.715
0.328
16.729
1,2665)
23.719
2.6608
0.7144
ReEvs-GLS
3,840
010000%
5.715
0.0006
7.768
0.0211
10.715
0.331%
16.741
1.3445
23.731
2.715%
0.735%
MoBl (Ouss)
3,800
0.000%
5.715
0,000%
7.767
0.012%
10.711
0.291
16,674
0.936%
23.445
1.476%
0.453 %
R
0.000%
0.000%
7,766
0.0031%
0.221%
0.97654
23.473
1,505%
0.4665%
#SEw-KGILS
3,800
0.00005
5.715
0.000%
7.267
0.0044
10.704
0.2211%
16.678
0,958%
23.478
1.615%%
0.466%
MCTS MCTS-AHD-RRE KGLSS
3,800
0.00015
0L00ONS
7.767
0,0066
16.662
0,9671
23.425
1.38916
0.411%
Moll KGLS (Ouns)
3,840
0.000
5.715
0,000%
7.766
0.002%
10.699
0.177%
16,652
0.805%
23.419
1.363%
0.391 %
applies each generated heuristic-optimizer to improve heuristics across all downstream tasks. 2) Invocation Frequency: In the outer loop, the meta-optimizer invoked once per iteration to generate M candidate hecuristi-optimmizess, In contrast, during the inner loop, each heuristic-optimizer is individually applied to generate K candidate heuristics for each downstream task. Consequently, the invocation frequency in the inner loop is higher than in the outer loop. Detailed prompts for optimizer and heuristic generation can be found in Appendix D.
4 EXPERIMENTS
We conduct extensive experiments to optimize various heuristic algorithms on several classical COP benchmarks, including TSP, CVRP, online and offline BPP (see Table 1, 2 and 3), with primary focus on cross-size generalization (see Appendix A for doos-distribution and cross-problen settings discussion). Additional results on other benchmarks (e.g.. TSPLib) and other optimization problems are provided in Table 8, 9 and 17 (see Appendix C). All experiments are conducted on servers with NVIDIA GeForce RTX 4090 GPUs and AMD Ryzen Threadripper PRO 7975WX CPU @ 4GHz.
Heuristic Settings. 1) Constructive Heuristic for TSP: In constructive heuristics, a solution is built incrementally, starting from a random node and iteratively selecting the next promising node based on predefined rule. The selected node is then appended to the current route to form a valid tour step by step. Since constructive heuristics focus on local optimization at each step rather than the global optimum, their performance is often suboptimal compared to other heuristic algorithms. 2) Improvement Heuristic for TSP: Guided Local Search (GLS) (Voudouris al.,, 2010) a metaheuristic that penalizes frequently used edges in local optima, steering the search away from less promising regions. Specifically, it modifies the cost landscape by adjusting the distance matrix, adding penalties to certain edges to prevent their repeated selection in subsequent iterations. In our experiment, we compare two GLS implementations from (Liu et al., 2024a) and (Ye et al., 2024). The implementation in (Liu et al., 2024a) follows standard GLS approach, combining basic local search method with dynamic edge penalties to guide the search. In contrast, the approach in (Ye et al., 2024) aligns more closely with Knowledge-Guiee Guided Local Search (KGLS) (Arnold & Sörensen, 2019), incorporating domain-spectie knowledge from the distance matrix to enhance the standard GLS framework. In our experiments, we use GLS and KGLS to represent two different settings. 3) Online BPP: We follow the settings of (Romera-Paredee et al., 2024) to develop a heuristic that assigns incoming items to bins in real time. The heuristic utilizes a scoring function to determine the most suitable bin for each item dynamically (Angelopoulos et al., 2023). We evaluate the generated heuristics on 100 Weibull instances for each problem size, ranging from 1,000 to 10,000, with bin capacities varying from 100 to 500. The lower bound lb for each instance is calculated as the ceiling of the total Σn 10
item weight divided by the capacity of a single bin: lb-
where w; is the weight of the
7

---

Published as a conference paper at ICLR 2026
Table 2:
Results on Online BPP.
Bin Capacity
Item Size
Best Fit
First Fit
FunSeareh
EoH
ReEvo
HSEvo
MCTS-AHD
MoH (Ours)
4.621%
5.018%
3.165%
3.294%
3.475%
3.748%
2.543%
2.553%
5k
4.149%
4,488
2.165%
0.827%
2.022%
1.08%%
1.769%
0.600%
10k
4.030%
4.308
2.008%
0.436%
1.821%
0.734%
1.647%
0.414%
1k
1.825%
2.025%
0.938%
1.645%
1.825%
1.825%
1.238%
0.848%
1.555%
1.665%
0.543%
0.366%
1.549%
1.555%
1062%
0.262%
10k
1.489%
1,578%
0.459%
0.188
1.489%
1.489%
1.036%
0.141 %
1k
1.131%
1 1.268%
0.654%
I 1.086%
1.131%
1.131%
0.922%
0,581 %
300
5k
0.919%
0.984%
0.352%
0.254%
0.919%
0.919%
0.785%
0.161 %
0.882%
0.924%
0.316%
0.115%
0.882%
0.882%
0.765%
0.079%
0.815%
0. 0.835%
0.519%
0.815%
0.815%
0.815%
0.755%
0.498%
400
5k
0.624%
0.672%
0.275%
0.191
0.621%
0.624%
0.608%
0.104%
10k
0.603%
0.634%
0.243%
0.098%
0.595%
0.603%
0.57918
0.054%
0.546%
0.522%
0.324%
0.695%
0.546%
0.546%
0.496%
0.373%
500
0.472%
0.507%
0.214%
1.119%
0,472%
0,472%
0.447%
0.090%
10k
0.448%
0.487%
0.196
0,075%
0.445%
0.448%
0.430%
0.032%
Average
1.607%
1 1.729%
0.825%
0.680%
1.240%
1.125%
1.000%
0.453%
item and c the bin capacity (Martello & Toth, 1990). 4) CVRP and Offtine BPP: For CVRP and Offline BPP, we develop heuristics under the Ant Colony Optimization (ACO) framework (Ye et al., 2024; 2023), a setting also adopted by (Dat et al., 2024) and (Zheng et al., 2025). Specifically, ACO maintains pheromone matrix "= and heuristic matrix 1) E R=**, where represents the number of vertices in each problem. Each entry Tú,j represents the priority of edge (i.j) selection in solution construction, which is iteratively updated based on previous solutions. For the heuristic information in j it encodes the problem-specific guidance for choosing edge (i. reflecting its local contribution to final solution quality. Our objective is to design more effective heuristic matrices y by leveraging problem-specific input features.
Baselines. 1) Traditional methods: We employ Concorde (Applegate et al., 2003) and OR- Tools (Furnon & Perron, 2023) to solve TSP, and compare with classic heuristics, including Nearest Neighbor for TSP. Best Fit and First Fit for online BPP. For OR-Tools, we use guided local search as the local search strategy. The time limit for solving each TSP instance is set to 20s for problem size < 100 and 40s for problem size 2 200. 2) five representative approaches: FunSearch (komera-Parede et al., 2024), EoH (Liu et al., 2024a), ReEvo (Ye al., 2024), HSEvo (Dat et al., 2024), and MCTS-AHD (Zheng al., 2025). We rerun their publicly available implementations in our training settings, as detailed below. 3) Neural methods: We also benchmark against neural solvers, such as POMO (Kwon et al., 2020), LEHD (Luo et al., 2023) and SIL (Luo et al., 2024), with results reported in Appendix C.
methods: We compare MoH with
Training and Inference. To ensure a fair comparison, all L.LM-baee methods are trained under identical experimental conditions for each problem setting and evolved without relying on any predefined seed heuristic. In the TSP scenario, all methods are trained on cross-size datasets comprising four tasks: TSP20, 50, 100 and 200, and generalized to larger instances of sizes 500 and 1000. In the online BPP scenario, all methods are trained on two tasks: 1,000 items with bin capacity 1,000, and 5,000 items with bin capacity 1,000. During training, iterations to T = 10 and maintain a population size of 10 for both heuristic-optimizer and heuristic populations. We control the computational budget of each method by limiting the number of heuristic evaluations to 1.000. A detailed analysis of computational costs is provided in Table 10, 11 and 12 (see Appendix C). At inference stage, the trained meta- oettmiopt is executed for 10 iterations. For the final results of heuristic performance, we use 128 instances for TSP heuristic evaluation and 100 instances for online BPP heuristic evaluation. All results reported in Tables and 2 reflect the average performance over the test dataset of the best-performing heuristic identified across three independent runs. All experiments use GPT 4o-mini (2024-07-18) as base LLM.
fix the number of outer loop
4.1 EMPIRICAL RESULT
Table presents a comprehensive comparison of our proposed MoH against baselines on TSP. The table includes best results for both constructive and improvement heuristics across TSP20-1000
8

---

Published as a conference paper at ICLR 2026
Table 3: Results CVRP and Offline BPP.
Problems
CVRP
Offliee BPP
Train
Generalization
Train
Generalization
Methods
20
50
100
200
No 100, C=
N=500, C=150
N=500, C=300
N=1000, C=300
ReEvo
4.826
9.339
15.901
28.224
41.984
207.406
102.438
204,438
HSEvo
4.858
9.250
15 940
28. 598
41.766
205.
102.438
204.
MCTS-AHD
4.843
9.165
15.630
28.041
41.656
204.609
102.625
204.734
MoH
4.704
9.059
15.563
27.512
41.625
205.453
102.125
203.750
instances in three independent runs. The optimality gap is calculated as the difference in cost between each heuristic's solution and the optimal solution, obtained using the Concorde solver (Applegate et al., 2003). In the constructive heuristic setting. MoH achieves the lowest average optimality gap of 11.792%, significantly outperforming existing LLM-based approaches. In the improvement heuristic setting, we evaluate MoH using both GLS and KGLS variants. Our approach consistently achieves the lowest optimality gap of 0.391% %, demonstrating superior solution quality across various TSP instances. Moreover, MoH demonstrates strong generalization performance on large-scale instances across both settings. These results confirm the effectiveness and adaptability of our approach in both heuristic categories and across different data regimes. Additional heuristic performance results with different baselines on TSPLib (Reinelt, 1991) are presented in Table 8 and 9 (see Appendix C). Detailed results for statistical performance are listed in Table 16.
Table 2 summarizes the best performance of MoH on the Online BPP across three independent runs, evaluated over a variety of settings with different bin capacities (100 to 500) and item set sizes (Ik, 5k, 10k). The reported metric is the proportion of excess bins used relative to the theoretical lower bound. Our method outperforms all competing baselines and traditional heuristics Best Fit, First Fit across nearly all instance settings, achieving lower average bin usage. These results indicate that MoH also performs well on online packing tasks, demonstrating strong adaptability and generalization in dynamic environments. Appendix E provides examples of the best-performing heuristics discovered by MoH in large-scale settings across these problems. We further conducted experiments on additional COPs including CVRP and Offline BPP, with the best objective values across three runs for different problem sizes summarized in Table Results indicate that MoH achieves optimal or near-optimal solution costs in almost all settings across different problem scales.
4.2 ABLATION STUDY AND FURTHER ANALYSIS
In this section, we provide a more in-depth analysis of MoH. The experiments shown in Fig. 4 and Tables 4 and 5 are conducted under the improvement heuristic (i.e., GLS) setting TSP. Results averaged over three runs, using TSPI00 and TSP200 as the downstream tasks during training.
Idea Generation. A key strength of LLMS lies in their powerful natural language processing capabilities. Integrating code generation and optimization tasks with natural language algorithm descriptions is therefore a natural approach. We incorporate these descriptions into the code generation process as ideas, enabling LLMs to fully utilize their language understanding abilities. This strategy goes beyond simple repeated sampling within the code space, allowing LLMs explore a broader and more diverse solution space. As shown in Fig. 4 and Table 5, we compare the performance of methods with and without natural language ideas. The utility score reflects the average performance across two downstream tasks during training. The results clearly demonstrate that incorporating such natural language descriptions improves training performance.
Different LLMs. In Table 4 and Table 14, we evaluate several LLMs beyond GPT 4o-mini (2024-07- 18) to assess the adaptability of MoH, including ol-mini (2024-09-12), deepseek-v3 (2024/12/26) and Qwen-plus-0919, The results demonstrate that our framework performs well across different LLMs. Furthermore, we observe that LLMs with more learnable parameters and larger context windows tend to produce longer and more complex optimization strategies. However, increased heuristic- heuristic-optimizer complexity does not necessarily lead to better performance of downstream heuristics. For instance, the more advanced ol ol-mini does not outperform 4o-mini on large-scale TSP instances.

---

Published as a conference paper at ICLR 2026
Table 4:
Ablation results of MoH-GLS
Table 5: Ablation results of MoH-GLS idea gen-
different LLMs for TSP.
cration and population size for TSP.
LLM
100
200
500
1000
Setting
Population Size
ika
100
200
500
1000
mini 0.035%
0.332%
L045%
1.710%
D.1209
0.563%
ol-mini
0.024%
0.353%
1.280%
2.119%
5
0.058%
0.339%
1,455%
2.338%
D3 V3
0.108%
0.413%
1.314%
2.527%
10
0.04398
0.300%
1.644%
2.420%
Qwen-Plus
0.057%
0.375%
1.753%
3.277%
Default
10
0.035%
0.332%
1.05%%
1.710%
Population size. For each downstream task, MoH maintains a heuristic population that allows the neuristic-optimiie to iteratively select, reference and refine promising candidates. Given the well- established effectiveness of few-shot prompting, it is crucial to retain elite heuristics from previous iterations to guide subsequent optimization steps, supporting both exploration and exploitation. As shown in Fig. 4 and Table 5, evaluate the impact of different population sizes during MoH training. The results indicate that a small population size limits the LLM's s ability to effectively leverage top-performing candidates when generating improved ones. To balance computational cost, training time, and exploration breadth in downstream heuristic tasks, we set the population size to 10. This choice ensures sufficient solution diversity while keeping overhead manageable.
Analysis of Meta-Optimizer. We take a deeper look into the meta-optimizers generated by MoH. In Appendix E, we present representative examples and analyze their underlying strategies. While some follow the EC framework (Fig. 9), similar to existing approaches, others adopt classical optimization paradigms, such as Ant Colony Optimization (ACO) in Fig. 10, Particle Swarm Optimization (PSO) in Fig. 11, Simulated Annealing in Fig. 12, Tabu Search in Fig. 13, and hybrid strategies in Fig. 15, which achieved the best performance in our evaluations. By leveraging diverse optimization principles and generating tailored prompts, MoH facilitates broader exploration of the extensive search space, enabling the discovery of more effective heuristics.
Complexity Analysis. Although MoH introduces an ad- ditional layer complexity compared with previous base- lines, empirical results show that it does not incur sig- nificant computational overhead (see Table 11 and 12
18-
Molt: Population
MolE Popelation Sie
Moli without
in Appendix). In terms of the efficiency of the gener- ated heuristics, we also conduct additional comparisons with classical solvers (i.e., Concorde (app, 2003) and OR- Tools (Furnon & Perron, 2023)) as well as lightweight learning-based solvers (i.e.. LEHD (Luo et al., 2023), SIL (Luo et al., 2024), and NeuOpt (Ma et al., 2023)) under (approximately) the same computational budget in Table 13 in Appendix. In general, our approach offers more favorable trade-off between computational cost and performance across nearly all problem sizes.
tumlity
42
Iterations
Figure 4: Training convergence curves under different settings.
5 CONCLUSION
We propose novel MoH framework, which leverages LLMs to generate effective meta-optimizers for improving COP heuristics. MoH extends the heuristic design paradigm by incorporating an outer loop for heuristic-optimizer design and employs a multi-task scheme to improve generalization and enable broader heuristic exploration. Experimental results demonstrate that heuristics discovered by MoH outperform both classical heuristics and existing LLM-based approaches. We believe MoH offers a new perspective on generating promising heuristics, with the potential to surpass human- designed ones in solving NP-hard COPs. We acknowledge certain limitations of MoH, such the search efficiency. The outer-loop and multi-task optimization may introduce additional computational overhead, highlighting the need for more efficient search strategies. Additionally, while our current scope focuses on classical COPs, MoH has the potential to address a broader range of COPs and even other classes of optimization problems, which we leave for future work.
10

---

Published as a conference paper at ICLR 2026
ACKNOWLEDGMENTS
This research is supported by the National Research Foundation, Singapore under its AI Singapore Al Research Fundamental Research Collaborative (US-NSF Researcher Call) (AISG Award No: AISG3- 11-2025-036USSSST and Singapore Programme (AISG Award No: AISG3-RP-2022-031) Any opinions, findings and conclusions or recommendations expressed in this material are those of the author(s) and do not reflect the views of National Research Foundation, Singapore. This research is also supported by National Natural Science Foundation of China No. 62473233. We would like to thank the anonymous reviewers and (S)ACs ICLR 2026 for their constructive comments and service the community.
REFERENCES
Concorde: A code for solving traveling salesman problems. http://www. tsp, gatech. edu/concorde. html, 2003.
Henrik Abgaryan, Ararat Harutyunyan, and Tristan Cazenave. Llms can schedule. arXiv preprint arXiv:2408.06990 06999, 2024.
Ali AhmadiTeshnizi, Wenzhi Gao, and Madeleine Udell. OptiMUS: Scalable optimization modeling with (mi) Ip solvers and large language models. arXiv preprinl arXiv:2402.10172. 2024.
Anton Andreychuk, Konstantin Yakovlev, Aleksandr Panov, and Alexey Skrynnik. MAPF-GPT: Imitation learning for multi-agent pathfinding at scale. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 39, pp.
2025.
Spyros Angelopoulos, Shahin Kamali, and Kimia Shadkami. Online bin packing with predictions. Journal of Artificial Intelligence Research, 18:1111-1141, 2023.
David Applegate, Robert Bixby, Vasek Chvatal, and William Cook. Concorde-03.10 2003. URL http:// www. mat h uwaterloo. wtatteeelo....tsse..eteeoo caaa://p/ccoocorrd///ellllaaaaao down lads. htm. Last accessed: 2020/2/26.
David L Applegate. The traveling salesman problem: a computational study, volume 17. Princeton university press, 2006.
Florian Arnold and Kenneth Sörensen. Knowledge-guided local search for the vehicle routing problem. Computers Operations Research, 105:32-46. 2019.
Irwan Bello, Hieu Pham, Quoc V Le. Mohammad Norouzi, and Samy Bengio. Neural combinatorial optimization with reinforcement learning. arXiv preprint arXiv: 2016.
Yoshua Bengio, Andrea Lodi, and Antoine Prouvost. Machine learning for combinatorial optimization: a methodological tour d' d'horizon. European Journal of Operational Research, 290(2):405-4211 2021.
Federico Berto, Chuanbo Hua, Junyoung Park, Laurin Luttmann, Yining Ma, Fanchen Bu, Jiarui Wang, Haoran Ye, Minsu Kim, Sanghyeok Choi, Nayeli Gast Zepeda. André Hottung, Jianan Zhou, Jieyi Bi, Yu Hu, Fei Liu, Hyeonah Kim, Jiwoo Son, Hacycon Kim, Davide Angioni, Wouter Kool, Zhiguang Cao, Qingfu Zhang, Joungho Kim, Jie Zhang. Kijung Shin, Cathy Wu, Sungsoo Ahn, Guojie Song, Changhyun Kwon, Kevin Tiemney. Lin Xie, and Jinkyoo Park. RLACO: an extensive reinforcement learning for combinatorial optimization benchmark. In SIGKDD Conference on Knowledge Discovery and Data Mining, 2025a.
Federico Berto, Chuanbo Hua, Nayeli Gast Zepeda, André Hottung, Niels Wouda, Leon Lan, Junyoung Park, Kevin Tiemney, and Jinkyoo Park. Routefinder: Towards foundation models for vehicle routing problems. Transactions on Machine Learning Research, 2025b. ISSN 2835-8856.
Jieyi Bi, Yining Ma, Jianan Zhou, Wen Song, Zhiguang Cao, Yaoxin Wu, and Jie Zhang. Learning to handle complex constraints for vehicle routing problems. In Advances in Neural Information Processing Systems, 2024.
11

---

Published as a conference paper at ICLR 2026
Aymeric Blot, Holger H. Hoos, Lactitia Jourdan, Marie-Eléonore Messaci-Marmion, and Heike Trautmann. MO-Pan P00PParmlLLS: A multi-objective automatic algorithm configuration framework. In Learning and Intelligent Optimization: 10th International Conference, LION 10, Ischia, Italy, May 29-June 1, 2016, Revised Selected Papers 10, pp. 32-47. Springer, 2016.
Jürgen Branke, Su Nguyen, Christoph W Pickardt, and Mengjie Zhang. Automated design of production scheduling heuristics: A review. IEEE Transactions on Evolutionary Computation, 20 (1):110-124, 2015.
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. Advances in neurul information processing systems, 33:1877- 2020.
Edmund K Burke, Matthew R Hyde, Graham Kendall, and John Woodward. Automatic heuristic generation with genetic programming: evolving a back-of-all-tradd or a master of one. In Proceedings of the 9th annual conference on Genetic and evolutionary computation, pp. 1559- 1565, 2007.
Edmund K Burke, Michel Gendreau, Matthew Hyde, Graham Kendall, Gabriela Ochoa, Ender Özcan, and Rong Qu slyper-heuristics: A survey of the state of the art. Journal of the Operational Research Society, 64(12):1695-177) 2013.
Edward G Coffman Jr, Michael R Garey, and David S Johnson. Approximation algorithms for bin-packing- an updated survey. In Algorithm design for computer system design, pp. 49-106. Springer, 1984.
Pham Vu Tuan Dat, Long Doan, and Huynh Thi Thanh Binh. HSEvo: Elevating automatic heuristic design with diversity-driven harmony search and genetic algorithm using llms. arXiv preprint arXiv:2412. 149.:9941.. 2024.
Darko Drakulic, Sofia Michel, Florian Mai, Arnaud Sors, and Jean-Mare Andreoli. BQ-NCO: Bisimulation quotienting for generalizable neural combinatorial optimization. In Advances in Neural Information Processing Systems, 2023.
Darko Drakulic, Sofia Michel, and Jean-Mare Andreoli. GOAL: generalist combinatorial optimiza-
tion agent learner. In International Conference Learning Representations, 2025.
Johann Dréo. Metaheuristics for hard optimization: methods and case studies. Springer Science & Business Media, 2006.
Lizhou Fan, Wenyue Hua, Lingyao Li, Haoyang Ling, and Yongfeng Zhang. NPHardEval: Dynamic benchmark reasoning ability of large language models via complexity classes, arXiv preprint 11X1::2312.14890, 2023.
Vincent Furnon and Laurent Perron. Or-tools routing library, 2023. URL URLhtttps://aeeelppeer google. com opt wtt.mmiitttttttoooooeeeo
Nyoman Gunantara. review of multi-bbecctive optimization: Methods and its applications. Cogent Engineering, 511::450 2018.
Abhishek Gupta, Yew-Soon Ong. and Liang Feng. Multifactorial evolution: Toward evolutionary multitasking. IEEE Transactions on Evolutionary Computation, 20(3):343-357, 2015.
André Hottung and Kevin Tierney. Neural large neighborhood search for the capacitated vehicle routing problem. In ECAI 2020, pp. 443 450. IOS Press, 2020.
Jiaxin Huang, Shixiang Shane Gu, Le Hou, Yuexin Wu, Xuezhi Wang, Hongkun Yu, and Jiawei Han. Large language models can self-improve. arXiv preprint 1rX1v:2210.116110, 2022.
Benjamin Hudson, Qingbiao Li. Matthew Malencia, and Amanda Prorok. Graph neural network guided local search for the traveling salesperson problem. arXiv preprint 1rX1v:2110.05291, 2021.
Frank Hutter, Holger Hoos, Kevin Leyton-Brown, and Thomas Stützle. ParamILS: an automatic algorithm configuration framework. Journal of artificial intelligence research, 36:267-306. 2009.
12

---

Published as a conference paper at ICLR 2026
Zangir Iklassov, Yali Du, Farkhad Akimov, and Martin Takac. Self-guiding exploration for combina- torial problems. arXiv preprint arXiv:2405. 2024.
Jiaming Ji, Tianyi Qiu, Boyuan Chen, Borong Zhang, Hantao Lou, Kaile Wang, Yawen Duan, Zhonghao He, Jiayi Zhou, Zhaowei Zhang, et al. Al Alignment: A comprehensive survey, arXiv preprint arXiv:2310 20..:230..19522 2023.
Juyong Jiang, Fan Wang, Jiasi Shen, Sungju Kim, and Sunghun Kim. A survey on large language models for code generation. arXiv preprint 2.xiv:24006000515.. 2024a.
Xia Jiang, Yaoxin Wu, Yuan Wang, and Yingqian Zhang. UNCO: Towards unifying neural combina- torial optimization through large language model. arXiv preprint arXiv:2408. 22224488.221444 12214, 2024b.
Xia Jiang, Yaoxin Wu, Chenhao Zhang, and Yingqian Zhang. DRoC: Elevating large language models for complex vehicle routing via decomposed retrieval of constraints. In 13th international Conference Learning Representations, ICLR 2025. OpenReview. net, 2025.
David S. Johnson, Alan Demers, Jeffrey D. Ullman, Michael R Garey, and Ronald L. Graham. Worst-case performance bounds for simple one one-dimensional packing algorithms. SIAM Journal on computing, 3(4):299 325, 1974.
Jean Kaddour, Joshua Harris, Maximilian Mozes, Herbie Bradley, Roberta Raileanu, and Robert McHardy. Challenges and applications of large language models. arXiv preprint arX11 2023.
Minsu Kim, Jinkyoo Park, et al. Learning collaborative policies to solve np-hard routing problems. Advances in Neural Information Processing Systems, 34: 14:10118-10400 2021.
Wouter Kool, Herke Van Hoof, and Max Welling. Attention, learn to solve routing problems! arXiv preprint arXiv:1803 08475, 2018
Yeong-Dae Kwon, Jinho Choo, Byoungjip Kim, Iljoo Yoon, Youngjune Gwon, and Seungjai Min. POMO: Policy optimization with multiple optima for reinforcement learning. Advances Neural Information Processing Systems, 33:21 18-8888811,, 198, 2020.
Sirui Li, Janardhan Kulkarni, Ishai Menache, Cathy Wu, and Beibin Li. Towards foundation models for mixed integer linear programming. arXiv preprint arXiv:2410.08288, 2024.
Yang Li, Jinpei Guo, Runzhong Wang, and Junchi Yan. T2T: From distribution learning in training to gradient search in testing for combinatorial optimization. Advances Neural Information Processing Systems, 36:50020-50040, 2023.
Shen Lin and Brian W Kernighan. An effective heuristic algorithm for the traveling-salesme problem. Operations research, 21(2):498-516, 1973.
Fei Liu, Tong Xialiang, Mingxuan Yuan, Xi Lin, Fu Luo, Zhenkun Wang, Zhichao Lu, and Qingfu Zhang. Evolution of Heuristics: Towards efficient automatic algorithm design using large language model. In Forty-first International Conference on Machine Learning, 2024a.
Fei Liu, Yiming Yao, Ping Guo, Zhiyuan Yang, Xi Lin, Xialiang Tong, Mingxuan Yuan, Zhichao Lu, Zhenkun Wang, and Qingfu Zhang. A systematic survey on large language models for algorithm design. arXiv preprint arXiv:2410.14771 2024b.
Fei Liu, Rui Zhang, Zhuoliang Xie, Rui Sun, Kai Li, Xi Lin, Zhenkun Wang, Zhichao Lu, and Qingfu Zhang. Llm4ad: platform for algorithm design with large language model. arXiv preprint arXiv:2412. 172877 2024c.
Fu Luo, Xi Lin, Fei Liu, Qingfu Zhang, and Zhenkun Wang. Neural combinatorial optimization with heavy decoder: Toward large scale generalization. In Advances in Neural Information Processing Systems, 2023.
Fu Luo, Xi Lin, Zhenkun Wang, Xialiang Tong, Mingxuan Yuan, and Qingfu Zhang. Self-improved learning for scalable neural combinatorial optimization. arXiv preprint arXiv:2403. 19561, 2024.
13

---

Published as a conference paper at ICLR 2026
Yining Ma, Zhiguang Cao, and Yeow Meng Chee. Learning to search feasible and infeasible regions of routing problems with flexible neural k-opt. In Thirty-seventh Conference on Neural Information Processing Systems, 2023.
Silvano Martello and Paolo Toth. Lower bounds and reduction procedures for the bin packing problem. Discrete applied mathematics, 28(1):59-70, 1990.
Sadegh Mirshekarian and Dusan Sormaz. Machine learning approaches learning heuristics for combinatorial optimization problems. Procedia Manufacturing, 77100-109, 2018.
Mohammadreza Nazari, Afshin Oroojlooy, Lawrence Snyder, and Martin Takác. Reinforcement learning for solving the vehicle routing problem. Advances in neural information processing systems, 31, 2018.
Gabriela Ochoa, Matthew Hyde, Tim Curtois, Jose Vazquez-Roddiguee James Walker, Michel Gendreau, Graham Kendall, Barry McCollum, Andrew J Parkes, Sanja Petrovic, et al. HyFlex: A benchmark framework for cross-domain heuristic search. In Evolutionary Computation in Combinatorial Optimization: 12th European Conference, EvoCOP 2012, Málaga, Spain, April 11-13, 2012. Proceedings 12, pp. 136-147. Springer, 2012.
Eneko Osaba, Javier Del Ser, and Ponnuthurai N Suganthan. Evolutionary multitask optimization: Fundamental research questions, practices, and directions for the future. Swarm and Evolutionary Computation, 75: 101203, 2022.
Nelishia Pillay and Rong Qu. Automated Design of Machine Learning and Search Algorithms. Springer, 2021.
Gerhard Reinelt. Tsplib traveling salesman problem library. ORSA journal computing, 3(4): 376-384, 1991.
Bernardino Romera-Paredes. Mohammadamin Barekatain, Alexander Novikov, Matej Balog, M Pawan Kumar, Emilien Dupont, Francisco JR Ruiz, Jordan S Ellenberg, Pengming Wang, Omar Fawzi, et al. Mathematical discoveries from program search with large language models. Nature, 625(7995):468 475,8447550000 2024.
Steven S Seiden. On the online bin packing problem. Journal of the ACM (JACM), 49(5):640 671, 2002.
Jialong Shi, Qingfu Zhang, and Edward Tsang. EB-GLS: an improved guided local search based on the big valley structure. Memetic computing, 10:333-350, 2018.
Jingyan Sui, Shizhe Ding, Boyang Xia, Ruizhi Liu, and Dongbo Bu. NeuralGLS: learning to guide local search with graph convolutional network for the traveling salesman problem. Neural Computing and Applications, 36(17):9687-97)) 2024.
Weiwei Sun, Shengyu Feng, Shanda Li, and Yiming Yang. CO-Bench: Benchmarking language model agents in algorithm search for combinatorial optimization. arXiv preprint arXiv:2504.044 10, 2025.
Yiwen Sun, Furong Ye, Xianyin Zhang, Shiyu Huang, Bingzhen Zhang, Ke Wei, and Shaowei Cai. AutoSAT: Automatically optimize sat solvers via large language models. arXiv preprint arXiv:2402.10705, 2024.
Zhiqing Sun and Yiming Yang. DIFUSCO: Craph-based diffusion solvers for combinatorial opti- mization. Advances in neural information processing systems, 36:3706-3731, 2023.
Paolo Toth and Daniele Vigo. Models, relaxations and exact approaches for the capacitated vehicle routing problem. Discrete Applied Mathematics, 123(1-3):48-551, 2002.
A Vaswani. Attention all you need. Advances Neural Information Processing Systems, 2017.
Oriol Vinyals, Meire Fortunato, and Navdeep Jaitly. Pointer networks. Advances in neural information processing systems, 28, 2015.
14

---

Published as a conference paper at ICLR 2026
Christos Voudouris, Edward PK Tsang, and Abdullah Alsheddy. Guided local search. In Handbook of metaheuristics, pp. 321-361. Springer, 2010.
Evan Wang, Federico Cassano, Catherine Wu, Yunfeng Bai, Will Song, Vaskar Nath, Ziwen Han, Sean Hendryx, Summer Yue, and Hugh Zhang. Planning in natural language improves IIm search for code generation. arXiv preprini arXiv:2409.03733, 2024.
Jason Wei, Maarten Bosma, Vincent Y Zhao, Kelvin Guu, Adams Wei Yu, Brian Lester, Nan Du, Andrew M Dai, and Quoe Le. Finetuned language models are zero-shot learners. arXiv preprint arXiv:2109.01652, 2021.
Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Fei Xia, Ed Chi, Quoc Le, Denny Zhou, et al. Chain-of-thouuht prompting elicits reasoning in large language models. Advances in neural information processing systems, 35:24824 2244372-44377 2022.
David H Wolpert and William G Macready. No free lunch theorems for optimization. IEEE transactions on evolutionary computation, 1(1):67-82, 1997.
Yaoxin Wu, Wen Song, Zhiguang Cao. Jie Zhang, and Andrew Lim. Learning improvement heuristics for solving routing problems. IEEE transactions on neural networks and learning systems, 33(9): 5057-5069, 2021.
Ziyang Xiao, Dongxiang Zhang, Yangjun Wu, Lilin Xu, Yuan Jessica Wang, Xiongwei Han, Xiaojin Fu, Tao Zhong, Jia Zeng, Mingli Song, et al. Chain-of-Fxperts When ilms meet complex opera- tions research problems. In The Twelfth International Conference on Learning Representations, 2023.
Dejun Xu, Kai Ye, Zimo Zheng, Tao Zhou, Gary G Yen, and Min Jiang. An efficient dynamic resource allocation framework for evolutionary bilevel optimization. IEEE Transactions on Cybernetics, 2024.
Shunyu Yao, Fei Liu, Xi Lin, Zhichao Lu, Zhenkun Wang, and Qingfu Zhang. Multi-objective evolution of heuristic using large language model. arXiv preprint 24Ki::2209.16661 2024.
Haoran Ye, Jiarui Wang, Zhiguang Cao, Helan Liang, and Yong Li. DeepACO: Neural-enhanced ant systems for combinatorial optimization. Advances in neural information processing systems, 36: 13706-43722 2023.
Haoran Ye, Jiarui Wang, Zhiguang Cao, Federico Berto, Chuanbo Hua, Haeyeon Kim, Jinkyoo Park, and Guojie Song. ReEvo: Large language models as hyper-heuristics with reflective evolution. arXiv preprint arXiv:2402.01144 1445, 2024.
Eric Zelikman, Eliana Lorch, Lester Mackey, and Adam Tauman Kalai. Self-Taught Optimizer (STOP): Recursively self-improving code generation. In Conference on Language Modeling. 2024.
Zhi Zheng, Zhuoliang Xie, Zhenkun Wang, and Bryan Hooi, Monte carlo tree search for comprehen- sive exploration in llm-based automatic heuristic design. In International Conference on Machine Learning, 2025.
Jianan Zhou, Yaoxin Wu, Wen Song. Zhiguang Cao, and Jie Zhang. Towards omni-generaliizale neural methods for vehicle routing problems. International Conference on Machine Learning, pp. 42769 2023.
Jianan Zhou, Zhiguang Cao, Yaoxin Wu, Wen Song, Yining Ma, Jie Zhang, and Xu Chi. MVMoE: Multi-task vehicle routing solver with mixture-of-experts. In International Conference on Machine Learning, 2024.
Yongchao Zhou, Andrei Ioan Muresanu, Ziwen Han, Keiran Paster, Silviu Pitis, Harris Chan, and Jimmy Ba. Large language models are human-level prompt engineers. arXiv preprint arXiv:2211.01910. 2022.
15

---

Published as a conference paper at ICLR 2026
APPENDIX
Frequently Asked Questions
17
Related Work
19
B.1 Neural Heuristics for COPs
19
B.2 LLMs for COPs
19
C Additional Results
20
C.1 Results on TSPLIB
20
C.2 Cost and Evaluation Comparison
20
C,3 Comparison with Classical and Learning- Based Solvers
22
C.4 Comprehensive Comparison of baselines on different L.Mss
22
C.5 Statistical Significance Analysis
22
C.6 Statistical Performance
23
C.7 Additional Results on other optimization problems
23
D Prompt Design
24
D.I Prompts for Couristic-Optimize Generation
24
D.2 Prompts for formulating heuristic generation
................................
25
D.3 Prompts for Heuristic Initialization
E Examples of L.MM-Genrraeed Heuristics and Meta-Optimizers
26
Broader Impacts
36
The Use Large Language Models
36
H Licenses
36
16

---

Published as a conference paper at ICLR 2026
A FREQUENTLY ASKED QUESTIONS
Comparison of MoH with other Optimization Frameworks. The meta-optimization of our framework also resembles other optimization methods such as Multi-Task Optimization (Gupta et al., 2015; Osaba et al., 2022), Multi-Objective Optimization (Gunantara, 2018) and Bi-level Optimization (Xu et al., 2024). However, we emphasize our method's distinctions below:
Traditional multi-task algorithms are explicitly designed to train unified model capable of solving multiple tasks simultaneously. typically via parameter sharing, knowledge transfer and genetic operators (Gupta et al., 2015). In such methods, cross-task generalization is learned during training. In contrast, we build MoH on a pre-trained L.M that already exhibits broad task-solving capability, requiring no additional training of the model itself. Moreover, MoH focuses on generating improved heuristics by employing diverse opti- mization strategies to explore the LLM search space. In LLM-based heuristic generation, integrating evolutionary multitasking (Osaba et al., 2022) (EMT) is constrained by the semantic variability of LLM- -eneerreet code and prompts: in practice, EMT can operate only at the heuristic layer- iee., searching over task-specific heuristics across downstream tasks-thereby reverting to a fixed LLM-EC pipeline. In contrast, our approach treats each task individually and does not assume EC as the underlying heuristic-optimizer, The meta-optimizer adaptively discovers, modifies, and even composes optimization strategics (EC otherwise) during training. yielding more flexible alternative that steps beyond previous LLM-EC design.
As for mujti-obeective optimization (Gunantara, 2018), it is typically employed to address conflicting objectives by identifying Pareto-optimal solutions that reflect trade-offs among them. However, this paradigm is also not well-suited to our motivation, which is to evolve optimizers rather than heuristics, using LLMs for CO.
For multi-level frameworks, our problem does indeed align with bi-level optimization structures (Xu et al., 2024), as both involve two-stage optimization process. However, what distinguishes our work is the application of bi-level optimization perspective to the domain of LLM-based code generation for heuristic design. This is not a straightforward extension, as it introduces several unique challenges: First, unlike typical bi-level problems where objectives and variables are clearly defined, both of our optimizer and heuristic are ALM--gnerated programs, whose implicit, program-level behavior (c.g., heuristic logic as code) must be evaluated by execution rather than closed-form analysis, thereby introducing additional abstraction and complexity to the problem. Second, bi-level optimization typically focuses on the co-optimization of both the upper and lower levels, where the two problems are often tightly coupled. In contrast, our approach primarily aims to improve the heuristics generated in the inner loop by exploring diverse optimization strategies in the outer loop. Moreover, the downstream tasks in our framework are flexible and can be adapted based on specific needs, making the approach more versatile across different types of problems.
In summary, while some of the mentioned optimization frameworks may hold potential, adapting them the domain of L.LM-based code generation for CO is non-trivial, particularly in way that aligns with our goal of evolving optimizers without introducing significant complexities or computational overhead.
Relationship of Meta-Optimizer and Neeristic-Optimizze In our framework, meta-optimizer and heuristic-optmmizer share the same functional structure (see Fig. 3) and meta-optimizer derived from heuristic-optimizenn, their fundamental difference lies in their hierarchical roles and optimization targets within our bilevel optimization framework: 1) Meta-optimizer (Outer Loop): Operates at a higher abstraction level. Its goal is to design heuristic heuristic-optimizers. It takes a population of heuristic- optimizers as input and evolves them find better optimization strategies. 2) Heuristic-optimizer (Inner Loop): Operates at the task level. Its goal is to solve specific downstream CO tasks. It takes a population of heuristics as input and evolves them to minimize the optimality gap for downstream problems (e.g., TSP, BPP). Despite these distinct roles, we implement both using unified functional template (see 3) to enable recursive self- self-immrovement. This allows the same L.M-eeerrteed function structure to adapt its behavior based on the specific inputs it receives. To be specific, 1) when acting as a Heuristic- Heuristie-Optimiierr population: Stores candidate heuristics for each task. utility: Evaluates task performance (e.g., optimality gap, cost). subtask: Specifies the task type and size (e.g., "TSP20");
17

---

Published as a conference paper at ICLR 2026
2) when acting as Meta- Meta-Optimizer population: Stores candidate heuristic-optimizer. utility: Aggregated performance across all downstream tasks. subtask: Is set to "Optimizer", directing the function to perform meta-level optimization (as shown in Algorithm 1).
Comparison with Meta-Prompt Optimization. We would like to clarify that characterizing MoH merely a "sophisticated meta-prompt optimization" may not fully capture the fundamental algorithmic shift introduced by our framework. The key distinction lies in moving beyond opti- mizing with a static strategy toward designing dynamic, executable search strategies. Our focus is on generating algorithmic code via LLM, not on "prompting prompts". We first clarify that our meta-optimizer is more than meta-prompt. Concretely, meta-prompt optimization focuses on tuning textual inputs to elicit a better response. In contrast, the MoH meta- veta-optimizer functions as an active algorithmic controller. does not simply "ask" the L.MM; it maintains a stateful pro- cess that includes population management, iterative code refinement, and feedback loops. These are structured algorithmic components that exist outside the prompt itself. This unique active algorithmic controller is formalized in our bilevel framework. Unlike prior works that rely on a single-level prompt template, MoH introduces an outer loop that dynamically searches for the optimization strategy itself. This allows MoH to explore the space of algorithms rather than just the space of prompts, a direction largely underexplored in LLM-based combinatorial optimization.
Generalizability Claim. The main focus of this paper is cross-size generalization, as highlighted in the abstract, introduc- tion, and experimental sections. While our framework can, in principle, be extended to support other forms of generalization, doing so would require addressing addi- tional challenges, which we leave for fu- ture work. We will make this scope and positioning clearer in the revised version. 1) Cross-problem generalization. We agree that MoH does not directly generalize a hcuristic learned for one CO problem completely different CO problem. This limitation is also shared by all baselines we compare against. Although our framework introduces a multi-task training mechanism that can, in principle, incorporate multiple problem settings, using single trained meta-optimizer across distinct CO problems may not yield strong performance, particularly due to the substantial struc- tural differences (e.g., objective, constraint) among CO problems. Nevertheless, we conducted preliminary experiment in which MoH was jointly trained on TSP-GLS, TSP-KGLS, and BPP- online tasks. The results (meandstandard error across 5 runs) are shown in Table 6. We leave further improvements to future work. 2) Cross-distribution generalization. In fact, switching to a cross- distribbttii setting does not modify the heuristic's internal structure, it merely changes the distribution of the instances used for evaluation. Because the heuristic design remains intact, our method can readily generalize across distributions, and it consistently delivers strong perfor- mance under such cross-distribution settings. We evaluate cross-distribution generalization in Ta- bles and 9 using TSPLIB, whose instances do not follow the uniform distribution used during training. We also evaluate the generalization ability of the meta-optimizer trained TSP200- Uniform by leveraging it to optimize hcuristics for the EBP200-Cluster task. Table shows the in-distribution )nnffo))U)iform) and distrbbbtitt (TSS200-CClster) performance, respectively.
Table 6: Results for Cross-Problem Training
Training Task
Gap
TSP-GLS FLZ--Sze200 TSP-KGLS 2000-Size200
0.384%+0.008%% 0.219%=0.023%
BPP-online-5k item, cap. 100 0..18%
Utility Function. Our rationale for us- ing weighhee utility is that CO heuris- tics typically degrade more significantly as problem size increases. Thus, we place greater emphasis on larger instances to encourage the meta-optimizer to discover heuristics that remain effective at scale, an effect that empirically supported by the table below. This weighting scheme high- lights performance on more challenging problems.
Table 7: Generalization results from Uniform Cluster TSP
TSP-GLS
TSP200-Cluster UnP200-Uniform
ReEvo HSEvo
0.331%
0.290%
0.328%
0.315%
MoH
0.291 %
0.268 %
Novelty of Optimizer. We do not emphasize the novelty of the meta-optimizer itself, since some of the generated meta-optimizers indeed resemble classical optimization strategies. In contrast to
18

---

Published as a conference paper at ICLR 2026
approaches that typically rely fixed evolutionary computation (EC) heuristic-optimizzz while leveraging LLMs to design heuristics for combinatorial optimization problems (COPs), our approach enables the automatic design of heuristic-optimizers that evolve CO heuristics from scratch, offering fresh insights into this field. By applying different optimization strategies within those generated heuristic-optmmizess, can explore more diverse heuristic search space, thereby improving the performance of the discovered heuristics.
B RELATED WORK
Traditional heuristic design for NP-hard COPs relies heavily on expert knowledge and is time- consuming to develop (Dréo, 2006). This has motivated the emergence of automatic heuristic design (Burke et al., 2013) as more efficient alternative (Pillay & Qu, 2021), leveraging metaheuristic or ML techniques to automate heuristic generation and optimization (Burke et al., 2007; Hutter et al., 2009; Blot et al., 2016; Mirshekarian & Sormaz, 2018). However, these approaches are often constrained by inflexible search and strong
dependencies (Ochoa et al., 2012; Branke al., 2015). Recently, neural solvers have gained attention as a promising alternative ( (Vinyals et al., 2015; Kool et al., 2018), employing deep learning to learn heuristics in a data-driven manner. Despite showing promise, they still face several challenges, including limited scalability and generalization, as well as high training overhead. More recently, the advent of LLMs has transformed the landscape of heuristic design. Their advanced language understanding and reasoning capabilities (Brown et al., 2020; Wei et al., 2022; 2021) have been increasingly exploited to enhance heuristic generation for solving COPs (Liu et al., 2024b; Nomera-PPaeede et al., 2024; Sun et al., 2024). Among recent efforts, most approaches combine the efficiency of evolutionary search with the adaptability of LLM reasoning via few-shot prompting, leading to a surge of interest in L.M-EC frameworks for heuristic design in COPs (Liu et al., 2024a; Ye et al., 2024; Yao et al., 2024). However, the use of fixed optimization strategy (e.g.. EC) in these frameworks often restricts exploration of the broader search space. In addition to serving as heuristic generators, LLMs have also been employed to directly generate solutions or formulate mathematical models for solving COPs. In the following, we provide comprehensive review of neural and LLM-based approaches.
B. NEURAL HEURISTICS FOR COPS
Unlike traditional hand-crafted heuristics, neural heuristics for solving COPs have rapidly advanced in recent years (Bengio et al., 2021; Berto et al., 2025a). These methods generally fall into two paradigms. 1) For constructive heuristics, Pointer Network (Ptr-Net) (Vinyals et al., 2015), lequence-to-sequeee model with differentiable attention mechanisms, was first introduced to directly learn permutationninvariaa solutions for TSP through supervised learning. This was extended by using reinforcement learning to improve performance (Bello et al., 2016), and further applied to CVRP (Nazari et al., 2018). With the rise of Transformer architectures (Vaswani, 2017), the attention- based model (Kool et al., 2018) was proposed to solve various COPs, inspiring series of subsequent works (Kim al., 2021; Kwon et al.. 2020: Drakulic et al., 2023; Luo al., 2023: Zhou et al., 2023; Bi et al., 2024). More recently, there has been a surge of interest in foundation models that aim to solve multiple COPs using a single, general-purpose model (Zhou al., 2024; Berto et al., 2025b; Drakulic et al., 2025). 2) Improvement heuristics (Hottung & Tiemeyy 2020; Wu et al., 2021; Li et al., 2023; Ma et al., 2023; Sun & Yang, 2023) leverage neural networks to guide local search for solution refinement (Hudson al., 2021; Sui et al., 2024). While these approaches can often produce (near-) )optimal solutions with extended inference times, they typically face challenges in scaling to large problem instances and generalizing across diverse problem settings.
B.2 LLMS FOR COPS
LLMs have recently gained widespread recognition and found broad applications across various domains (Ji et al., 2023; Kaddour et al., 2023), significantly influencing research directions in combinatorial optimization. In particular, recent studies have explored the application of LLMs in multiple facets of CO, including enhancing algorithm design (Liu et al., 2024a; Romera-Pareee 2024; Ye et al., 2024), automating the formulation of CO problems (AhmadíTeshniz et al., 2024; Jiang et al., 2025: Li et al., 2024; Xiao et al., 2023), developing CO-specific benchmark datasets (Fan al., 2023; Sun et al., 2025), directly solving COPs (Abgaryan et al., 2024; Iklassov al., 2024), and
et al.,
19

---

Published as a conference paper at ICLR 2026
integrating LLMs into domain-speciff foundation models to construct unified frameworks capable of addressing a wide spectrum of CO tasks (Andreychuk et al., 2025; Jiang et al., 2024b). As LLMs continue to evolve rapidly, they exhibit great potential to support the development of more automated, generalizable, and efficient problem-solving frameworks in the field of CO.
C ADDITIONAL RESULTS
C.I RESULTS ON TSPLIB
We further evaluate method on the widely used TSPLib dataset across various instance sizes. As shown in Table 8 and 9, we compare the TSP-GLS and TSP-KGLS settings against EoH (Liu et al., 2024a), ReEvo (Ye et al., 2023), HSEvo (Dat et al., 2024), MCTS-AHD (Zheng et al., 2025), Neural Combinatorial Solvers (Kool et al., 2018; Kwon et al., 2020; Luo et al., 2023; 2024) and GLS algorithms (Hudson et al., 2021; Sui et al., 2024; Voudouris et al., 2010; Shi et al., 2018; Amold & Sörensen, 2019) across instances of different scales. The results are split into two tables: one for instances smaller than 200, and another for sizes ranging from 200 to 1000. To handle the distributional diversity in TSPLIB, we adopt an instance-level heuristic selection strategy. Instead of using a single "best" heuristic from Table I, MoH evaluates all size-specific best heuristies trained on the uniform distribution and selects the best-perfie rest-performing one for each TSPLIB instance. This allows MoH to adapt to the heterogeneous characteristics of TSPLIB instances. For faimesss, all baselines follow the same instance-wise selection procedure, where each method chooses its best-performing heuristic among its size-specific candidates. Consistent with the results in Table 1, our generated heuristics outperform both GLS and KGLS baselines on most TSPLib instances. Heuristic 2 and show examples of the best heuristics generated for the TSP-GLS and TSP-KGLS settings of size 1000, respectively.
Table 8: Results on TSPLib instances with sizes smaller than 200.
Ncani Sulver
GLS Algrethero NeerdGGS
TSP-ELS
TSPAKGES
Instance
AM
POMO
LEHD
SIL
GNNGLS
GLS
EBGLS
KGGLS
Exit
HSEw
Moll
ReEvo
HSEVo
MICTS-AD
Moll
hetlins?
4.17
[104
0.00
0.01
0.14
0.00
0.01
01.0%
0.01
0.01
0.03
0.83
0.01
0.01
01.0%
0.03
<170
1.74
0.31
0.33
0.31
0.76
0.00
0.31
0.31
0.31
0.31
0.31
0.31
0.31
0.21
0,31
0.31
el76
1.59
1.18
2.56
1.15
0.16
0.00
1.37
1.18
1.38
1.18
1.18
1.18
1.18
1.18
1.38
1.18
pe76
8100
0.00
0.04
0.80
01000
0.50
0.00
0.00
0.69
0.00
0.30
0.00
4.02
0.41
0.12
0.02
0.23
0.03
0.82
0.00
0.56
0.02
0.22
682
0.02
0.02
0.52
0.2
knr8100
5.14
1152
0.26
0.99
0.15
0.55
0.35
0.000
0.25
0.00
0.00
0.00
0.00
0.00
0.50
0.00
kmo1100
0.97
11.88
0.30
0.01
1.57
1.77
0.50
0.01
0.01
0.01
0.03
061
0.01
0.01
0.91
0/01
2.72
0.54
0.00
0.00
0.8
0.20
0.30
0.00
0.00
0.8
0.00
0,50
0.00
61110)
3.41
0.01
0.01
1.01
0.00
0.01
0.01
0.1
0.01
0.91
000
40101
2.09
1.
2.31
0.20
0.36
3.76
1.91
2.07
1.74
1.83
1.76
1.78
1.78
1.76
176
Sni05
1.74
0.52
0.36
0.33
0.61
0.65
0.83
0.06
0.28
0.03
0.03
083
0.03
0.03
0.88
000
0.52
0.00
6.40
0.00
0.30
0.00
0.00
080
0.00
0.00
0.50
0.00
Neri27
5.91
13.32
0m
1.95
2.31
0.59
0.39
0.42
0.01
0.0
081
0.0
0.94
0.10
001
1.16
16
0.55
0.25
3.52
1.10
1.00
0.56
0.11
0.0
001
081
000
0.0
0.51
001
5.06
0.5
0.65
000
3.30
2.33
2,01
0.28
0.24
0.00
0.00
060
0.00
0.01
0,50
0.00
0.53
0.09
0.00
0.00
0.00
0.00
AsoAISO
1.78
0.70
1.40
0.00
2.56
0.71
1.75
0.36
0.67
0.00
0.00
080
0.00
0.00
0.50
0.00
2.44
1:17
0.76
000
3.26
3311
1.01
8100
0.66
0.00
0.00
0.0
0.00
0.00
0.30
0.00
p9152
7.49
1.05
13.04
0.19
312
0.00
0.19
0.19
0.19
0.00
0/00
060
0.19
0.19
0.19
0/00
UL50
0.05
0.00
1.2
0.90
6.34
0.78
696
0.00
0.00
0.00
0.50
dL98
373.02
17.29
0.25
0.45
4.75
1.26
2.08
1.57
0.31
0.32
0.26
021
0.20
0.28
0.45
0.20
7.11
1.38
0.6
0.99
2.03
0.86
0.74
0118
0.71
0.13
0/6
0.00
0.31
0.09
0.06
0/00
kmi6300
6.56
1.44
0.16
0.01
7.59
1.34
1.45
1.37
0.69
0184
0.05
0.6
0.01
0.1
0.31
0/1
C.2 COST AND EVALUATION COMPARISON
We present comprehensive cost comparison across methods by reporting their average computational metrics, i.e., the L.MM request counts, token usage, evaluation numbers, and performance, for the two studied problem settings: TSP-GLS (Table 10 and 11) and CVRP (Table 12). For TSP-GLS, we use instances of size 100 and 200 for both training and inference to ensure fair comparison between MoH and baseline methods. For CVRP, training involves instance sizes of 20 and 50. All results are averaged over three runs, with token usage evaluated using the GPT-4o-mini API.
From the comparison between Table 10 and 11, want to highlight that for the same problem type, the time cost is determined solely by the problem size. This is because the overall runtime is dominated by the evaluation phase- i.e., executing the generated heuristics to compute their scores.
20

---

Published as a conference paper at ICLR 2026
Table 9: Results TSPLib instances with sizes ranging from 200 1000.
Neural Solver
TSP-GLS HSEvo
TSP-KGLS
Instance
POMO
LEHD
SIL
EoH
MoH
ReEvo
HSEvo
MCTS-AHD
MoH
rs225
3.60
0.28
0.00
0.00
0.00
0.00
0.00
0.00
0,00
0.00
Isp225
3.17
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0,00
0.00
pr226
1.42
1.11
0.04
0.00
0.04
0.00
0.00
0.00
0.00
0.00
pr264
2.80
0.00
0.00
0.00
0.00
0.00
0.00
5.23
0.30
0.30
pr299
4.94
2.81
0.01
0.07
0.53
0.11
0.13
0.07
0.08
0.01
lin318
4.72
1.41
0.69
0.55
1.13
0.38
0.32
0.27
0.43
0.29
rd400
6.37
1.00
0.00
0.78
0.82
0.29
0.17
0,38
0.45
0.20
11417
8.51
7.76
2,42
0.64
0.68
0.62
0.60
0.49
0.70
0.49
pr439
7.87
3.37
2.46
1.01
1.23
0.56
pcb442
5.36
3.11
1.12
0.28
0.35
0.15
0.08
d493
9.67
9.49
0.24
1.12
1.11
0.51
0.61
0.67
1.42
0.45
u574
11.86
2.73
0.28
1.12
0.87
0.80
0.86
1.49
1.39
0.83
rat575
12.46
3.02
0.85
2.55
1.25
1,35
1.67
1.60
1.47
1.04
p654
11,30
3,30
2.77
0.21
0.31
0.15
0.11
0.10
d657
12.72
1.26
1,02
0.96
0.54
u724
16.57
3.27
0.19
1.06
2.07
0.99
0.91
0.92
0.88
0.72
rat783
18.11
3.91
0.65
2.46
1.98
2.18
1.68
1.65
1.86
1.11
pr1002
20.00
4.44
0.51
1.83
1.11
1.14
1.21
1.27
1.17
0.90
Average
8.77
3.56
0.53
0.84
0.92
0.57
0.57
0.70
0.66
0.40
Running a heuristic on a larger instance (e.g.. 200 nodes) naturally requires more CPU time than on smaller instance (e.g., 100 nodes), affecting all methods equally. In contrast, token consumption and LLM request counts are primarily dictated by the algorithmic design and prompting strategy of different problem settings, rather than by the problem size. The LLM's role is to generate heuristic code (i.e., algorithmic logic), which is inherently agnostic to the scale of the specific problem instance. This claim is also empirically supported by the comparison between Table and Table 12.
Table 10: Cost of different methods on TSP-GLS setting with size 100.
Methods
Time/mins
Input Tokens
Output Tokens
Total Tokens
LLM requests
Evaluation count
Results- Gap
MoH Train
196.9
927537.0
589250.3
1516787.3
1347.7
989.3
0.034%
MoH Inference
46.7
526404.3
169028.3
695432.6
311.3
282.7
0.036%
FoH
223.2
1282802.0
390085.5
1672887.5
1119.0
1001.7
0.055%
214.7
1291765.3
297487.0
1589252.3
691.3
Table 11: Cost of different methods on TSP-GLS setting with size 200.
Methods
Time/mins
Input Tokens
Output Tokens
Total Tokens
LLM requests
Evaluation budget
Results-Gap
MoH-Train
238.4
743837.0
458358.7
1202195.7
1248.3
971.7
0.373%
MoHl Inference
62.7
398064.3
138350.0
536414.3
276.7
240.3
0.398%
FoH
326.4
1437973.0
446583.7
1884556.7
1256.0
992.0
0.535%.
291.7
1188412.3
1579077.3
679.7
1005.0
0.448%
Table 12: Cost different methods on CVRP+ACO setting.
Inpur Tokens
Output
Tokens
requests
kudget
Results-Obj. 20
Results-Oly.
MoH--rii
321.1
98X2160
720292.3
1726908.3 3
1456.7
938.1
4.851
9.262
MoH Mofe-inference
107.5
479618.0
155284.3
632902.3
230.3
307.3
4,837
9.254
ReEvo
231.2
2122500200
677652.0
2800454.0
1431.
1000.0
4.877
9.521
670590.0
MCTS
1330.3
2001902.0
$25037.3
3426939.3
1490.0
1002.7
4.881
9.235
21

---

Published as a conference paper at ICLR 2026
C.3 COMPARISON WITH CLASSICAL AND LEARNING-BANE BASED SOLVERS
We also make additional comparison with classical solvers (i.e., Concorde and OR-Tools) as well as lightweight learning-basee based solvers (i.e.. LEHD (Luo et al., 2023). SIL (Luo et al., 2024), and NeuOpt (Ma et al., 2023)) under (approximately) the same computational budget in Table 13. Specifically, we control the solving time across all methods to be comparable to the inference time of MoH, except for Concorde, whose runtime cannot be constrained. Classical solvers often suffer from scalability issues due to the NP-hard nature of CO problems, while tearning-based solvers face generalization challenges (e.g., NeuOpt struggles to generalize to larger instances and SIL struggles in generalization to smaller sizes). In contrast, our approach offers a more favorable trade-off between computational cost and performance across nearly all problem sizes. This highlights the practical value of MoH in real-world scenarios where routine, varied-scale CO solving is required, as the task-level deployment setting more accurately reflects practical applications of heuristic design and justifies the initial computational investment.
Table 13: Optimality gap(%) (%) and averaging solving time(s) of instances of different problem sizes across different baselines.
Problem Size
100
200
500
1000
Methods
Gup
Average Time
Gap
Average Time
Gap
Average Time
Gap
Average Time
Concorde
0.000%
0.260s
0.000%
1.094s
0.000%
11.878s
0.000%
229.528s
OR-Tools
2.529%
2.024s
3.843%
3.001s
4.751 5S:
10.007s
5.001
45.099
LEHD
0.375%
1.086s
0.446%
2,709s
0.792%
9.066s
1.680%
44,469s
SIL
4.073%
1.122s
2.545%
2.159s
1.459%
8.789s
1.051%
43.025s
NeuOpt
0.471%
1.432s
0.414%
4.411s
125. 864%%
45.314s
MoH-GiSS MoH-K KGL.S
0.012%
1.075s
0.291 %
2.335s
0.936/S
7.818s
1.476%
21.681s
0.002%
0.394s
0.177%
1.497s
0.805%
10.866s
1.365%
45.518s
C.4 COMPREHENSIVE COMPARISON OF BASELINES ON DIFFERENT LLMS
To provide a broader evaluation, we compare multiple LLMs across different baselines and problem sizes. This allows us to verify the consistency of performance trends beyond single setting. Table 14 reports results for GPT 4o-mini, ol-mini, DeepSeck-v3, and Qwen-plut combined with EoH, HSEvo, and MoH over problem sizes 100-1000. Each entry shows mean error with standard deviation. Overall, MoH demonstrates clear superiority across settings. Compared to EoH and HSEvo, MoH achieves consistently lower errors, particularly on larger problem sizes (500 and 1000).
Table 14: Optimality gaps (mean% std%) of baselines on different LLMs across 5 runs.
Baselines
Problem Size
GPT 4o-mini
ol-mini
DeepSeck-v3
Qwen-plus
100
0.051%+0.011%% 0.426%:+0.043%
0.052%:40.010%
0.072%:10.339%
0. 40. 142%
200
0.439%
0.471%:2..0879
0.780% 289%
EoH
500
1.927%:+0.49) %
1.486% 145%%
1.692%:20.483 482%
2.007%++0.535%%
1000
3.323%+40.5166
2.857%::0.27 %
3.362%
3.485%++0.295 295%
100
0.076% +..66%
0.042%:+0.01
0.042%::0.. 0266%
0.044%+40.00
40.022%
200
0.729%
0.441%:+0.020
0.590% 2..36% 236%
0.497%++0. 168%
HSEvo
500
1.726% 1.26%
1.811%::0.09)
1. 1.842% 605%
1.755%=+0.537%
1000
3.26 1 4.75%%
3.493%:40.222
3.006% +0.55%
2.797% 433%%
100
0.031% +.0200
0.024
0.080+0.066
0.040=0.01 1%%
200
0.345
0.36420.0119 %
0.392% 20.0%% 027%
0.387%+40.0188
MoH
500
1.353%
1.47%%
1. +.625%%+0.3555
1000
1.89%%+0.308%%
1.33%%20.2989%
2.669%+40.310S
3.283%
C.5
STATISTICAL SIGNIFICANCE ANALYSIS
To further validate the robustness of our results, we conduct statistical significance tests comparing MoH against wide range of baselines under both TSP constructive and TSP-KGLS settings. As
22

---

Published as a conference paper at ICLR 2026
shown in Table 15, conduct one-sided Wilcoxon signed-rank tests the objective values of the heuristics generated by MoH and the corresponding baselines. The results are averaged from three best runs of each method. The results demonstrate that our method consistently achieves statistically significant improvements over all baselines. In particular, all p-values are far below the 0.05 threshold, providing strong evidence that the observed gains are not due random chance. This indicates that the observed improvements of heuristic performance by MoH are not due to random chance but are statistically significant.
Table 15: One-sided Wilcoxon signed-rank test p-values comparing MoH with basclines.
Prohlem
TSP constructive
TSP-KGES
Baselines
FunSearch
EoH
ReEvo
HsEvo
MCTS-AHD
ReBvo
HsEvo
MCTS-AHD
MoH-200
2.73E-55
148E-15
161E-03
3,55B-22
8.4SE-12
5.30E-03
L14E-02
1.59E-04
MoH-
2.12F
1.99E 19
4.86F.09
8,75E
3.56F: 27
3,90E-11
7.12F 15
4.05E-07
MoH-1000 1000
2.40E664
1.61E-34
2.25B-31
1.70E-64
5.26E-34
1.01E-66
2.78E-36
1.10E-08
C.6 STATISTICAL PERFORMANCE
To further validate the robustness of our results, conduct repeated evaluations under each problem size and report the mean + standard deviation across multiple runs (see Table 16). This statistical summary demonstrates that MoH not only achieves the best average performance but also maintains low variability, confirming the reliability and stability of our proposed method.
Table 16: Statistical performance of different baselines for TSP.
Problem Size
20
50
100
200
500
1000
Constructive TSP
Punsearch
9277
16.099%+1 340%
18.000%==1 .497%
75777
22.54%
22.873% 9766%
Eoft
10. 11.5%-10.9999
13.180%
851%
14.485%
.076/9
15.890%211 0000%
16. 1o.144%%0..6722
17.224%::0.6729
Relivo
9.51955+0 1925%
12.3175++0
269%%
15.3779%
14 6545554442222 2.30%
15.412% +3.3335
16. 16.2588%-+ 1944%
HsEvo
10.938%
14.105% 1.639%
16.31,5%f1.1223%
18.108%211.66
18. 2.0%%
MCTS-AHD
8.40% 140%
12.545%
2845
14.128%=2.55 511%%
15. 15.727%
16.813%% 1A445
17.306/9:11 089%
MoH
8..39%
12.307% 12.019%
13.046%
14.103%110.446
14.778%:2046699
15.867 10.1..
Improvement TNP
FoH-GEES
000%
00005-+00000%
0.051%
0.426%
199275+0299155
3.32355-4
HsEvo-GLS
0.000% 000%
0.000.100000%
0.076% 20 0666
0.729% 310%
1. 1.726%8 22266
26.11 775%
ReEva-GLS
0.000%++0.000% 40.00%
0.000%=00.000%
0,063 =0.027%
0.627% 340%
2.060% 4444%
3.491% 665%
MoH
0.000% 10.000%
0.000%
0.031 20.000%
20.039
20.280%
20.308%
ReEve-KGLS S
+00005+00000%
00005550000
(00055=0.00025
0.223%
0.981% S500 (104%%
1.63665 0220%
HsEva- KGLS
+000%%+000000%
0000%+00000%%
0.006% =0.00%%
0.228%:+0..000
1.003% 00469
I. 1667% 071 1%
MCTS-AHI--E
000008-00.000%
00000--000000
0.0111%=50.000
0.22991-40.025%
(1942%
1.4789
S
1.000%:40.0005
+0.00%%
20.001
0.891 +0.03%
1.4744 3774%s.0..0 #0.0088
C.7 ADDITIONAL RESULTS ON OTHER OPTIMIZATION PROBLEMS
In principle, our method can be applied to other heuristic optimization tasks that can be represented as a function and have a corresponding evaluation metric (utility). To this end, we conducted additional experiments on the Quadratic Assignment Problem (QAP) and the Acrobot system control task (Liu et al., 2024c). For Acrobot, the reported metric is the minimum number of steps required to complete the task across 100 randomly initialized conditions. For QAP, the cost is averaged over 128 instances. The results are listed in Table 17.
Table 17: Performance comparison on Acrobot and QAP tasks.
Task
Domain
Metric
ReEvo
MoH (Ours)
Acrobot QAP
Robotics Control
Min. Steps
99.65 + 37.60
88.76 = 14.19
Facility Layout
Cost 1
5.913,687.95
5,833,648.08
23

---

Published as a conference paper at ICLR 2026
D PROMPT DESIGN
D.1 PROMPTS FOR HEURISTIC-OPTIMZZE GENERATION
We present the prompt used to format and generate the heuristic-optimizzr, along with those embedded within the seed meta-optimizer to guide idea generation and code synthesis for both the heuristic- optimizer and downstream tasks, as illustrated Fig. In addition to the predefined prompt constraints, also integrate additional judgment mechanisms into our framework to ensure the explainability and efficiency of generated hyper-heuristics, mitigating potential impact LLM output uncertainty MoH performance.
Prompt for Heuristic-Optimizee Generation
Task:
You should design efficient metaheuristic using the following constraints. Your so- lution code should balance exploration and exploitation creatively.
Firstly, describe your meta-heuristic, including optimization strategy and main optimization steps in one sentence. The description must be inside brace and marked as comment.
Next, implement in Python a function named "optimize_algorithm: This function should accept five inputs: 'population', 'utility', aigguage_mddel jubtask_prompp and 'subtask'. The function should return three output: 'best_idea', 'best_ cosss_souution, "best_utiligs "utility" is function that evaluates solutions based on score function, 'subtask _porppppp""" is the format for model responses, 'task' is the name of the problem to be optimized. The function returns 'best idea' 'best solution' 'best _usilityyyyyy which are the idea behind best solution, best code together with utility.
Note:'language llanguage_ modell is an instance of the language model class used for code generation, batch.temperature),, return responses_ list" for multiple request L.M and "def prompt(self, expertise, message, temperature) request to I.M,, 'population'
sagguageemdeer"eoo with function "def prompt_bor prompt_batchlexxerttee batchleepperiisemesssss
result" for single dictionary several historical best solutions of task, you only use the following functions to operate: "def eet_solution___ index(self,, task jask_amme, index): return item", to get solution by its utility rank; "def' get random solution(sell, task name): return item". to get a random solution from the population; the item returned above is dictionary with keys 'best sol' and "utility". Other functions you can use are: "def get subtask size(self, task name):" return size the population.
Prompts inside Seed Meta-Optimizer
1. Idea Generation:
Given the following heuristic for task: ["els_soo']] with its idea: ['idea'] and utility score: l'utility"]. "Summarize the key idea from this heuristic, then provide several totally different or refined ideas from the given one to design improved algorithms with lower utility score. "Provide single string as the answer, less than 50 words. Your response should he formatted a json structure.
2. Heuristic Code Solution Generation:
Improve the following solution: (selected solution). You must return improved solution. Formatted follows: follows:(subtask, _ubtask_promppt).
better solve the problem, you are encouraged develop new solutions based on the direction proposed: [direction] You will be evaluated based on a score function. The lower the score, the better the solution is. Be as creative
can under the constraints.
Figure 5: Prompts for generating the heuristic-optimizer and those within the seed version.
24

---

Published as a conference paper at ICLR 2026
D.2 PROMPTS FOR FORMULATING HEURISTIC GENERATION
In this section, we present prompt example used to guide the generation of downstream COP heuristics, shown in Fig. 6. For different tasks, only the function signature and corresponding problem size are modified accordingly.
D.3 PROMPTS FOR HEURISTIC INITIALIZATION
To maintain population diversity, we initialize a population before training and retain elite solutions for idea modification during inference. Accordingly, we present the prompts used to generate diverse ideas that guide code generation at the start of each stage.
You expert TSP solver. Given set of nodes with their coordinates, you need to find the shortest starting node. The task can be solved step-by-stey by
route that visits cach node once and returns starting from current node and iteratively choosing the next node. Help me design novel algorithm that is different from the algorithms in literature to select the next node in each step.
First, describe your new algorithm and main steps in one sentence. The description must be inside brace and marked function named "select next node". This function should accept 4 input(s): "cur- rent node", "destination node". "univisited nodes", diistance matrix". The function should return I output(s): Inext_node"
comment. Next, implement it in Python
courent_nndee 'destination pnttaattionnode'',. 'next mext_node'', and 'unvisited_ noddessd_d'ee are node IDs. 'distance_man mmttrix' is the distance matrix of nodes. All are Numpy arrays. Do not give additional explanations. Your solution should be designed and fit for the task (prob) with problem size (size).
Figure 6: Prompts for generating constructive heuristics for TSP.
1. Generate seed direction in training stage.
You are an expert in the domain of optimization heuristics and combinatorial optimization problems. Your task is to design hcuristics that can effectively solve optimization problems. The problem is (problem) with corresponding size [size]. According the task description: desscrctccpption Provide several high-level directions utility as result. Format your response as a ISON codeblock
generating the seed prompt, each aimed minimizing the [{ ""direction": "ccoteett:: "Your first direction suggestion here.", " "content": "Your second direction suggestion here.", "content": "Your third direction suggestion here.", "content": "..."
Generate/Modity seed direction in inference stage.
You are an expert in optimization heuristics, tasked with summarizing key insights to design improved al- gorithms. Given the following hcuristics for the problem: (problem): (solution), please summarize the key insights in the heuristics to design improved algorithms for larger sized problem problem with corre- sponding size (size). Formatted json structure: "'json( ("insights":""coom "coottt""""""tttt""""""""" "content", "content" "content"] []] j. Remember each insight inside the list should one sentence than words.
3. Generate Code by Idea.
You expert in the domain of optimization heuristics and combinatorial optimization problems. Your task design heuristies that will implement a Python algorithm to solve a problem as well as possible. The optimization problem is (problem) and the size you should focus on is (size). The output function is formatted as folhoos ffomula_stry You are encouraged to develop the algorithm that follows the direction: (direction).
can effectively solve optimization problems. Write function
Figure 7: Prompts for code and idea generation during the initialization of training and inference.
25

---

Published as a conference paper at ICLR 2026
E EXAMPLES OF -LM-GENERATER HEURISTICS AND META- SETA-OPTIMIIERS
In this section, we present the best-perforiing heuristics for the largest instance size of each problem in Heuristic 1-6, along with several examples of generated meta-optimizers shown in Fig 8-15. These examples demonstrate that MoH can produce diverse, explainable, and effective optimizers that extend beyond traditional LLM-EC, incorporating a wide range of optimization strategies and generating high-quality heuristics for downstream tasks. For clarity and space efficiency. non-essential code elements are omitted while preserving the core optimization logic.
nunpy ap
selecs_next_nee jeuzzens_noda, destinat code, Lnvislted_nedes, distance natrini: can_unvisited Lon Len/unvaedioonnode
iE nun_umviaiter sindiiieddddee =- 0:
Nona
distances silstance_nlli www.m/tteeiiooeettttooooo unvisised_
avg_distance = ap.nean stncess
threshold 3.5 svg_distance
unvisiled_
threshold]
if len (ciase_nndess)
for node in close_notees:
iimediat_ddiatance distance__att nataia fa node]
futuro_pavingg mp.sumtdiiscanoonnntttaa
close_nodes!) (Len(cisso_nodes) 12
Len (eloas_nndes) > wlaw
diversity_acora ap.nean wtllsttnneenaaaiee (node, unviaised (imnddiace_distane + imeeiate
fotore sevings))) diseraity.score,
nut acosee:
nadenades unvialted_noden Idiscances threstoid]
for node in far_nodes::
[ande]
(ounc mode]
naxe_pade nin (scozes, kayesccres.os3 scores Nona return next mode
Heuristic 1: Best constructive heuristic discovered for TSP with size 1000.
des update_osqe__t www.:tttnncceeeeeeeooo.o..
local_opt_teux adga_n_uscd):
updated edoe _ddee_ddaaacce op.copy (edge_distanee
cun_nodes ien [occa__ooptt
vindow_size
for sange (our_)
cuzrent_city local_ [p]] roz 1 in range 11, windov_cize 11:
nun_nodes
next_city local_opt_touz [ox
used seee__igg_.courd wugp..n.aeedtrrrent.... seal_eilt]
12 used_odga__oou
scalina_factee ap.icg jused_errs__eaa
0.5
wedatten_.ddeeddsstaeee fourenanooueee cixiycity scaling
dedee (ntestaannnneetttteete ausren=_city! scaline_
clse:
factor op.expt-0.1 uaed_edge_conn))
aex__city] dacty_factor
updated ad edge stance [next ourren < decay facterr
edoe quality adgg__dataane [ourrent eitby, nixi_city] edged_edge_count
uodated_odee cecee__ddo____aa distancc [cursent_city next_city] adea_guality
updatedd edge ance city, carrert_nriy) t- edge_quelity
secuen updated distance
Heuristic 2: Best improvement heuristic discovered for TSP-GLS with size 1000.
Ampost nunpy as ap
def adaptive_indie wnw.caiciatooosssssoooooooooooee (distance_matriaa
nun_nodww distance mecennetiii shape[0] Indicators
(munnnodes, aun_nades))
min_edge no.full (nan_nodes, np.inr) minumnige(0]
visited op.2eros (nun_nodes,, dtypesbool) toral_mst__re
fox cange (oda))
ag.arcnin tap. (apr (vialted, ap.inf, min_edge)) visitedlul True
min_ndge(u]
v rance (nun_nodes):
18 not visited[vl and min_edge[vi iissanoe_matziat v] Inverted distance tocal_densstyy sp.ein(inverted dereetedddiee ance exisel)
natrixlu, v] innneddeevv
(distance
np.eyeinun nodes)
Fox in sange (oddaa)
for te inde_indicttos Locillddmsityf
frun nodes):
docal_demitty)))
Lotai_density(1))
total denantyy lill
edge_cost distance iistanne_mmeiia netrixli, ]] mosta__mtt_ooe / (oun_rades 1)1
sycie_penalty np. (iinnvereee discance, ceatancceeatrttiieeeeeeee inverted disaane, sieettddddissanoommerrii mattenii..
Inverted disterce netrixli, ill distance natrix j1. 0.2)
26

---

Published as a conference paper at ICLR 2026
indicacoEs (O made__adioate pycte_ edge_coss) nox_indicater ng. (indica:ces)
it nex_indicater >: indicators nax_indicator
indieatces
Heuristic 3: Best improvement heuristic for TSP-KGLS with size 1000.
Ampost nanpy ap
det scoze titem, ainst:
scores zeros (bins, dtype-flosti
foasible_sins bins iscm)
=ossible_iins.size return scores
nox_capacity ap.max
scoresibing mex_capcitty
semaining_caprcitty ifoas-ble_bing iten) Foariblo_Dins
itcn_ratia iten teasteic_blas
sroximity_pensitg ip.whareettesciile..btt >- iten 0.99, -b, 03 np.shere iteas:ble_blne item o. 90,0,
underut:lisatien ponaity
(p.maximum(C,
reasials_ bins)
sconesibine iter] (remaining paa.iinnnepppcityt proxinity_peealtg unsorutt iten_ratiol ** 31
Heuristic 4: Best heuristic for online BPP with 10000 items and bin capacity of 500.
computa_edgge sacpate__ee____eeeeeeeeeee
(doransddietttcc_mettt import numpy ng
coordicates, eapacityl:
man_nedes distanes_natrlle shapelO]
pcs.piioniinneess ap.zeres (nun_nodes, aun_rodes!)
total demand demand docay_factor 0.35
[denands]
adaptive aipas 1.5
edept ve beta = 2.5
for in
1 in range nodee))
if 1 and denanes cepacity:
distance_scos 11 idistance_matrikg 31 1e-6)) ** adaptive_bette cenandeljl
demand_
demand total demand else
sharonone_
adiatancs_natriitte 31
decay_festor
explorstion_ tactor edge pronieingmessll. semands capasityl
cdistance__ecs score ** adaptive_betar Idenand, score
edaptive_sipha) pharonons_ieva. exploration_fector bége_premisinngeee
Hcuristic 5: Best heuristic for CVRP ACO with size 200.
Lngust
def panpute__air (damand, capasityl: demand. shape(0]
bwu.iat up-serou (o, n))
indid_indicss ng..heze for velii__indicess j valid_intiows:
(deehhe (demand <e capacity) [3]
12 !a
total demand[i] denard]]]
dotan_d dapacily:
beuristic_matiixi11 capacity tocal_domand (domand[[1 danand[
For in cange(o1:
for sange [n):
dimgie_demend denend demand [i]
if damand de capacity:
heuristic matrix(i] op.sun (beur Fox sange fal:
(searis:i natrix(1113) capacity demand denandl axisel)
Frequency pount
for j range in):
15 la and te
seuristic met.eceettteiii1133333
treguency_ccmme count
denand group sp..iiitiizeteee toemannd,,, binaenp,1 Tepace (6, cepacity, nun=5))
For yroup in sange (1, 5t:
group indices np.ubere (demand pntlgroug_indiieee
gropp_ind.css
for in group indices:
if heurietie_nacesiilie
and heuristio weriiatic..trriiile
for in renge
For j in range (n):
trequency
and trequency count
(Ereguency count frequency
return iounatrix
Heuristic 6: Best heuristic for Offline BPP with 1000 items and bin capacity of 300.
27

---

Published as a conference paper at ICLR 2026
Seed Meta-Optimizer Example
def optimize
utility, language_modee
subtask subtask_prompt, subtask)
expertise "You are expert in the demain of designing meta optimization strategy and combinatozial optimization problems. Your task is to design houristics that can effectively solve optimization problems."
# Step 1: Select random solution from the population selected_solution
stlllatiionnget_rnnnn____ssss sraatommssluttion (subtask) # Step 2: Generate directions for improvement direction_
E"Given the following heuristic for subtask: |selected_solite solutionl' i'dea') and best_sol'11 with its idea: ('ttiliyy") utility score: [selected_solutionn "Summarize the key idea from this heuristic, then provide several totally different ideas from the giver one to design improved algorithms with lower utility score,
"Provide single string the answer, less than 50 words. Your response should be formatted as a json structure: [""oottt \", "ccntent \ccntent\", \"content\"]]
response language_medel. prrooppppmdppppppptt prompt (expertise, direction_prompt, temperature-1)
di directions = j loads (extract_ (response)) ["insights"]
# Step 3: Create messages based generated di rirccioons sassage_batch
for direction in directions:
message =
f"Improve the following solution: An" fn
An "An"
f"You must return an improved solutien. Formatted as follows: n|subtask Battaskkppompt/aar
E"To better solve the problem, you are encouraged to develop new solutions based the direction proposed: (direction
"You will evaluated based on a scere function. The lower the score, the better the solution.An!
"Your response must. firstly provide a surmary of the key idea comment, followed by the inside brace and marked code implemmntation.
"Be as creative as you can under the constraints."
message batch. append (message)
Step 4: Generate new solutions using the language model responses languagge_nmdlllroooteeaaaaccc (experrtiss,,,, message_batch, temperature=I)
new_solutions extract_coooe (taeracc_codeerresoossesss
new_ideas extract_idea (resspnses)))))
# Step 5: Evaluate new solutions solutions_ wtt.onss.wtthtuiiiiiee
(iieea, solution, (eility idea, subtask) ) for idea, solution in zip ideas, rew_solutions) best _est_idea, best past_sclutien, best past_utility-s min( celuuiios_withhutiiittte return best_idea, best_sclution, best_utility
key-lambda x[2))
Figure 8:
The seed meta-optimizer used for training, which randomly selects previous solutions and generates new directions for improvement.
28

---

Published as a conference paper at ICLR 2026
Genetic Algorithm (GA)
def opt ml gee_algrrthh (popu ion, utility, language_mon model, jubtask_prompe aubtaak) if Adaptive ive Genetic Aigozithm incorporating selection, crossovez, and mutation to enhance saiution di diversity expertise "You are an expert In the domain of designing neta optimizst lon strategy and design heurissics
convergence
combinatorial optimization problema.
effectively solve cptimizaticn prohlems."
f Step i: Select top-performing solutions for breeding
elite_ccunt max population.gee get__eettss____eeeeeeee__ taubtaak) // 131 Top 109 elites elites coopulation.ett_olll by (subtask, i) for in range(ellle_couut) 2: Geserate directions for diversity through crossover and mutation directicn directicn_prompt = "Given the top performing solut ions, suggest innovative crossover create diverse
offapring. Provide your respanse an JSON with keys srossovar_mathode and 'matation_methods' each conca'ning a of strateglea."
mutation strategies
Language_nodell promot (expertise, direct ios Lemperature-0.7) directiens json. (extractettcttccooom code (response)
crossover_methods directiens. t"musstionn ion_nttoooe []] dizections. set( get mutation_mete offepring solutions using the genersted strategies
"crosssver_nethont
Step
offapring = []
for method crossover_metheen
for i range(ellte_count)
parenti elites[i]
parent2 elites[
elite_count]
crossover_pronee = E"Apply the following crossever strategy to combine these two
solutions: Ansolution
"ppthhnntn.pprrett (atrrentti"bettppooo ['best_sot [nn Mnnoouutito
"pythoon (parent2 ('rent2['bess ["best_sol ( "n)""" "inttrattgg [nethod] AnFrovida the new offspring solution as JSON with keys "Idea" and "best_aol", oéEapring_responeee lenguage_nodel.pet
lexpertise, croseover_proapt,
temperature=0..7
offapring_dare jaon. loads (extreet code (offaprinn_rrepponse)) method mstation_methode
for elite in elites:
mutation_prcmp: ["Apply the following mutation strategy to this solutions
nSolutien: ""pythonin/elite ('best_sol' 1Fz PM NeStrategy (meshod] AnProvide the mutated solution as JSON with keys 'ides' and "bess_nol",, ".
mutatton_reaponse larguage_maat madell proopt (expertise, mutation pon_nppp,,, temperature =0.71
mutarion_data json. (extract_ (mstation (mstatioo_rreppreee offepring applnng inutattoaaat
Step 4: Evaluste offepring and select the best solutions aolutions_iith_atillleee atiiiitiiiiiiies
(child[' idea'l, child("best_eol"), teilitty(hhiddd'esttt.. chi Idlld subtoski) for child in offapring
Tholude
maintain
clita_scluse selta_ccluiions = (elita["idea'), "iett(bbet__cc")) elita('stee ('ttiliyy) for clite in
elitea
all_candidales wtt.iiion_tttttttllliii.
elite_solutions
best_ides, best_scluticn, best_urility candidatss, key=lambda x: x(2)) return best_idea, best_solution, best_utility
Figure 9: An example of the meta-optimizer generated by LLM, which employs Genetic Algorithm (GA) to balance exploration and expoitation, similar to previous LLM-EC heuristic- cerritiicoptimmzzr.
29

---

Published as a conference paper at ICLR 2026
Ant Colony Optimization (ACO)
algoriths employa Ant Colony Opt imiimiaiio (ACO) principles simulate foraging process of ants for efficient bin packing soiutions, enabling ad ive phezomone adjustment and heuristic guidance to batance exploration explottation for omtimal results.)
def optimize__lgpritiiipppr ipprrittmpppuulttion utility, language_model, subtark_prompt, subtask); expertise "You and comhinatorial opt imizatttion problems. Your task 1a to deaign heurlatics that can effeclively
expert in the domain of designing neta aptimization strategies
optimization problems."
8 Paremeters for 200 and solution selection ant count
elite_ccunt population_size population. get__sstask__aaeeeeeeeeeeeeetete size (subtask)
Step Select olite solutions based stiliey for inieial pherosone distribution
elite_sclution
range (minteeittle wtt.ooottoo__y__ndeeee (nd_xxxxxxxxtattt 1)
wwgulaatoo..get_eoltiioo.oo
population_siia
initialize gheromene leveis for directions based on elice solutions
pheromone_Jeves (soll'best (''bettt_set bea'] 1.0 for sol sdite_sclution
2: Senerate solution directions using Anc Colony Optimization principles directicn []
for range
direction ["Uaing the ACO principles for the task (aubbaakk), generate
solution direction that addresses the bin packing problem. Consides current elite selutions: [[so] bestteeolll for sol in [oita_solu:ion]] And ensure that the output follows this formats (eubtask_pronpt). Provide surmazy
the idea in braces.
direction apo Step responses
rect wtw.ppeellllelpppp.eeeeeee (expertise, Airection_prompe Lempezatuze=O..)
the Janguage model
new diw_diiestions extract_code (cesponses)
Step 4: Evaluate new solutions colutioos_uff wtt_uutthttttttttttta = for direction in nev_directions:
try:
Evaluate
soiution's acillty
score utility ldirection, "Derived frem &CO strategy", subtask) scluticns_win wtw.iinsswttuuutiittttt appiend
sppena((directtio)) (ddirection, score)
# liddate pheromone based on the quality of the direction sesromone_leveile (direction)
inerrmone_levelss.co (direction, 2.01 2.0 (score
1e-6)
except Exception as
# Skip utilily evalvation fails
Step 5: Seiect the best new solution based on its utility score not wttrsoon_kiii_ntiisiiiii..
Fal Failback to the beat exlsting solution if no new so solutions are val beat_exist
wow.llttooo.geeolutttte byy index (subtask, 0) beat_existting.c utillty")
idaa),, pest_existing ing.g.get "eat__aol"), best_extlstinggget)
directin,,,,,,, best_utility min utilitiee. key-lambda x[11)
return "Anz tolony optimized direction", dese_dire best_atility
Figure 10: example of the L.M--genrrted meta-optimizer that utilizes Ant Colony Optimization (ACO) its underlying mechanism.
30

---

Published as a conference paper at ICLR 2026
Particle Swarm Optimization (PSO)
def
utillty, language_nodel, subtask_proope subtask)
expertise *You are an expert in ...
Parameters for PSO
particle_count = 15
iterations
inertie_weight - 0.5 Controls exploration versus expicitation
cognisive_paran
Personal attraction/leazait factor
socisl_param 1.2 Societal sctraction/Iearniie factor
Step initialize particies with random solutions in the population
beat peat_personal1 _aoluttonoo = oet__rardor_collsionnniiii
(etticlee=ppcuulatiion htt.llaiioo.gtt.rrdooooooo
for sangelparticca_.coum))
global_besi_Bolutt
sintparticles, key-lambda atility(x (x "Initial 250" sustask)] Initialize global best solution Step iterate through
process
for range (iteration)
for particle in particles:
aurrent_acors utility(parttii ('bast b'bast_sol')) "PsO iteration", subtask)
Update personai beat If current acore is better
cuzrent_acoze utility beees__prrsoaa_so0 iollttionssee ions (particles (particle]]
best_sol<), "ps0 personal best", subtask):
update zlobal best if current innriiicce....o//parrilal... (particle]] current_score <
bett_peraonal_oouriio (particlea..nddas
particle
is better
(""ootttto "Pso giobal best",
subtask):
plobal_pest_solutiee - particle
Generate directions using influences
new new_direst directionss =
for particle in particles: Compare new velocities pesitions P50 rand))
r2 ap.raaddm.randu Signitiiv__vlocitye index [twarricllllleetttt.eee
cognitive_peran ri
secial_veloctty sociel_param r2 bestdll_eest__ouutiie [best__sll"] particle
['best_soi'1]
new_pasition particle["hest_cnm
inortia_weight (cognitiive__elocctyy
socfal_velocityy
prompt f"Using FS0 principles, generate solution for the task [aubtask]] based on the solution [new_pos'tions Ensura to fol fol low this format: { sublack_orompt) Provide surmary braces," step
wee._direticcss.epeedppppp appead
respenses nex solutions batet_bstchhhee (expartise, nev_directions, temperature=0.1)
language
senerated_solutioonn
extract_codd code (responses)
Step 5: Evaluete solutions for new_sclution in generated_selutfone
try:
score = utility (oew_nolution, "Derived from PsO*, subtask)
score
('bast_sol))
global best",
subtask):
global_best_solle ( ['best_aol's new_solution, futility's score)
except Exception:
continue & skip errors in utility evaluation
return "Particle optimized direction", bediat_o sow_ottttttttooo/oootooooo
Figure 11: An example of the meta-optimizer generated by LLM, which employs Particle Swarm Optimization (PSO) to balance exploration and expoitation.
31

---

Published as a conference paper at ICLR 2026
Combination of Adaptive Simulated Anncaling with Differential Evolution
def aetimmiae_agoorithmmeee
utilliy, Language_model sublask_prompl, sublask)
swarm_size
perens_nolstions (population.get_ sorrddon_oouttionnnieeeeo current_utilittes [parent] [futt Hty') for parent in paront_solutions)
zange
initial__empzsture 1.0
coeling_rate = 0.35
iterations -
best_solution
best_atility ap.inf
cerparature intrial__emparacure
for lteration in range
Dynamic temperatuze adjustment based on utility improvements temperature maxiinitiel_reeperaaa
iecoling_rate iteration), 0.01) Ansure
Lempezature doeen't go below threshold
piitial_prompts f"Using parent_ sorens_aolutionn]) with utilities:
following seluticns: [
devise advanced enhancement stteggie by blending Adapt simulated Arnealing and fifferential Evelution Ior the Lask: (sublasx]. Return sizelegies in JSON Tormal: ">3aaonha ((Vstsategiess>: [>"s:sategyl"", \"strategy2\+ .... \"strategyl)" ""." expert optimization strategist.",
raspense wawguage_modsl.pppog. wttpm/mdddl.ppppppp...p. (*You
initiel_prompta temperatuze-0.1)
porantial_ttrategiess ison.loads (extract_ (respensej) ("strategies":
for strategy in cottertil__ttattgiiiee modi:iied_ooltiionn for in perent_solations:
Apply each scrategy to parent solutions
message ==implement the stretegy: (strategy] on the parent solutien:
python\s/parent poast_soll]i's ( Generata madifiad solution adhoring to (subtaak_promptt Focus on creat ing diverse and high-quality outputa." response language_model.ppoe ("Yeu an expert optimization stracegist.", massage, tempera:uread,1)
new_solution extract_code (response)
new_utility (silityyee_seoutiiinn "Optimized by Adaptive Simulated Annealing and DE*, subtask! neu_utility best_utility:
best_utility new_utility beat_aolution rex_solution
Explore new areas based probaciiicy
elif rgp.random.aante
s.p.ep (futility iparent ("best_so1") ), fOriginal', subtask
} nev_utility) temperature)
modified aopereiccnnccceeeeeedei (nex_solttt
Opt tonais Teplement dirferential avolution combintng beat solutifons
if modi
hest_af_man wtw.offfoddiieetttecco.
min/modi fied_scllllleeee ved_scluttoon, key=lambda 5: ut liyli,, "Mods
subtask))
nem_utility of
*Frem Diffecential Dvoiution",
subtask)
nev_utility < best_utility:
best_utility new_utility
best_solutia
fest_of_modificee
beat_idea Idea 'Taplemen Tmppeemented hybrid approach Adsptive Stmulated Annealing and Differential Evolution for eahanced search efficiency."
return best_ides, best_solution, best_utility
Figure 12: An example of the meta-optimizer generated by LLM, which combines Adaptive Simulated Annealing with Differential Evolution explore solution space while refining candidates.
32

---

Published as a conference paper at ICLR 2026
Tabu Search
def aatpiiiitaggrrtthmmmm (pepulation, utility, language_model) subtask_promge subtask)
# This algorithm utilizes Tabu Search to dynamically explore and adapt solutions, avoiding cycling while maximizing job performance and minimizing makespan.
solucton__oolution pepulation. aataatiiiongetradddnnnoeee colutionssubaasss best ins,_solution, best_utility selected_scluticnn best_ssl') in (' cabu_list set
iterations
max ittertiinnnnttt
while iterations max iterations:
# Generate candidate solutions
candidates = []
for in r3)ge(3): Generate candidate solutions directionnprompt =
f"Given the heuristic: (selected ('best_sol')
with its idea:
f" [[[selected_sclutte [' ]] and utility score:
(utiluittyoutiiitlll l'uullity'l,,
"Generate a modified or new solution. Format as: f" """sutaask_prong new job scheduling netrics."
Ensure it is innovative and consider
response language_model
("Yeu are an expert optimization. direction prompt, temperature=0.1) modidate_ccde extract_ code (response)
aaapadat Evaluate candidates
candicato_utilliccs []
for candidate in candidates:
candidate_idda ietract_idea (candidate)
candidate_utility utility candidate_codde candidate_idea, subtask)
candidate uardidate_tilitieee. sttnee_ttiiittissppeeeet (candidate idea, candidate, candidate_utilityy:
# Apply Tabu Search Logic
feasible_cancidatess Ic for in candidate_utillttis if c[1] not tabu_list]
if best_candidate
key=lambda x: x(2))
best_ lest_candidate_iie _ieeate___deee___eeee best catdttaandddteee_ee stteiiiitt_sslttinn
rest__cardidate_utiiitt
best_candidate
Updato best solution found a better one
if best_candidate_uuility best_utility;
best_soluticn, best_utility = best_caaddaate__olutiiee htt.oeeesooiiiooooooocaaaaaaooo.
best_ccandiddctappti1f1
selected seleette(_ccclrricm[(1eee111
sel
best best_candidate_jieee
best_canddidate_solit
past_candidatt_uutil
Update Tabu 1ist
tabu ldd (best
(beet__annidaeeeeeeeeeeeeeeeaaee candidate solution)
len (tabu list) 10: Maintain fixed size
tabu_list. iterations += return selested_sclutionn idea'), best_selution, best_utility
Remove the oldest entzy
Figure 13:
An example of the meta-optimizer generated by LLM, which employs Tabu Search to dynamically explore and adapt solutions.
33

---

Published as a conference paper at ICLR 2026
A Neighborhood Search-like Strategy
def optimize_algorithm (pepulation, utility, language_modes
subtask pooopp,, subtask)
8 This approach utilizes Variable Neighborhood search to explore different solution neighborhoods dynamically to enhance optimization performance.
def generate_neighborhoode (solutiion)
Generate modi fied solutions (neighborhoods) by tweaking routes based on the original soiution neighborhoods
Simple route perturbation: swap two routes or modity delivery sequences
for i in range
if (scluciion
['best_sol'] beesissoooeetttoo______ [il, list) and lenlsoution
$ Swap two locations in the route
new_soluticn = solution Iaalution['beee b'ees__oo'' new_solution(i] new_solution(i] [0:2] Swap first two for simplicity neighaorhoods. appendd ("oestt_o"""" new_solution, "utility": utility (new_solution, subtask)))
now_solution(i
return neighborhoeds
selected solution pepulation. get random solution (subtask) neighborhoods generate hewgheooooghhbohooooooooooooooo (selected solution)
8 Collect insights to improve solutions based on the generated neighborhoods
message_batch for neighbor neighborhoods:
message =
E"Improve the fellowing solution:\n \n"
\n "nn
2"You must return an improved solutien. Formatted as follows:\ n/subtask_proct prompt
"To better solve the problem, consi der how the neighborhood structure changes the zation landscape.
message
nessage_batch.co batch. (messaggee
responses language prompt pprppp_baacchhathtpperrppaaaaaee optimizations. mass,ge_batch, temperaturo=0.7) new_solutions extract_code (responses) now_ideas oxtract_idea (responses)
batch ("You are an expert in
# Evaluate the new solutions and select the best one [(idea, solution, utility (solution, idea, subtask)) for idea, solution zip(new_ideas, rew_solutions)) best best_ideat _iee,dea,___ best sosttici best_ best_utility = min( wit.annssiittttttiiiee.e...ee.ee key=lambda x121)
return best_idea, best_salution, best_utility $ "beet_soll; new_solution, "utility"; (tilityynew_soluttii task)
Figure 14:
An example of the meta-optimizer generated by LLM, which utilizes Variable Neighbor- hood Search to dynamically explore solution neighborhoods to enhance optimization performance.
34

---

Published as a conference paper at ICLR 2026
Adaptive Exploration-Exploitatonn Strategy with Dynamic and Tabu Mechanisms
def optimi seaalgor hm (population, utility, language_monn model,, subtask_prompt, aubtaak):
elite_count Fazameter Initialization diversity_count
phecomone_leveli - (1
1: Seiect elite and diverse solutions
popslation_size population.get get_geettack_sizzeee.e.. (oubtaskk
elita_solutiier [populati an.get_aolution indroon_y_iidxxxx (subtask, for in range
elite_count, popilation_siiee
diverse_solutions. [pepulation.get_.aatt htt.://cc......eeeenooosttttttaa (subtask)
range!
diversity_count:]
seleated soaated_oolutions alite_solutions + diverse_solutions
Step 2: Generate dynamic exploration insights with adeptive for solution in selected_ coluciionn temperature
solution (
else 0,75 Dynamto adjustment
prompt ["Siven the solution [sslltiion [o (arlooonbbbbeeeee ['beet_sol]]] with utility acuze "faolutiont'
utility'l)', please suggest innovative optimization strategies could enhance this cade. Return your recommendations in uSON format: "ijoon iiN"ingghtto\" [1 content)", "conttnt +...+
respense language_model, ppoopp (expertise, prempt, comprattur-ttemecrture
try:
insights 1soa. (ext (responael) ["insights"]
for insight in insights:
paeromone_levell [insight] theromone_levels..o gez (insight, 1.0)
except 35son:8800accodefrro: KoyError!:
step 3: directions besed pheromone levels (dynamie evaluation)
forted_inaight sorted peroonelleellsseeee ls key=lambda x: x(1., reverse=Trual top Lop_iasighis [insight
insight vorled_insighte if insight # Step 6; Create batch messages for generacing aptimized solutions based on directions message_bazad
for direction in iop_iniights:
measage [*3efine the solution for the task "subbaas] by Iocusing on this optimization approach: rectioonn Conaider elita solutions: [[ol ('best_sal") for sol in elite_aolutionn J. Enaure your output adheres to the foll Cormat: p addition, provide summary changes
bassage_btcca.ppppe appendd (message)
through prompto reaponses language_noe sese__odeee,___ ppropp__atton (expertise mesaage_bacch, temperature-0.99 é Step 6: Evaluate and keep the best pezforming solutions solut https/swthhattiitttttt for responae in responses:
Step 5: Generate new solutions using the language
try:
new_solution extract_code (response)
important_1dea extract_ dea (response)
utility Inew solution, important_idea, subtask)
cheromene_levelee (tweeelssmmorratt____.oo [important_idea) 1e-6)
paeromone_eevels, iimportant_ideet 1.0) + 12.0
wuuiiinsswithuutiiiii. (imi:es.sppeeddmmmmtttttttt (important_idee, new_solution, score))
except excepticn:
continue
solutions_ soluttooss_tto__ttille wis_ Rest Solation Selection
beat_existing
gegulation.geieeottiiii bideeyyy_xieeeeeeee (sublask,
return best pest_exissing(" calissing(iiees idde'),, best best_existing!" beestiinggbeetssslll best_scl') best_ best_existing (uiiliiy
best_ides best_sclution, best_utility min(solutions_ wtt.iooswwt_h__iiitteeee. key=lambda x[2))
Adjustments for ongoing adaptability of pheromenes
key theromcne_ievele
phercmone_levee [key] += 0.9 & Mild evaporation aolow expioration
zeturn best_idea, best_solution, best_utility
Figure 15: An example of the meta-optimizer generated by LLM, which employs an adaptive exploration-exploitaiii strategy that combines real-time performance evaluation of solutions with dynamic exploration rates. This approach customizes search focus within a genetic algorithm framework enhanced by adaptive, tabu-like mechanisms for efficient solution refinement, achieving the best performance during inference.
35

---

Published as a conference paper at ICLR 2026
F BROADER IMPACTS
This work explores a general framework for improving COP heuristics through LLMs. By introducing meta-optimization structure, our method demonstrates how LLMs can autonomously generate and improve heuristics across diverse problem domains such as TSP, CVRP, and BPP. Potential social impacts MoH may include: 1) Improved optimization capabilities in practical applications such as logistics, manufacturing, and resource allocation; 2) Bridging AI and Operations Research (OR) by designing a unified framework that benefits both communities, especially when solving problems with larger sizes; 3) Lower barrier to high-quality algorithm design, especially in low-resource or less-studied problem domains where handcrafted heuristics are not readily available. Meanwhile, potential negative impact of our method lies in the reliance on LLMs, where both training and inference involve substantial token usage. This can lead to increased energy consumption and raise environmental concerns due to the computational resources required.
G THE USE LARGE LANGUAGE MODELS
In addition enhancing the paper writing, LLMs serve generate and refine optimizers, as well as to support downstream heuristic generation. More precisely, LLMs are used to produce and optimize code implemeentations aimed developing high-performing heuristics for solving COPs. A detailed workflow of LLM involvement is presented in Section 3.
component of our methodology to
H LICENSES
We list all the used assets and their licenses in Table 18.
Table 18: Used assets and their licenses.
Тype
Asset
License
Usage
Concorde (Applegate et al., 2003)
Available academic research use
Evaluation
OR Tools (Furnon & Perron, 2023)
Apache-2.0 2,0 license
Evaluation
FunSearch
et al., 2024)
MIT License
Evaluation
BoH (Liu et al., 2024a)
MIT License
Evaluation
Code
ReEvo (Ye al., 20244
MTT License
Evaluation
HSEvo (Dat al., 2024)
MIT License
Evaluation
MCTS-AHD (Zheng et al., 2025)
MIT License
Evaluation
POMO (Kwon et al., 2020)
MIT License
Evaluation
LEHD (Luo al., 2023)
MIT License
Evaluation
Dataset
TSPLib (Reinelt, 1991)
Available for any non-commercial use
Evaluation
36


Reviews:
[{"review_id": "14zufM0Hh9", "reviewer": "Reviewer_LJhB", "summary": "The paper proposes MoH (Meta-Optimization of Heuristics), a two-level LLM-driven framework that searches not just for task heuristics, but for the heuristic-optimizers that generate them. An outer loop uses a meta-optimizer to invoke an LLM and produce candidate heuristic-optimizers, while an inner loop applies each candidate to evolve concrete heuristics for downstream COP tasks. Selection uses utility on validation tasks, and training is multi-task to encourage cross-size generalization. Experiments on TSP and online BPP report state-of-the-art gaps, especially on larger, unseen sizes. Further ablations suggest benefits from maintaining populations and using natural-language idea descriptions.", "questions": "Which two main loops make up the MoH framework?\n\nDoes the paper claim cross-size generalization on TSP?\n\nDoes the method rely on a population of candidates during search?\n\nWhat is the difference between heuristic-optimizers and meta-optimizers?", "limitations": "", "rating": 8, "confidence": 4, "soundness": 4, "presentation": 3, "contribution": 4, "strengths": "Ablations show benefits from the proposed idea and examine different LLM backends and population sizes.\n\nThe paper provides concrete examples/analysis indicating discovered strategies can resemble or hybridize classic metaheuristics.\n\nThe paper shows strong empirical results and cross-size generalization on TSP and Online BPP. The proposed MoH often outperforms baselines.\n\nMulti-task training and controlled evaluation budgets are thoughtfully designed to encourage generalization.", "weaknesses": "Improvements over strong baselines can be modest in some settings. Further discussion should be included.\n\nAccording to experimental setups, main tables emphasize best-of-three runs, which can overstate gains versus mean/variance reporting.\n\nThere might exist sensitivity to LLM choice and prompts. The robustness under model drift is uncertain.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "9TIyFW7Dhm", "reviewer": "Reviewer_9XE1", "summary": "This paper proposes Meta-Optimization of Heuristics (MoH), a novel framework that uses Large Language Models (LLMs) to generate generalizable heuristics for Combinatorial Optimization Problems (COPs). Unlike prior methods that optimize heuristics directly, MoH operates at the optimizer level. It aims to discover a highly effective \"heuristic-optimizer\" by meta-learning. This process involves optimizing a meta-prompt that guides an LLM to sample and refine heuristics. The meta-optimizer is trained to maximize a utility function across a diverse set of tasks (e.g., TSP instances of varying sizes), with the goal of producing heuristics that generalize well. The authors evaluate MoH on the Traveling Salesperson Problem (TSP), demonstrating improved performance over several baseline methods.", "questions": "- If the experiment is run multiple times, will it produce a \"meta-optimizer\" with similar performance, or are the results highly variant? This is a critical point for assessing the method's reliability.\n- Why was the utility function weighted by the size of the task? Figure 1 seems to suggest that performance suffers when emphasizing larger instances. What is the performance of the MoH framework when using a uniform weight for all task sizes? What is the performance if the baselines use the weighted utility?\n- What is the performance of the final heuristics obtained at the end of the training phase? This would help clarify the exact performance gain and cost attributed to the separate inference stage.\n- Could you clarify the practical difference between the heuristic generation strategy used in the MoH inference stage and the strategy used by the baseline EoH?\n- What specific heuristic was tested on the TSPLIB benchmark? Was it the single best heuristic from Table 1? If so, from which problem size distribution was this heuristic generated?\n- Why are the results for the ReEvo baseline missing in the TSP-GLS case in Table 1?\n- The paper requires careful proofreading to correct several typos (e.g., \"generats\" in line 74, \"hsmaller\" in line 928).", "limitations": "", "rating": 4, "confidence": 4, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "- The core idea of optimizing the optimizer (via a meta-prompt) rather than just the heuristics themselves is a novel and interesting approach to leveraging LLMs in the optimization domain.\n- The paper correctly identifies generalizability as a key weakness in existing heuristic generation methods and explicitly designs its utility function to reward performance across different task distributions (i.e., problem sizes).", "weaknesses": "- While the method is described with complex terminology, its core mechanism appears to be a sophisticated form of meta-prompt optimization. The \"meta-optimizer\" is, in essence, a highly-tuned prompt that guides the LLM to sample effective heuristics. This idea, while implemented well, feels intuitive and perhaps more incremental than a fundamental breakthrough, which may limit the paper's conceptual contribution.\n- The paper's \"generalizability\" claim is weak and potentially misleading. Firstly, the framework does not generalize across problem domains; it requires training a new, specialized \"meta-optimizer\" for each problem class (TSP, BPP, CVRP). Secondly, even within a single problem, the generalization is limited to varying instance sizes from the same data distribution. There is no evidence that the optimizer generalizes to new instances drawn from a different distribution (e.g., from uniformly distributed TSPs to clustered TSPs).\n- The experimental comparison to baselines appears to be unfair. MoH's computational cost includes both a training phase (1,000 heuristic evaluations) and a separate, additional \"inference stage\" to generate the final heuristics. In contrast, baseline methods like EoH are presented as more \"online\" and may not have this distinct (and costly) inference phase. For a fair comparison, the baselines should be allocated a total computational budget equal to the sum of MoH's training and inference costs. This is particularly concerning given that the performance gains reported in Table 1 are incremental.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "OQnhDMgP6t", "reviewer": "Reviewer_9cf3", "summary": "The paper presents Meta-Optimization of Heuristics (MoH), a framework that uses large language models (LLMs) to generate effective, interpretable heuristics for combinatorial optimization problems (COPs). Unlike traditional methods, which rely on predefined evolutionary computation (EC) heuristics or single-task training, MoH uses meta-learning to automate the design of meta-optimizers, enabling broader heuristic exploration and better generalization to new problems. The authors demonstrate MoH’s superiority over existing LLM-based methods on classic COPs like the Traveling Salesman Problem (TSP) and Bin Packing Problem (BPP).", "questions": "1. Please list up and carefully describe any questions and suggestions for the authors. Think of the things where a response from the author can change your opinion, clarify a confusion or address a limitation. This is important for a productive rebuttal and discussion phase with the authors.\n2. While MoH performs well on classical COPs, its scalability to more complex, real-world optimization problems (e.g., dynamic environments, non-classical COPs) has not been thoroughly tested. Can you provide any insights into how MoH might adapt to these problems? Have you considered testing MoH on real-world benchmarks or dynamic problem settings?\n3. Multi-task learning is a strength of MoH, but it could also lead to overfitting, especially when tasks are not sufficiently diverse or are too similar. The paper does not clarify how overfitting is mitigated during training. Could you elaborate on the strategies used to ensure that the framework generalizes well across tasks? Did you apply any regularization techniques, cross-validation, or other safeguards to address this risk?\n4. While MoH expands the heuristic search space, how does it perform when compared to simpler heuristic methods or other optimization techniques, especially in terms of efficiency? Given the large search space, how does MoH ensure it doesn’t waste resources on ineffective explorations? Could you provide a more detailed analysis of MoH’s efficiency in exploring vast search spaces, especially for large and complex problems?", "limitations": "", "rating": 4, "confidence": 3, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "1. MoH introduces the idea of meta-optimization within LLM-based heuristics for combinatorial optimization, addressing key limitations of existing methods like the lack of diversity in heuristic exploration and challenges in generalization.\n2. Extensive experiments demonstrate that MoH outperforms both traditional and LLM-based heuristic methods across various settings, showing its ability to tackle problems like TSP and BPP effectively.", "weaknesses": "1. While the authors claim that MoH does not incur significant computational overhead, the introduction of a meta-optimization layer adds complexity, which may increase the time and resources required, especially for large problems.\n2. Though MoH performs well on classical COPs, its scalability to more complex or non-classical optimization problems (e.g., real-world applications) has not been thoroughly tested.\n3. While multi-task learning is a strength, it could also lead to overfitting on the training tasks if not managed properly. The paper doesn't provide a clear strategy for mitigating such risks.\n4. While MoH increases the exploration space, there is no detailed analysis of how efficiently it can explore very large or complex search spaces in comparison to simpler heuristics or other optimization techniques.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "yw8blBTMwE", "reviewer": "Reviewer_7Ges", "summary": "This paper introduces Meta-Optimization of Heuristics (MoH), a new framework that uses large language models (LLMs) to automatically discover effective heuristic optimizers for combinatorial optimization problems (COPs). Unlike prior work that relies on fixed evolutionary computation (EC) strategies and single-task training, MoH operates at the optimizer level. It employs an outer loop for meta-optimization (optimizer design) and an inner loop for heuristic design. The framework uses multi-task training to improve generalization and supports diverse optimizer discovery through (self-)invocation mechanisms within LLMs. Extensive experiments—including TSP, online/offline Bin Packing, and CVRP—demonstrate that MoH outperforms traditional, neural, and recent LLM-based heuristic generation methods in both performance and scalability.", "questions": "- During inference, you run MoH for 10 iterations and select the best-performing heuristic based on the final results. Is this strategy also applied to other baselines?\n- In Table 8, you distinguish between MoH-Train and MoH-Inference, but Table 9 only mentions MoH. Could you clarify this difference?\n- For Tables 8 and 9, testing on large instances increases MoH inference time but not the other methods, correct? The same concern applies to token consumption. I suggest clarifying the entire procedure.", "limitations": "", "rating": 4, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- The shift from fixed heuristic-optimizers to directly searching for optimizers automates the design of entire optimization frameworks, not just heuristics. This represents a conceptual advance over state-of-the-art LLM-based automatic algorithm design.\n- MoH is benchmarked across multiple tasks and demonstrates state-of-the-art or competitive optimality gaps, especially when generalizing to larger instance sizes.\n- Cost analysis—covering LLM requests, tokens, and wall time—helps contextualize practical applicability.", "weaknesses": "- Despite noteworthy generalization to larger problem instances and other COPs, the empirical evaluation focuses on classical, well-studied benchmarks (TSP, BPP, CVRP). The approach hasn't been applied to real-world, domain-constrained, or industrial COPs, which may limit its immediate practical impact. It would be interesting to see how it performs on real-world optimization problems that LLMs are unfamiliar with. Author comments on this aspect would be valuable.\n- The distinction between meta-optimizer and heuristic-optimizer is confusing, especially since they turn out to be the same in the end.\n- The comparaison to other baselines seems not fair, see questions for details. I would like to hear rebuttal from the authors before making the final decision.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)