Refining Hybrid Genetic Search for CVRP via Reinforcement Learning-Finetuned LLM
Authors: Rongjie Zhu
Cong Zhang
Zhiguang Cao
Venue: ICLR 2026 Poster (2026)
Keywords: Capacitated Vehicle Routing
Large Language Model
Reinforcement Finetuning
Primary area: optimization

Abstract: While large language models (LLMs) are increasingly used as automated heuristic designers for vehicle routing problems (VRPs), current state-of-the-art methods predominantly rely on prompting massive, general-purpose models like GPT-4. This work challenges that paradigm by demonstrating that a smaller, specialized LLM, when meticulously fine-tuned, can generate components that surpass expert-crafted heuristics within advanced solvers. We propose RFTHGS, a novel Reinforcement learning (RL) framework for Fine-Tuning a compact LLM to generate high-performance crossover operators for the Hybrid Genetic Search (HGS) solver, applied to the Capacitated VRP (CVRP). Our method employs a multi-tiered, curriculum-based reward function that progressively guides the LLM to master generating first compilable, then executable, and finally, superior-performing operators that exceed human expert designs. This is coupled with an operator caching mechanism that discourages plagiarism and promotes diversity during training. Comprehensive experiments show that our fine-tuned LLM produces crossover operators which significantly outperform the expert-designed ones in HGS. The performance advantage remains consistent, generalizing from small-scale instances to large-scale problems with up to 1000 nodes. Furthermore, RFTHGS exceeds the performance of leading neuro-combinatorial baselines, prompt-based methods, and commercial LLMs such as GPT-4o and GPT-4o-mini.

Published as a conference paper at ICLR 2026
REFINING HYBRID GENETIC SEARCH FOR CVRP VIA REINFORCEMENT LEARNING-FINETUNDD LLM
Rongjie Zhu Cong Zh Zhiguang Cao
SShooo of Teacher Education, Nanjing University of Information Science and Technology. China "College of Computing and Data Science, Nanyang Technological University, Singapore School of Computing and Information Systems, Singapore Management University, Singapore smu. rzhul 1451141114 4514 4 @mail. 1. com, cong. z2a gmail com, zgcacl smu. edu. sg
ABSTRACT
While large language models (LLMs) are emerging as automated heuristic de- signers for solving vehiele routing problems (VRPs), state-of-the-an approaches predominantly rely on massive, challenges this paradigm by demonstrating that smaller, specialized LLMs, when finely tuned, can generate components that surpass expert-designed heuristics within advanced solvers. We introduce RFTHGS, a novel Reinforcement learn- ing (RL) framework for Fine-Tuning a small LLM to produce high-performance crossover (and subpopulation) operator for the Hybrid Genetic Search (HGS) solver to solve the capacitated vehicle routing problem (CVRP). Our method uti- lizes a multi-tiered, curriculum- bassd reward function that progressively guides the LLM to first produce compilable code, then executable operators, and finally, components that exceed human expert-designed ones. Additionally, we introduce an operator caching mechanism work in conjunction with the reward function, discouraging plagiarism and promoting diversity during training. Experimental results demonstrate that our fine-tuned LLM generates crossover (and subpopula- tion) operator which significantly outperforms those designed by human experts in HGS. This performance advantage is consistent, holding from umall-scale in- stances and generalizing to large-scale problems of up to 1000 nodes. Further- more, RFTHGS surpasses leading neuro-combinatorial baselines, prompt- methods, and commercial LLMs, including GPT-4o and GPT-o4-mini.
purpose models like GPT-4. This work
INTRODUCTION
Combinatorial Optimization Problems (COPs) represent a fundamental class of computational chal- lenges that arise across diverse domains including supply chain management, logistics, scheduling, and network design (Bengio et al., 2021). These problems, characterized by their discrete decision variables and complex constraints, are often NP-hard in complexity, making exact solutions compu- tationally intractable for large-scale instances. For decades, researchers have developed specialized algorithms, heuristics, and metaheuristics to approximate optimal solutions, yet these approaches typically require significant domain expertise and manual design efforts (Papadimitriou & Steiglitz, 1998). The emergence of large language models (LLMss), with their remarkable reasoning and pattern recognition capabilities, has introduced transformative paradigm for tackling COPs. By leveraging their natural language processing and generative abilities, LLMs enable automated ap- proaches that reduce reliance on manual algorithm design and expert intervention (Sun et al., 2024; Novikov et al., 2025). The investigation of LLMM-baed approaches for solving Vehicle Routing Problems ( (VRPs) represents one of the most cutting-edy frontiers in this field.
Initial studies explored the use of LLMs as end-to-end solvers for VRPs (Yang et al., 2024). How- ever, these purely generative approaches often yield solutions that are substantially inferior to those from conventional or deep learning- ing attributed to the fact that LLMs are notoriously prone to hallucinations (Kalai et al., 2025). Consequently, a more promising direction is to integrate LLMs not as autonomous solvers but as
solvers and are frequently infeasible, i.e., shortcom-
Both authors contributed equally

---

Published as a conference paper at ICLR 2026
intelligent operators within established optimization frameworks, such evolutionary algorithms. In this hybrid paradigm, the LLM acts as a strategic generator or refiner within an iterative loop. For instance, Liu et al. (2024c) employ purposse LLMs to guide an evolutionary process, using in-context prompting to perform crossover and mutation. An alternative approach inverts this relationship, using evolutionary computation not a framework for the LLM to power, but mechanism to guide the LLM itself in generating increasingly effective heuristics. Representative works like EoH (Liu et al., 2024b) and ReEvo (Ye et al., 2024) iteratively refine LLM-ggnerated heuristic through evolutionary selection. In contrast, another line of research prioritizes general- purpose frameworks for diverse VRP variants, favoring broader generalization at the cost of perfor- mance. Methods such as ARS (Li et al., 2025a), which leverage predefined structures to generate constraint-checking functions, and DRoC (Jiang et al., 2025), which utilizes retrieval to produce code that invokes external solvers like OR-Tools (Furnon & Perron, 2025), demonstrate improved generalization and robustness against code execution failures. Nonetheless, a significant perfor- mance gap remains with conventional and deep learning- based solvers, echoing doubts about the immediate application of existing methods to large-scale problems. Given that real-world instances rely on advanced solvers, critical question is whether we can finetune small LLMs to optimize key components within these solvers to achieve beyond expert performance, presenting a challenging yet promising research frontier.
We introduce RFTHGS, a reinforcement learning (RL) framework for fine-tuning a reasoning LLM with 14B parameters to autonomously generate effective crossover (and subpopulation) operator for the Hybrid Genetic Search (HGS) algorithm (Vidal, 2022), thereby enhancing its performance in solving large-scale Capacitated Vehicle Routing Problems (CVRP). Our RL approach utilizes solu- tion quality as the key feedback signal, instantiated as a structured, tiered reward design. This reward designed to guide the learning process through three progressive stages. First, since instruction- tuned models often fail to produce syntactically valid code, we reward compilable outputs. Second, an additional reward granted if the operator code executes successfully without runtime errors or timeouts. Finally, for the compilable and executable code, the relative improvement in solution quality on a predefined set of CVRP instances compared to a baseline expert-designed operator, translates into linear reward (positive for outperforming the baseline, negative otherwise). It is important to emphasize that the CVRP instances serve exclusively to steer the feedback mecha- nism by evaluating operators generated by the LLM and are not provided as input to the model itself. To prevent reward hacking and repeated generation of the same nigh-performing operator, we incorporate operator buffer mechanism. This mechanism penalizes the model for producing du- plicates, thereby explicitly incentivizing diversity in the discovered solutions. Through this iterative refinement process. our framework empowers the LLMM-generan the performance of handcrafted operators designed by human experts. Extensive experiments verify that the L.LM-generated crossover (and subpopulation) operator delivers substantial improvements over the expert-designed operator in HGS, achieving superior performance on both small and large- scale instances (up to 1,000 nodes) on real-world benchmark. Furthermore, it surpasses all leading neuro-combinatorial and prompt-based LLM baselines by significant margin. This work provides, to the best of our knowledge, the first evidence that a small-scale reasoning LLM (14B parameters) can be fine- tuned via RL to produce critical components that exceed the performance of those in state-of-the-art, expert-designed solvers.
operators to ultimately exceed
2 RELATED WORK
2.1 ON THE REASONING ABILITY OF LARGE LANGUAGE MODELS
The development of large language models (LLMs) with advanced reasoning capabilities has evolved through several key phases, beginning with Chain of Thought (CoT) prompting, which explicitly guides models to generate intermediate reasoning steps, significantly improving perfor- mance on tasks like arithmetic and commonsense reasoning (Plaat et al., 2024; Wei et al., 2022). This approach was further enhanced by inference-time strategies such as Self-Consistency (aggre- gating multiple reasoning paths) and Tree-of-Thoughe jectories), which reduce errors and improve robustness in multi-step problem-solvine (Yao et al., 2023). A major shift occurred with the integration of reinforcement learning (RL), where models are trained using verifiable rewards (e.g.. correct answers in mathematical problems or code execu- tion results) to incentivize logical reasoning without relying solely on supervised fine-tuning (Xiang
(ToT) (exploring branched reasoning tra-
2

---

Published as a conference paper at ICLR 2026
et al., 2025; Xu et al., 2025). For instance, DeepSeek-RI (DeepSeek-Al et al., 2025) and OpenAI's ol series (OpenAI et al., 2024) exemplify how RL-driven self-improvement and scaled inference- time compute enable deliberate, step-by-step reasoning. Additionally. hybrid methods such as Mi- crosoft's 's rStar-Math (Guan et al., 2025) (which integrates Monte Carlo Tree Search for problem decomposition) and retrieval-augmented generation (RAG) (Lewis et al., 2020) combine the pattern recognition capabilities of LLMs with external tools for rigorous symbolic operations. Recent ad- vancements also focus on test-time training and outcome-based exploration enhance adaptability and diversity in reasoning paths, while interpretability research aims to ensure faithful internal rea- soning processes (Song et al., 2025). Despite progress, challenges such as hallucination, scalability, and generalisation persist, driving ongoing innovation in architectures and training paradigms (Sho- jaee et al., 2025). In contrast to the well-recognized success in solving math problems, training the L.LM with reasoning capabilities to generate operators that can outperform the default expert- designed ones in advanced VRP solvers remains challenging and largely unexplored.
2.2 SOLVING CVRP WITH LLM IN THE LOOP
The integration of LLMs into Vehicle Routing Problems (VRPs) has primarily advanced through prompting-basee methodologies (Yang et al., 2024; Jiang et al., 2025; Liu et al., 2024c; Huang et al., 2024), which leverage the robust reasoning capabilities of state-of-the-arr (SOTA) off-the- shelf LLMs (e.g., GPT-o3-mini). For instance, ARS (Li et al., 2025b) uses LLMS to automatically generate constraint-awarc heuristics for solving complex vehicle routing problems by synthesizing natural language descriptions into executable code, which can construct heuristics for 90% of com- mon VRP variants with different constraints. Similarly, Hercules (Wu al., 2025) employs Core Abstraction Prompting (CAP) to derive high-performance heuristics by abstracting core components from clite solutions, though remains dependent on powerful closed-source LLMs. Recent work has explored automatic heuristic design using LLMs. Representative works like EoH (Liu et al., 2024b) and ReEvo (Ye et al., 2024) iteratively refine L.M generreee heuristics through evolution- ary selection. Most recently, CALM (Huang et al., 2025) extends this paradigm by integrating rein- forcement fine-tuning of the LLM into the evolutionary loop, allowing the model and its generated heuristics to co-evolve. In contrast, finetnning-basee approaches for COP remain relatively sparse, often due to challenges like catastrophic forgetting, computational costs, and overfitting when adapt- ing pre-trained models to specialized domains. While eeffumttr-eeffcient methods like LoRA (Hu et al., 2021) mitigate some issues, fine-tuning small open-source L.LMS (e.g.. LLaMA (Touvron et al., 2023)) generate operators for widely-used solvers like Hybrid Genetic Search (HGS) (Vi- dal, 2022) remains an open problem. Current efforts focus primarily on prompting, leaving a gap in developing specialized, lightweight models that can efficiently integrate with solver frameworks without relying on API-dependent, proprictary LLMs.
3 PRELIMINARY
HGS For Solving The Capacitated Vehicle Routing Problem (CVRP). The Hybrid Genetic Search (HGS) (Vidal, 2022) algorithm represents a state-of-the-art metaheuristic framework promi- nently applied to complex combinatorial optimization problems, particularly vehicle routing prob- lems (VRPs). As an extension of the classical genetic algorithm, HGS distinguishes itself through tight integration of population-based evolutionary search and intensive local improvement proce- dures. Its core mechanism involves maintaining a diverse population of solutions that are iteratively refined through process of selection, crossover, and local search. Within this framework. the crossover operator is the primary mechanism for global exploration by recombining genetic mate- rial from parent solutions to generate novel offspring. Rather than producing trivial combinations, it constructs high-quality solution skeletons that effectively inherit desirable attributes from both parents. These offspring solutions subsequently undergo rigorous local search, which acts upon the foundation laid by crossover to exploit the solution space locally and achieve feasibility and opti- mality. This synergistic interplay, where crossover provides a robust starting point for deep local exploitation, is a critical factor in the documented efficacy of HGS, enabling it to navigate the trade- off between exploration and exploitation effectively and consistently produce high-quality solutions for routing problems.
3

---

Published as a conference paper at ICLR 2026
Plesac optánice speraot..
Malisfaed Rowani Panctin
Gareniid Openkin
PyVRR
Objective Value
Reward, 2%
LUR
Generated
PyVRR
Abor Instsnck
Objective Valuey
Reward 2
PPVRP
408
(Box [nstemoe)
Objeztivn Valte
Reward
Guncrated Openior F
Updet with DAPO Loos
The reinforcement learning pipeline of RFTHGS. The framework iteratively optimizes an LLM to generate effective crossover operators for HGS. Each iteration consists of generating code from structured prompt, evaluating the operator's performance on a validation set (using in- cremental compilation for speed), calculating multi-faceted reward, and updating the L.LM policy. The LLM only sees operator examples, not problem instances or the solver codebase.
Figure I:
The Group Relative Policy Optimization (GRPO) Algorithm. GRPO (Shao et al., 2024) is an improved variant of Proximal Policy Optimization (PPO) (Schulman et al., 2017). The key innova- tion of GRPO lies in utilizing a normalized reward function to compute advantages, where the mean and variance are estimated through Monte-Carky sampling (with sample size G) from the current policy TT (-[12; at step & for each input (prompt) æ. For given parameters 3 > 0, and a reference policy Tref (usually the base model), the GRPO objectíve optimization problem formulated as:
ग (t)lz) The (y|z) Axe (z, v), clip
<tyl
max E (- min
(+(()²) 1 2,1+E) Amg Ana( (2.
-BLL (Te)
(1)
where KL denotes Kullback-Leible divergence, and An, represents GRPO advantage function:
r(a, vi) - r(x, r(x, M1.)
r(z,y:) - m((re))
Ar, (x) = v BEgg (r( Wi) ET r(r, 11
1ses
(2)
(o²((re))+ +E
with the advantage estimated by sampling a "group" of size for each input x, and μ and a represent the empirical mean and standard deviation, respectively.
4 METHOD
We introduce RFTHGS, reinforcement learning framework that fine-tunes large language models (LLMs) to generate crossover operators that outperform expert-designed ones in the Hybrid Genetic Search (HGS) solver (Vidal, 2022). The framework uses solution quality the key reward signal to guide the L.M toward generating increasingly effective operators. As shown Figure 1, RFTHGS an iterative closed feedback loop. Each iteration consists of prompting the LLM to generate new operators, assessing their performance on a set of predefined CVRP instances, and then employing the feedback reward refine the LLM via reinforcement learning.
Specifically, each iteration begins by constructing few-shot CoT context (see Appendix A.3) that contains: (1) instructions specifying key properties (c.g.. diversity and quality) and steps for gen- erating a high-quality operator, and (2) examples of existing operators that illustrate the required structure and syntax. The L.MM then generates new crossover operators based exclusively on this prompt, with no references to other operators (or modules) in the HGS library or access to specific CVRP instances. For evaluation, each generated operator integrated into the HGS library, and the code is recompiled to test a fixed problem benchmark set. We employ incremental compi- lation to speed up this recompilation step. Finally, performance metrics such as compilability and improvement over baseline operators combined into multi-faceted reward, which is used to finetune the LLM via reinforcement learning. The RFTHGS framework automates the design of optimization operators, and demonstrates that small LLMS can evolve components that outperform
4

---

Published as a conference paper at ICLR 2026
Obj
Multi-facet Reward Function
P
-0.7
Incremental
PyVRP
Compilation
LLM
-0.8
-0.9
Generated Operator
Figure 2: (a) HGS as the environment for evaluating the quality of LLM-generated operators. We use the incremental compilation technique to boost the computation objective values. (b) The multi-faceted reward function.
human-designed ones in a statc-of-the-ar CVRP solver. We also verify that the RFTHGS framework generic and can be applied to optimize other operators (or modules) within HGS, as demonstrated in Appendix A.5, where we also apply RFTHGS to optimize the subpopulation operator as another example.
4.1 ONE-STEP POMDP MODELLING
We formulate the operator optimization as a one-step Partially Observable Markov Decision Process (One-Step POMDP), with formal definitions of state, action, reward and policy given as follows.
State: The state X E X denotes a tokenized prompt received by the LLM, comprising both the task instructions and examples of the target operator to be optimized. To mitigate context overload, the imput is restricted to the target operator itself (e.g., the crossover operator in HGS), rather than the entire library of the solver. This restriction results in a partially observable environment, as the model only perceives subset of the full state space (i.e., the complete solver repository).
To enhance the versatility of the learning process, we maintain buffer of few-shot examples con- taining operators generated by the LLM during training well as those designed by human experts. At each iteration, we randomly sample examples from this buffer to construct the prompt. This ap- proach enables the LLM to learn from and attempt to improve upon both its previous generations and expert-designed operators. Furthermore, enriches the diversity of the initial states (i.c., prompts), which helps prevent overfitting and encourages broader exploration.
Action: The action (or response) Y ~ p(-|X E X) generated by our operator-refining LLM is a sequence of tokens Y E )' consisting of two main parts, where p(- X) is the conditional probability from which the actions are sampled. The first part is a reasoning segment enclosed between the special tokens <think> and </think>, where it explains the steps and logic it plans to take for the optimization task. Following this, it outputs optimized version of the code for the target operator.
State Transition:
The state terminates after the action is generated, since only allow round of optimization for the operator. Therefore, there are no state transitions, and the POMDP is one-step.
Reward: The reward E IR is scalar evaluating the quality of LLM-generated operators. Please refer to Section 4.2 below for details of the reward function.
The policy network is a base large language model, denoted by To(+ X E X, with trainable parameters that parameterize the conditional probability distribution p(-|X) X) from which the optimized operator is sampled. In this work, we focus on relatively small LLMs (c.g.. 14B parameters), which can be either a pretrained base model or an instruction- variant.
Policy Network:
Herr use the symbol X X' to represent the state to highlight that the RL task here is different from the conventional RI. ones, where the initial state is the input the I.I.M.
5

---

Published as a conference paper at ICLR 2026
Table 1: Performance comparison of baselines and our method for CVRPLIB across problem sizes. Light gray columns indicate generalization to unseen problem sizes, while light gray rows represent generalization to higher iterations. The darker gray intersection areas highlight double generalization across both dimensions. Bold values denote best performance among all methods; asterisks (*) indicate that the results are unavailable.
[100, 200)
E [200, 400)
E (400, 600)
[080, 800)
E [800, 1000)
Methods
Gapti (J) Time (4)
Capss (3) Time (6)
Gapti (}) Time
Geprit (}) Time (6)
Gaps% (3) Time (6)
Conventional Solver
HGS-PYVRP (Wouda al., 2024)
0.62
12.45
1.85
28.16
1.95
58.41
2.62
91.31
2.32
121.04
VYVR 1ono (Wouda al.
14.88
110.54
HGSS-F VRP Izno (Wouda el al., 2024)
0.52
17.25
1.56
42,98
1.69
88.40
2.32
129.58
2.10
173.15
OR-Tools (Purnon & Perros, 2025)
4.26
88.83
5.05
172.07
4.98
296.53
6.71
416.95
4.65
532.32
LKH (Helsgaun, 2000)
POMO (Kwon
13.30
0.41
14.64
22.07
21.57
2.32
MTРOМO (Liu al., 2024g)
6,50
0.98
8.79
1.02
16.58
1.89
26.56
2.81
28.19
4.33
MVMrF (7hou 2024)
5,46
0.80
16.59
4.17
RF-OOO (Berto
2025)
5.67
0.54
12.28
2.86
15.31
RF MoE: L (Berto al., 2025)
7.15
0.85
7.67
1.58
10.76
2.79
15.15
3.95
15,70
5.84
AM (Kool at., 2019)
200.75
0.30
253.08
301.0K
1.51
280 49
2.01
DespACO
76.18
18.37
61
123.89
KeuroLKII (Xin et 2021)
1.96
L,02
NeuOpt (Ma all. 2023)
3.51
Prompting Bused Method With LLM
MCTS-AND (hhng al., 2025)
4.35
22.50
ReEvo (Yo ol. 2024)
22.11
4.96
96.55
10.78
107.40
23.99
163.62
41.72
144.22
61.49
GPTiegoo (Huns al., 2024)
0.62
12.1
1.85
29.3
1.95
59.4
2.62
91.7
2.32
119.2
GPT et
15.0
(IPT405000 (Hunst et al, 2024)
0.52
17.7
1.56
43.1
1:69
87.3
2.32
130.5
2,10
172.8
GPT-a3300 (Jarch et al., 2024)
0.62
11.96
1.85
30.20
1.98
58.94
2.62
91.39
2.32
119.94
(Jacch GPT-03000
2024)
GPT-03 aco (Tecch 2024)
0.52
17.65
1.56
43.53
1:69
K7
2.32
130.71
2,10
172.93
GPL-o4-minlano (laech et al., 2024)
0.70
10.9
1.78
26.1
1.99
54.4
2,80
86.7
2.28
114.8
mini.ono
2024)
13.7
106.3
GPT-ol-mini s0o (Racch 2024)
0.58
16.3
1.58
38.6
1.74
79.4
2.61
137.1
2.11
169.6
Ours
KFTHGSS000
0.70
13.14
1.67
29.60
1.83
61 61.65
2.59
92.17
2.24
118.59
RFTHCS
0.52
14.33
1.62
36.16
1.76
74.16
2.35
110.43
2.17
143.87
XFTHGS,.90
19.13
4.2 THE MULTTIFACETTD REWARD DESIGN WITH ANTI- CACHR
Building on the insight that carefully crafted, multi-faceted rewards are crucial for effective RL (Narvekar et al., 2020; Eppe et al., 2022; Huang et al., 2025), we developed multi-tiered reward function to decompose the learning process. Particularly. the reward function follows a curriculum learning principle, guiding the L..M through progressive stages to evolve operators that exceed those designed by human experts. To ensure the robustness of this approach, we further introduce two key innovations: mechanism to prevent reward hacking by deterring plagiarism of prompt examples, and method to significantly accelerate the training process.
To mitigate reward hacking and encourage the exploration of unseen operators, we introduce a caching mechanism that leverages Abstract Syntax Trees (ASTs) deter plagiarism. The AST provides structured, hierarchical represen- tation that abstracts away unnecessary syntactic details like punctuation and formatting to capture the essential logical structure of the generated operators. We cache the AST representations of all few-shot operator examples in the prompt. For each operator generated by the policy лT. its AST compared against those in the cache. A penalty invoked by the reward function a substantial match is detected, indicating direct copying. This approach promotes diverse exploration by pe- nalizing redundant operator generation. Appendix A.7 presents the details of the AST comparison mechanism.
Anti-Plagiarism Cache With Abstract Syntax Tree.
HGS With Incremental Compilation As The Evaluator. We have to integrate each generated op- erator into the HGS library evaluate its quality. This process inevitably requires recompiling the

---

Published as a conference paper at ICLR 2026
repository, which will incur prohibitive computational overhead, especially for large training batch sizes. Nonetheless, recompiling the entire library is unwarranted when only a single, small code snippet (the generated operator) is modified. To address this bottleneck, we employ an incremental compilation technique that selectively recompiles only the modified code and its dependencies, re- ducing recompilation time to approximately 25% of compiling the whole library and significantly accelerating the training speed.
Here we give the formulation of our three-stage reward function. First, the reward function assigns a reward of -0.8 for syntactically correct and compilable operator to encourage a rapid transition from invalid code and improve exploration efficiency, or penalty of -1 for invalid output. Upon achieving compilability, the function then assesses executability, penalizing runtime failures such as timeouts. An operator that executes successfully receives a reward of -0.7, independent of its solution quality. Finally, for executable operators, performance is evaluated on a predefined set of CVRP instances, with the reward quantified as the relative improvement over expert-designed benchmarks according the following calculation:
o C o E C EEEEE
-0.8
r())
(3)
-099
max (-0.7, [ØHGs (Pespen) 0HaS
(o)]] / (Pespen)
o o E.of
In this formulation, o represents the generated operator, while C, and P correspond to the sets of compilable, executable, and plagiarized code, respectively. For evaluation, the HGS library is recompiled to include the generated operator o. The performance metric oHas is then calculated as the average result on J random CVRP instances. To benchmark the effectiveness of our continuous reward design in Equation 3, we compare it with a discrete version where we employ a +1 reward if the generated operator outperforms the baseline operator in Table 3 (details in Appendix 5.4). We find that our continuous reward offers feedback proportional to performance gains, enabling sustained refinement and explaining its superior performance.
4.3 THE REINFORCEMENT LEARNING ALGORITHM
We use DAPO (Yu et al., 2025) as the reinforcement learning algorithm for training our operator refining network. Specifically, DAPO an improved version of GRPO with four adjustments: 1). Mechanism. Unlike GRPO following the original PPO setting where unified clip ra- tion adopted for the positive and negative responses, DAPO decouples the clipping range into a higher upper bound (Shigh) and a standard lower bound (êtow), allowing the policy to more aggres- sively increase probabilities for promising but initially low-likelihood tokens. This promotes greater exploration and diversity in generated responses, effectively preventing entropy collapse where the model becomes overly deterministic; 2). Dynamic Sampling. This strategy filters out prompt groups where all sampled responses are either all correct all incorrect, these yield zero advantage and provide no learning signal. By replacing them with new prompts that exhibit varied performance, DAPO ensures every training batch contains meaningful gradients, improving training efficiency and stability without sacrificing throughput. However, in our paper, we deprecate this design as the reward signal in our case is continuous, specifying a wide range of situations from uncompilable code to superior performance gain against the baseline operators that all contribute useful learning signals for the LLM learn; 3). Token- Policy Gradient Loss. Unlike GRPO, which averages losses at the response level, DAPO calculates and aggregates the loss over all tokens in the batch before averaging. This ensures each token's contribution to the gradient is weighted equally, pro- viding more precise updates for long reasoning chains and better reinforcing correct steps in lengthy responses; 4). Overlong Reward Shaping. To address the issue truncated lengthy responses that may contain valid reasoning, DAPO employs two strategies: Overlong Filtering excludes these re- sponses from training updates to avoid misleading penalties, and Soft Overlong Punishment applies gradual. -eepededd penalty beyond certain token threshold to encourage conciseness with- out harshly punishing correct but verbose reasoning. The pseudo-code of our algorithm is shown in Alg Algorithm 1.
7

---

Published as a conference paper at ICLR 2026
Training Curve
Exceeds Expert
1600
Avg Reward
1400
0.0
Count
-0.3
400
Training Step
200
400
Reward Curve
-0.6
Smoothed Curve
-1.0-0. 1.00-..-..660----22 0.0
Expert Desgined Baseline
800
200
400
600
800
1000
Reward Value
0.2 0.4
0.0000
Step
(a)
(b)
Figure 3: Training dynamics of the RFTHGS framework. (a) Average reward per step, showing stable convergence. (b) Evolution of the reward distribution, illustrating the effectiveness of the multi-faceted reward function in guiding the learning process.
EXPERIMENTS
EXPERIMENT SETTINGS AND BASELINES
We give the details of the configurations of our RFTHGS algorithm. Specifically, we initialize the policy with Qwen- 14B reasoning LLM (Yang et al., 2025). For DAPO, we follow its optimal settings reported in the original paper with Ehigh 0.28 and Ejow 0.2. The batch size set to 16 and the rollout group size is also set to 16. Therefore, the policy model will generate 256 crossover operators for each step. For calculating reward during training, we use fixed set of 30 CVRP instances sampled from the CVRPLIB X instances (Uchoa et al., 2017), restricting the selection to those with at most 400 nodes. During the testing phase, we sample 16 operators and report the performance of the best one. The final evaluation of our method is performed on the CVRPLIB benchmark, which encompasses a wide range of instance sizes from small scales to industry-level scales (up to 1000 nodes). We benchmark RFTHGS against variety of baselines CVRPLIB X instances (Uchoa et al., 2017). These include the statc-of-the conventional solvers, neuro-combinatoriaa techniques, and prompting strategies that utilize commercial LLMS such the GPT4 series. To ensure an equitable comparison for the LLM-based approaches, we consistently sample 16 operators and select the best one for each. Further details on the baselines are available in Table 1. Detailed breakdown of the computational resources required for RL training is provided in Appendix A.6
5.2 PERFORMANCE ON CVRPLIB
Table compares RFTHGS against a di- verse of baselines on CVRPLIB in- stances, including conventional heuristics, couo-coomiinttriial methods, and prompting techniques that utilize commercial LLMS such as the flagship GPT-40 series. The results demonstrate that RFTHGS outperforms all baseline methods by a substantial margin. This superior performance is underscored by its excep- tional generalization capability to large-scale problems unseen during training. Notably, although trained exclusively on instances with n < 400, our approach generalizes effectively to instances of up to 1000, which are validates the potential of refining advanced solvers via learned components for complex combina- torial optimization problems. The code comparison between human-expert designed operators and
Table 2: Successful compilation rate.
Successful Compilation Rate
GPT-4o
GPT-o3
GPT-o4-mini RFTHGS-14B
3/16
9/16
3/16
16/16
than twice the size of the largest training instances. This
8

---

Published as a conference paper at ICLR 2026
Table 3: Ablation study on reward design. F the 14B LLM trained with reward in Equation 4. FRTHGS, is the 14B LLM trained with reward in Equation 3. Shaded areas are gener- alization results.
n. [100, 200)
E [200, 100)
n.
[100, 600)
E (600,800)
n E [800, 1000)
Methods
Gap% (U) Time (6)
Gap% (4) Time (6)
Gapti (4) Time (4)
Gap% (L) Time (4)
Gap%i (Q)
Time (6)
IGS
0.62
12.45
1.85
28.16
1.95
58.41
2.62
91.31
2.32
121 04
FRTHGS
0.83
11.75
1.78
28.89
1.92
58.12
2.64
94,30
2.30
120.28
FRTHGS,
0.70
13.14
1.67
29.60
1.83
61.65
2.59
92.17
2.24
118.59
our LLM-optimized operators is presented in Supplementary Material, highlighting the key modi- fications introduced by the LLM that contribute to improved performance. We also compare with COMPASS (Chalumeau et al., 2023), a novel state-of-the-ar NCO method for solving VRPs. The results in Appendix A.8 shows the consistent superior performance of RFTHGS against COMPASS.
Another key observation is that our RFTHGS framework enables a 14B-parameter LLM to outper- form trillion-paramete GPT reasoning models (GPT-40, GPT-o3, GPT-o4-mini). This advantage is demonstrated through both the quality of the modifications and their practical efficacy. As shown in Table 2, our model achieves perfect successful compilation rate of 16/16, substantially exceeding the rates of the GPT models (3/16, 9/16, and 3/16, respectively). Crucially, while the GPT models often introduce numerous modifications, these changes consistently fail to improve performance. This is evident in Table 1, where the crossover operators modified by these GPT models exhibit performance identical the original, unmodified operator, confirming that no functionally help- ful modifications were made. In contrast, our RFTHGS- modifications that yield consistent performance gains. This suggests that task specifie fine-tuning can be more effective than using general-pe general-purpose purpose model of much larger scale.
model produces targeted, effective
5.3 LEARNING PATTERN ANALYSIS
Figure 3a presents the learning curve of the RFTHGS framework, demonstrating stable and mono- tonic convergence. The average reward increases smoothly without significant oscillations, indicat- ing well-structured learning landscape with the effective design of reward function. A crit- ical inflexion occurs around step 200, where the generated operator surpasses the expert-designed baseline, marking the transition from learning executable operators to discovering superior heuris- tics. Beyond this intersection point, the curve continues to show consistent improvement, ultimately achieving substantially higher performance. This smooth progression shows that RFTHGS can ef- fectively guide the L.M in generating increasingly sophisticated crossover operators. Figure 3b reveals the underlying learning patterns through the dynamies of the reward distribution. The heat map exhibits a clear curriculum learning pattern that precisely echoes our multi-tiered reward de- sign. Initially, the density concentrates at lower rewards as the model masters syntactic correctness and compilability. Subsequently, the distribution shifts toward intermediate rewards, corresponding to the phase where operators become executable and yield valid solutions. Finally, the density cen- ter progressively migrates to the highest reward region, indicating the refinement toward operators that consistently outperform human-designed ones. This three-phase progression validates the ef- fectiveness of our hierarchical reward structure in decomposing the complex operator design task into manageable learning stages.
5.4 GENERALIZATION PERFORMANCE ON ITERATIONS
The generalization capability of RFTHGS is further assessed across two joint dimensions, i.c., itera- tion count and problem size. Regarding iteration generalization, models trained with an 800-iteration budget (RFTHGSsoo) maintain robust performance when evaluated at higher budgets of 1000 and 1200 iterations, consistently outperforming expert-designed baselines. This indicates that the opti- mized operators retain their efficacy beyond their training configuration. In terms of problem size, although trained exclusively on instances with n < 400, RFTHGS generalizes effectively to signif- icantly larger problems (up to = 1000). This joint generalization underscores the robustness and strong -itt---diittibttion scalability of our method. The results are shown in the last two rows (grey areas) of Table 1.

---

Published as a conference paper at ICLR 2026
6 CONCLUSION
This paper introduces RFTHGS, reinforcement learning framework that optimizes operators in the Hybrid Genetic Search (HGS) solver for solving the Capacitated Vehicle Routing Problem (CVRP). By fine-tuning with domain-specitfe rewards, we demonstrate that specialized small LLMs can sur- pass large general and deep thinking ones like GPT-4o, HPT-o4-mini, and GPT-o3 with trillions of parameters. Our core innovation is novel RL-based fine-tuning paradigm guided by solution qual- ity, featuring a multi-tiered reward mechanism with anti-plagiarism caching for progressive learning. Extensive experiments operator generated by our method demonstrates superior performance over the expert-designed op- erator within the HGS framework, achieving substantial improvements, particularly on large-scale instances with up to 1,000 nodes. To our knowledge, this the first work to show that small, fine- tuned LLM can generate operators that exceed expert-crafted components in a leading combinatorial optimization solver. In future work, we plan to extend this framework to more other operators inside HGS and solve more types of VRPs.
CVRPLIB benchmarks confirm that the crossover (and subpopulation)
ETHICS STATEMENT
This study involves no personal data, human subjects, or other sensitive content and therefore does not raise significant ethical concerns. The only potential risk lies in the fact that operators generated by LLMs may contain bugs which, if deployed without thorough validation, could cause losses.
REPRODUCIBILITY STATEMENT
To ensure the reproducibility of our work, we have provided comprehensive experimental de- tails throughout this paper. Section 5.1 presents complete experimental configurations and en- vironment specifications, while the Appendix A.3 includes detailed prompts. These materials provide sufficient information for independent reproduction of our experimental results. Further- more, we open-source our full code base and model weights to further improve reproducibility: http://giitbbb.oommmaaddds////tt
ACKNOWLEDGMENTE AND DISCLOSURE OF FUNDING
This research supported by the National Research Foundation, Singapore under the Al Singapore Programme (AISG Award No: AISG3-RPGV--025-0177)
REFERENCES
Yoshua Bengio. Andrea Lodi, and Antoine Prouvost. Machine learning for combinatorial opti- mization: a methodological tour d'horizon. European Journal of Operational Research, 290(2): 405-421 2021.
Federico Berto, Chuanbo Hua, Nayeli Gast Zepeda, André Hottung, Niels Wouda, Leon Lan, Juny- oung Park, Kevin Tiemey, and Jinkyoo Park. Routefinder: Towards foundation models for vehicle routing problems, 2025. URL https: //tpavvvvvv//////////eaa//aaaav/ org/ abs/ 15007.
Felix Chalumeau, Shikha Surana, Clément Bonnet, Nathan Grinsztajn, Arnu Pretorius, Alexandre Laterre, and Thomas D Barrett. Combinatorial optimization with policy adaptation using latent space search. In Thirty-sevemth Conference on Neural Information Processing Systems, 2023. URL hwt.ttttt//. / wwwpr/veeeeeeeeeeeoemeeeee net forum? id=vpMBddt 9H1.
Del,Seck-AL, Daya Guo, Dejian Yang. Haowei Zhang, Junxiao Song, Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma. Peiyi Wang, Xiao Bi, Xiaokang Zhang, Xingkai Yu, Yu Wu, Z. F. Wu, Zhibin Gou, Zhihong Shao, Zhuoshu Li, Ziyi Gao, Aixin Liu, Bing Xue, Bingxuan Wang, Bochao Wu, Bei Feng, Chengda Lu, Chenggang Zhao, Chenggi Deng, Chenyu Zhang, Chong Ruan, Damai Dai, Deli Chen, Dongjie Ji, Erhang Li, Fangyun Lin, Fucong Dai, Fuli Luo, Guangbo Hao, Guanting Chen, Guowei Li, H. Zhang, Han Bao, Hanwei Xu, Haocheng Wang, Honghui Ding.
10

---

Published as a conference paper at ICLR 2026
Huajian Xin, Huazuo Gao, Hui Qu, Hui Li, Jianzhong Guo, Jiashi Li, Jiawei Wang, Jingchang Chen, Jingyang Yuan, Junjie Qiu, Junlong Li, J. L. Cai, Jiaqi Ni, Jian Liang, Jin Chen, Kai Dong, Kai Hu, Kaige Gao, Kang Guan, Kexin Huang, Kuai Yu, Lean Wang, Lecong Zhang, Liang Zhao, Litong Wang. Liyue Zhang, Lei Xu, Leyi Xia, Mingchuan Zhang, Minghua Zhang, Minghui Tang, Meng Li, Miaojun Wang, Mingming Li, Ning Tian, Panpan Huang. Peng Zhang, Qiancheng Wang, Qinyu Chen, Qiushi Du, Ruiqi Ge, Ruisong Zhang, Ruizhe Pan, Runji Wang, R. Chen, R. L. Jin, Ruyi Chen, Shanghao Lu, Shangyan Zhou, Shanhuang Chen, Shengfeng Ye, Shiyu Wang, Shuiping Yu, Shunfeng Zhou, Shuting Pan, S. S. Li, Shuang Zhou, Shaoqing Wu, Shengfeng Ye, Tao Yun. Tian Pei. Tianyu Sun, Wang, Wangding Zeng. Wanjia Zhao, Wen Liu, Wenfeng Liang, Wenjun Gao, Wenqin Yu, Wentao Zhang. W. L. Xiao, Wei An, Xiaodong Liu, Xiaohan Wang, Xiaokang Chen, Xiaotao Nie, Xin Cheng, Xin Liu, Xin Xie, Xingchao Liu, Xinyu Yang, Xinyuan Li, Xuecheng Su, Xuheng Lin, X. Q. Li, Xiangyue Jin, Xiaojin Shen, Xi- aosha Chen, Xiaowen Sun, Xiaoxiang Wang, Xinnan Song, Xinyi Zhou, Xianzu Wang, Xinxia Shan, Y. K. Li, Y. Q. Wang. Y. X. Wei, Yang Zhang, Yanhong Xu, Yao Li, Yao Zhao, Yaofeng Sun, Yaohui Wang, Yi Yu, Yichao Zhang, Yifan Shi, Yiliang Xiong, Ying He, Yishi Piao, Yisong Wang, Yixuan Tan, Yiyang Ma, Yiyuan Liu, Yongqiang Guo. Yuan Ou, Yuduan Wang, Yue Gong, Yuheng Zou, Yujia He, Yunfan Xiong, Yuxiang Luo, Yuxiang You, Yuxuan Liu, Yuyang Zhou, Y. X. Zhu, Yanhong Xu, Yanping Huang. Yaohui Li, Yi Zheng, Yuchen Zhu, Yunxian Ma, Ying Tang. Yukun Zha, Yuting Yan, Z. Z. Ren, Zehui Ren, Zhangli Sha. Zhe Fu, Zhean Xu, Zhenda Xie, Zhengyan Zhang, Zhewen Hao, Zhicheng Ma, Zhigang Yan, Zhiyu Wu, Zihui Gu, Zijia Zhu, Zijun Liu, Zilin Li. Ziwei Xie, Ziyang Song, Zizheng Pan, Zhen Huang. Zhipeng Xu, Zhongyu Zhang, and Zhen Zhang. Deepseek-rl: Incentivizing reasoning capability in llms via reinforce- ment learning. 2025. URL https:/ /tt///axxivv org/ abs///b///// 2501 12948.
Manfred Eppe, Christian Gumbsch, Matthias Kerzel, Phuong DH Nguyen, Martin V Butz, and Stefan Wermter. Intelligent problem-solving as integrated hierarchical reinforcement learning. Nature Machine Intelligence, 1(1):11-20, 2022.
Vincent Fumon and Laurent Perron. Or-tools routing library, 2025.
URL https://
developers . google. com/ wttpptimiitttttootttteoteeooooee
Xinyu Guan, Li Lyna Zhang, Yifei Liu, Ning Shang, Youran Sun, Yi Zhu, Fan Yang, and Mao Yang- rstar-math: Small lims can master math reasoning with self-evolved deep thinking. arXiv preprint arXiv:2501.0455 04519, 2025.
Keld Helsgaun. An effective implementation of the lin kernighnn traveling salesman heuristic. European journal of operational research, 126(1) ): 1030 2000.
J. Edward Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, and Weizhu Chen. Lora: Low-rank adaptation of large language models. ArXiv, abs/2106.090 09685, 2021. URL https: //api. semant wtw.mmatiicccooaa....cccccoa wttp/orppoooorooo.ooooo :2: 235458009.
Sen Huang, Kaixiang Yang. Sheng Qi, and Rui Wang. When large language model meets optimiza- tion. Swarm and Evolutionary Computation, 90:101663. 2024.
Ziyao Huang, Weiwei Wu, Kui Wu, Jianping Wang, and Wei-Bin Lee. Calm: Co-evolution of algorithms and language model for automatic heuristic design, 2025. URL https://arxivv. org/ abs/ abaaabbs555.... 2505. 12285.
Aaron Hurst, Adam Lerer, Adam P Goucher, Adam Perelman, Aditya Ramesh, Aidan Clark, AJ Os- trow, Akila Welihinda, Alan Hayes, Alec Radford, et al. Gpt-4o system card. arXiv preprint arXiv:2410.21276, 2024.
Aaron Jaech, Adam Kalai, Adam Lerer, Adam Richardson, Ahmed El-Kishky, Aiden Low, Alec Helyar, Aleksander Madry, Alex Beutel, Alex Carney, et al. Openai ol system card. arXiv preprint arXiv:2412.16720 2024.
Xia Jiang, Yaoxin Wu, Chenhao Zhang, and Yingqian Zhang. Droc: Flevating large language models for complex vehicle routing via decomposed retrieval of constraints. In 13th international Conference on Learning Representations, ICLR 2025. OpenReview. net, 2025.
Adam Tauman Kalai, Ofir Nachum, Santosh S. Vempala, and Edwin Zhang. Why language models hallucinate, 2025. URL https: http://wwww///////////////// 2509. 04664.
11

---

Published as a conference paper at ICLR 2026
Wouter Kool, Herke van Hoof, and Max Welling. Attention, learn to solve routing problems! In International Conference on Learning Representations, 2019. URL httpp:://pppeeeeeeee
bet//oorumidd-yyee SRaYm.
Yeong-Dae Kwon, Jinho Choo, Byoungji Kim, Iijoo Yoon, Youngjune Gwon, and Seungjai Min. Pomo: Policy optimization with multiple optima for reinforcement learning. Advances Neural Information Processing Systems, 33:21188 188821198,, 2020.
Patrick Lewis, Ethan Perez, Aleksandra Piktus, Fabio Petroni, Vladimir Karpukhin, Naman Goyal. Heinrich Küttler, Mike Lewis, Wen-tau Yih, Tim Rocktäschel, Sebastian Riedel, and Douwe Kiela. Retrieval-augmenter generation for knowledge-intensive nlp tasks. In H. Larochelle, M. Ranzato, R. Hadsell, M.F. Balcan, and H. Lin (eds.), Advances in Neu- ral Information Processing Systems, volume 33, pp. 4459-94744 Curran Associates, Inc., neurips cc httpl/apaalleee/eellelaaaaaaseea 2a11:/66992222200777000222222223
2020. URL
www.eeeeee---p..e.eeee pdf.
Kai Li, Fei Liu, Zhenkun Wang, Xialiang Tong, Xiongwei Han, Mingxuan Yuan, and Qingfu Zhang. ARS: Automatic routing solver with large language models. arXiv preprint arXiv:2502. 15359, 2025a.
Kai Li, Fei Liu, Zhenkun Wang, Xialiang Tong, Xiongwei Han, Mingxuan Yuan, and Qingfu Zhang. Ars: Automatic routing solver with large language models, 2025b. URL https: penttp::/araaa /arxiv. org/ htgg/abs//0 0/////222.115555
Fei Liu, Xi Lin, Zhenkun Wang, Qingfu Zhang, Tong Xialiang, and Mingxuan Yuan. Multi-task learning for routing problem with cross-problem zero-shot generalization. In Proceedings of the 30th ACM SIGKDD Conference Knowledge Discovery and Data Mining, pp. 1898-1908, 2024a.
Fei Liu, Xialiang Tong, Mingxuan Yuan, Xi Lin, Fu Luo, Zhenkun Wang, Zhichao Lu, and Qingfu Zhang. Evolution of heuristics: towards efficient automatic algorithm design using large language model. In Proceedings of the 4/st International Conference on Machine Learning, JCML'24. JMLR.org, 2024b.
Shengcai Liu, Caishun Chen, Xinghua Qu, Ke Tang, and Yew-Soon Ong. Large language models as evolutionary optimizers. In 2024 IEEE Congress Evolutionary Computation (CEC), pp. 1-8, 210.C//00990.222221111111111
2024c. doi:
Yining Ma. Zhiguang Cao, and Yeow Meng Chee. Learning to search feasible and infeasible gions of routing problems with flexible neural k-opt. Advances in Neural Information Processing Systems, 36:49555-444 49578, 2023.
Sanmit Narvekar, Bei Peng, Matteo Leonetti, Jivko Sinapov, Matthew E Taylor, and Peter Stone. Curriculum learning for reinforcement learning domains: A framework and survey. Journal of Machine Learning Research, 21 21(181):1-50, 2020.
Alexander Novikov, Ngån Vū, Marvin Eisenberger, Emilien Dupont, Po-Sen Huang, Adam Zsolt Wagner, Sergey Shirobokov, Borislav Kozlovskii, Francisco JR Ruiz, Abbas Mehrabian, et al. Alphaevolve: A coding agent for scientific and algorithmic discovery. arXiv preprint arXiv:2506.13131, 2025.
OpenAl, Aaron Jacch, Adam Kalai, Adam Lerer, Adam Richardson, Ahmed El- Ei-Kishky, Aiden Low, Alec Helyar, Aleksander Madry, Alex Beutel, Alex Carney, Alex Iftimie, Alex Karpenko, Alex Tachard Passos, Alexander Neitz, Alexander Prokofiev, Alexander Wei, Allison Tam, Ally Bennett, Ananya Kumar, Andre Saraiva, Andrea Vallone, Andrew Duberstein, Andrew Kondrich, Andrey Mishchenko, Andy Applebaum, Angela Jiang, Ashvin Nair, Barret Zoph, Behrooz Ghor- bani, Ben Rossen, Benjamin Sokolowsky, Boaz Barak, Bob McGrew, Borys Minaiev, Botao Hao, Bowen Baker, Brandon Houghton, Brandon McKinzie, Brydon Eastman, Camillo Lugaresi, Cary Bassin, Cary Hudson, Chak Ming Li, Charles de Bourcy, Chelsea Voss, Chen Shen, Chong Zhang, Chris Koch, Chris Orsinger, Christopher Hesse, Claudia Fischer, Clive Chan, Dan Roberts, Daniel Kappler, Daniel Levy, Daniel Selsam, David Dohan, David Farhi, David Mely, David Robinson,
12

---

Published as a conference paper at ICLR 2026
Dimitris Tsipras, Doug Li, Dragos Oprica, Eben Freeman, Eddie Zhang, Edmunn Wong. Eliz- abeth Prochl, Enoch Cheung, Eric Mitchell, Eric Wallace, Erik Ritter. Evan Mays, Fan Wang, Felipe Petroski Such, Filippo Raso, Florencia Leoni, Foivos Tsimpourlas, Francis Song, Fred von Lohmann, Freddie Sulit, Geoff Salmon, Giambattista Parascandolo, Gildas Chabot, Grace Zhao, Greg Brockman, Guillaume Leclere, Hadi Salman, Haiming Bao, Hao Sheng. Hart An- drin, Hessam Bagherinezhad, Hongyu Ren, Hunter Lightman, Hyung Won Chung, Ian Kivlichan, Ian O'Connell, Ian Osband, Ignasi Clavera Gilaberte, Ilge Akkaya, Ilya Kostrikov, llya Sutskever, Irina Kofman, Jakub Pachocki, James Lemnon, Jason Wei, Jean Harb, Jerry Twore, Jiacheng Feng, Jiahui Yu, Jiayi Weng, Jie Tang, Jieqi Yu, Joaquin Quiñoonero Candela, Joe Palermo, Joel Parish, Johannes Heidecke, John Hallman, John Rizzo, Jonathan Gordon, Jonathan Uesato, Jonathan Ward, Joost Huizinga, Julie Wang. Kai Chen, Kai Xiao, Karan Singhal, Karina Nguyen, Karl Cobbe, Katy Shi, Kayla Wood, Kendra Rimbach, Keren Gu-Lemberg, Kevin Liu, Kevin Lu, Kevin Stone, Kevin Yu, Lama Ahmad, Lauren Yang, Leo Liu, Leon Maksin, Leyton Ho, Liam Fedus, Lilian Weng, Linden Li, Lindsay McCallum, Lindsey Held, Lorenz Kuhn, Lukas Kon- draciuk, Lukasz Kaiser, Luke Metz, Madelaine Boyd, Maja Trebacz, Manas Joglekar, Mark Chen, Marko Tintor, Mason Meyer, Matt Jones, Matt Kaufer, Max Schwarzer, Meghan Shah, Mehmet Yatbaz, Melody Y. Guan, Mengyuan Xu, Mengyuan Yan. Mia Glaese Mianna Chen, Michael Lampe, Michael Malek, Michele Wang, Michelle Fradin, Mike McClay, Mikhail Pavlov, Miles Wang, Mingxuan Wang. Mira Murati, Mo Bavarian, Mostafa Rohaninejad, Nat McAleese, Neil Chowdhury, Neil Chowdhury, Nick Ryder, Nikolas Tezak, Noam Brown, Ofir Nachum, Oleg Boiko. Oleg Murk, Olivia Watkins. Patrick Chao, Paul Ashbourne, Pavel Izmailov, Peter Zhokhov, Rachel Dias, Rahul Arora, Randall Lin, Rapha Gontijo Lopes, Raz Gaon, Reah Miyara, Reimar Leike, Renny Hwang. Rhythm Garg, Robin Brown, Roshan James, Rui Shu, Ryan Cheu, Ryan Greene, Saachi Jain, Sam Altman, Sam Toizer, Sam Toyer, Samuel Miserendino. Sandhini Agar- wal, Santiago Hernandez, Sasha Baker, Scott McKinney, Scottie Yan, Shengjia Zhao, Shengli Hu, Shibani Santurkar, Shraman Ray Chaudhuri, Shuyuan Zhang, Siyuan Fu, Spencer Papay, Steph Lin, Suchir Balaji. Suvansh Sanjeev, Szymon Sidor, Tal Broda, Aidan Clark, Tao Wang. Tay- lor Gordon, Ted Sanders. Tejal Patwardhan, Thibault Sottiaux, Thomas Degry, Thomas Dimson, Tianhao Zheng, Timur Garipov, Tom Stasi, Trapit Bansal, Trevor Creech, Troy Peterson, Tyna Floundou, Valerie Qi, Vineet Kosaraju, Vinnie Monaco, Vitchyr Pong, Vlad Fomenko, Weiyi Zheng, Wenda Zhou, Wes McCabe, Wojeiech Zaremba, Yann Dubois, Yinghai Lu, Yining Chen, Young Cha, Yu Bai, Yuchen He, Yuchen Zhang, Yunyun Wang, Zheng Shao, and Zhuohan Li. Openai ol system card, 2024. URL https: / arxiv. org/ abs 2412.16720.
Christos H Papadimitriou and Kenneth Steiglitz. Combinatorial optimization: algorithms and com- plexity. Courier Corporation, 1998.
Aske Plaat, Annie Wong, Suzan Verberne, Joost Broekens, Niki van Stein, and Thomas Bäck. Rea- soning with large language models, a survey. CoRR, 2024.
John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. arXiv preprint arXiv:1707.063477 2017.
Zhihong Shao, Peiyi Wang, Qihao Zhu, Runxin Xu, Junxiao Song, Xiao Bi, Haowei Zhang, Mingchuan Zhang, Y. K. Li, Y. Wu, and Daya Guo. Deepseekmath: Pushing the limits of mathe- matical reasoning in open language models, 2024. URL https::////// //arxivv..i//////// org/ abs/2402 03300.
Parshin Shojace, Iman Mirzadeh, Keivan Alizadeh, Maxwell Horton, Samy Bengio, and Mehrdad Farajtabar. The illusion of thinking: Understanding the strengths and limitations of reasoning models via the lens of problem complexity, 2025. URL https: /aaxiiivvv. hargg/bbs2506. 06941.
Yuda Song, Julia Kempe, and Remi Munos. Outcome-based exploration for Ilm reasoning, 2025. URL htttht:/p///xxt org/ abs/ /20.9.259.... 06941.
Yiwen Sun, Furong Ye, Xianyin Zhang, Shiyu Huang, Bingzhen Zhang, Ke Wei, and Shaowei Cai. Autosat: Automatically optimize sat solvers via large language models. arXiv preprint arXiv:2402. 10705, 2024.
13

---

Published as a conference paper at ICLR 2026
Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, Aur elien Rodriguez, Ar- mand Joulin, Edouard Grave, and Guillaume Lample. Llama: Open and efficient foundation language models. ArXiv, abs/2302. 13971, 2023. URL httpt:ttt///tc /api htt.a/p..aaaatte semant www.emattiichhlllloooco cscholar. htt//CorpuSt::200
9404.
Eduardo Uchoa, Diego Pecin, Artur Pessoa, Marcus Poggi. Thibaut Vidal, and Anand Subramanian. New benchmark instances for the capacitated vehicle routing problem. European Journal of Op- erational Research, 257(3):845-85) 2017. ISSN 0377-2217. doi: https://do..rr//0.111166
2016.08.012 URL https: www. s c encedi rect. wwt. http://ceennc/aaticeeeeeee 2037722171630655
Thibaut Vidal. Hybrid genetic search for the evrp: Open-source implementation and swap* neigh- borhood. Computers & Operations Research, 140: 140:105643, 2022.
vndee. Ilm-sandbox: Lightweight and portable LLM sandbox runtime. https:t/tttthtttt com/ vndee Lim-sandooxx 2025.
Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Fei Xia, Ed Chi, Quoe V Le, Denny Zhou, al. Chain-of-thought prompting elicits reasoning in large language models, Advances in neural information processing systems, 35:24824-24837, 2022.
Niels A. Wouda, Leon Lan, and Wouter Kool. PyVRP: a high-performance VRP solver package. INFORMS Journal on Computing, 36(4):943-955, 2024. doi: 10.1287/jjo:.22230055. URL https:
0tt/:11.111110 1287/2177822 joc wttp:oc...2....00 2053.0055.
Xuan Wu, Di Wang, Chunguo Wu, Lijie Wen, Chunyan Miao, Yubin Xiao, and You Zhou, Ef- ficient heuristics generation for solving combinatorial optimization problems using large lan- guage models. In Proceedings of the 31 ACM SIGKDD Conference Knowledge Discov- ery and Data Mining V.2, KDD '25, pp. 3228-3239, New York, NY, USA, 2025. Associa- tion for Computing Machinery. ISBN 9798400714542. doi: 10.1145/3711866.3775588 https::/doi.org/ http://......................1.. 014444/37111666688 3736923.
URL
Violet Xiang, Charlie Snell, Kanishk Gandhi, Alon Albalak, Anikait Singh, Chase Blagden, Duy Phung, Rafael Rafailov, Nathan Lile, Dakota Mahan, al. Towards system reasoning Ilms: Learning how to think with meta chain-of-thought. arXiv preprint arXiv:2501.0468) 2025.
Liang Xin, Wen Song, Zhiguang Cao, and Jie Zhang. Neurolkh: Combining deep learn- ing model with tin-kernighan-helsgamn heuristic for solving the traveling salesman problem. In M. Ranzato, A. Beygelzimer, Y. Dauphin, P.S. Liang, and J. Wortman Vaughan (eds.), Advances in Neural Information Processing Systems, volume 34, pp. 7472-7483. Curran Associates, Inc., 2021. URL https: htttpp:://proceeedngg. neurips cc/ /c/ paper_files. Pappee...d.d...............
Fengli Xu, Qianyue Hao, Zefang Zong, Jingwei Wang, Yunke Zhang. Jingyi Wang, Xiaochong Lan, Jiahui Gong, Tianjian Ouyang, Fanjin Meng, al. Towards large reasoning models: A survey of reinforced reasoning with large language models. arXiv preprint arXiv:2501.09680 2025.
An Yang, Anfeng Li, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Gao, Chengen Huang, Chenxu Lv, et al. Qwen3 technical report. arXiv preprint arXiv:2505.09388, 2025.
Chengrun Yang, Xuezhi Wang, Yifeng Lu, Hanxiao Liu, Quoe V Le. Denny Zhou, and Xinyun Chen. Large language models as optimizers. In The Twelfth International Conference on Learning Representations, 2024. URL https: /ttp::ss////peevvveeeeee net/ forum? Cowrumiid-bbbraee. Bet.--GG-bbA............
Shunyu Yao, Dian Yu, Jeffrey Zhao, Izhak Shafran, Tom Griffiths, Yuan Cao, and Karthik Narasimhan. Tree of thoughts: Deliberate problem solving with large language models. Ad- vances in neural information processing systems, 36: 180111809-1122 2023.
Haoran Ye, Jiarui Wang, Zhiguang Cao. Helan Liang, and Yong Li. DeepACO: Neural-enhanced ant systems for combinatorial optimization. In seveth- Conference on Neural Information Processing Systems, 2023. URL https: http:p::///peeeeeeee /ttp///pp///eeeeeeeeeee/ee www.eereewewwneee.eeeee nttp: d=cd5n
14

---

Published as a conference paper at ICLR 2026
Haoran Ye, Jiarui Wang, Zhiguang Cao, Federico Berto, Chuanbo Hua, Haeyeon Kim, Jinkyoo Park, and Guojie Song. Reevo: Large language models as hyper-heuristics with reflective evolution. In A. Globerson, L. Mackey, D. Belgrave, A. Fan, U. Pa- quet, J. Tomezak, and C. Zhang (eds.), Advances in Neural Information Process- ing Systems, volume 37, pp- 43571-43608. Curran Associates, Inc., 2024. URL
https://prroeeesings neurips cc/paper sttparrppscpppppee_.eeee////appp decd544d Paper- Poper-CCofererce.ppd..
Qiying Yu, Zheng Zhang, Ruofei Zhu, Yufeng Yuan, Xiaochen Zuo, Yu Yue, Weinan Dai, Tiantian Fan, Gaohong Liu, Lingjun Liu, Xin Liu, Haibin Lin, Zhiqi Lin, Bole Ma, Guang- ming Sheng, Yuxuan Tong, Chi Zhang, Mofan Zhang, Wang Zhang, Hang Zhu, Jinhua Zhu, Jiaze Chen, Jiangjie Chen, Chengyi Wang, Hongli Yu, Yuxuan Song, Xiangpeng Wei, Hao Zhou, Jingjing Liu, Wei- Ying Ma, Ya-Qin Zhang, Lin Yan, Mu Qiao, Yonghui Wu, and Mingx- uan Wang. Dapo: An open-source Ilm reinforcement learning system at scale, 2025. URL https: /arxivv.. org/ / 14476.
Zhi Zheng, Zhuoliang Xie, Zhenkun Wang, and Bryan Hooi. Monte carlo tree search for com- prehensive exploration in LLM-based automatic heuristic design. In Forty-second International Conference on Machine Learning, 2025. URL
net forum? iid=
DolodZzYHr.
Jianan Zhou, Zhiguang Cao, Yaoxin Wu, Wen Song, Yining Ma. Jie Zhang, and Chi Xu. Mvmoe: Multi-task vehicle routing solver with mixture-of-experts. Proceedings of Machine Learning Research, 235:61800 61824, 2024.
15

---

Published as a conference paper at ICLR 2026
A
APPENDIX
A.1 ABLATION STUDIES ON REWARD DESIGN
To benchmark the effectiveness of our continuous reward design (Equation 3), we compare it with discrete version defined as follows:
o C
0.8
OE o
ra(o)
0.9
OE E
0
O P, tHGs (Dexpen) oi (o) O o & P. this (Despen) > DIGS (0)
Table 4: Ablation study reward design. FRTHGS& is the 14B LLM trained with reward in Equation 4. FRTHGS, is the 14B LLM trained with reward in Equation 3. Shaded areas are gener- alization results.
n. [100, 200)
E [200, 400)
[400, 600)
E [600, 800)
m E [800, 1000)
Mothods
Gapt8 (L) Time (s)
Gap% (4)
Time (s)
Gap% (4)
Time (s)
Gap (L) Time (s)
Gapf% (3)
Time (s)
HGS
0.62
12.45
1,85
28.16
1.95
58.41
2.62
91.31
2.32
121 04
FRTHGS
0.83
11.75
1.78
28.89
1.92
58.12
2.64
94.30
2.30
120.28
FRTHGSS
0.70
13.14
1.67
29.60
1.83
61.65
2.59
92.17
2.24
118.59
Table 4 demonstrates the clear advantage of our continuous reward design. RFTHGS, consistently outperforms the discrete-rewart variant RFTHGS&, particularly on larger problem sizes. The dis- crete reward's 's binary nature (0 or 1) provides limited guidance. Once an operator beats the baseline, the gradient vanishes as all improvements receive the same reward, and advantage is thus (Equation 2). In contrast, our continuous reward offers feedback proportional to performance gains, enabling sustained refinement and explaining its superior performance.
A.2 THE USE OF LARGE LANGUAGE MODELS (LLMS)
We clarify that all intellectual contributions in this work, from initial idea conception and algo- rithm design to experimental implementation and result validation, were conducted exclusively by the human authors. While we employed L.MMs during the manuscript preparation phase to refine language expression and improve readability, their role was strictly limited to linguistic polishing. The manuscript's structure, core arguments, and all substantive content were determined entirely by the human authors, with LLMs serving merely grammatical accuracy, similar to traditional proofreading services.
auxiliary tool for enhancing clarity and
A.3 PROMPT TEMPLATE
ROLE: Expert C++ Optimization Engineer for Vehicle Routing Problems
You are a senior C++ optimization engineer with expertise in algorithmic optimization, particularly for Vehicle Routing Problens (VRP) Your task is analyze and improve the s crossover algorithm.
file'
## TASK OVERVIER
You are given the file selective www.tteeeccange.coo.m (full listing below). Your goal is make ONE small, reliable modification that tends to create
children with better penalised cost (Solution quality t) while keeping runtime and interface intact.
16

---

Published as a conference paper at ICLR 2026
## THINKING PROCESS REQUIREMENTS
1. First, thoroughly analyze current implementation understand: The algorithm's s purpose and workflow
Key decision points and hcuristics
Performance bottlenecks optimization opportanities
Any constraints that must be preserved
2. Generate at least 3 different modification approaches, evaluating each on:
Potential improvement to solution quality
Impact on runtime performance
Compatibility with existing code Risk of introducing bugs or side effects
3. For your chosen modification:
Justify why it's s likely to improve solution quality Verify it maintains the function signature and behavior
Doublc-check for compatibility with the rest of the codebase Consider edge cases and verify robustness
*********
## HARD RULES (Mandatory Veri fication Checklist)
1. O Keep the function signature and namespace exactly the same: pyvrp: cerossoer:ss: weleeeeeeeeeeeeeeeeeeeeeeeeeee (...)
2. 0 The file must still compile under C++17 with the current Pinclude lines.
You may NOT remove #inelude directives.
3. Do not change any public headers, class interfaces, or external behaviour
except for the improved offspring quality.
4. DO NOT fabricate or use non-existent, or unment oned attributed or methods.
Verify fy every method you use exists in the provides ded code documentation.
5. Wrap the code you output with cppp and "..
6. D Mark ALL your modifications with clear *// MODIFY; XXX" comments explaining the change.
7. You must make at least one modification; DO NOT copy the original code.
8. Before finalizing, double-check that your mocification:
Does not introduce new parameters
Does not change the functien's contract
is focused on improving solutien quality, not runtire
1s fully compatible with the existing codebase
Uses only documented methods and attributes
#*
## DELIVERABLES (strict):
A. < 2-sentence summary of the optimization idea, clearly explaining how it improves solution quality.
B. Output the FULL C++ code with your modi fications. Mark all changes with "// VODIPY: xxx" comments.
C. Brief explanation of verification process and why you' re confident the modification will:
17

---

Published as a conference paper at ICLR 2026
Improve solution quality
Maintain compatibility with existing codebase Not significantly impact runtime performance
## SCORING AND EVALUATION
We will benchmark on a fixed randon seed over several CVRP instances.
Your patch should reduce the average opt imai gap in 2908 of the instances witnout
increasing total runtime by >38.
Key considerations for high-quality solutions:
More efficient route structures (fewer vehicles, shorter routes)
Better client assignment to routes based spatial relationships
Improved handling of capacity constraints
Frescrvation of high-quality route segments during crossover Better diversity in the gencrated offspring
#f seleecive_routeeexclannee cppp (code) ...
I Extra Information:
II DOMAIN KNOWLEDGE; CVRP AND CROSSOVER OPERATIONS
The Selective Route Exchange a crossover operation for the Capacitated Vehicle Routing Probiem (CVRP). The algorithm:
1. Selects routes from two parent sclutions
2. Exchanges these routes to create offspring
3. Aims to preserve beneficial route structures while creating new combi combinations
### Key Optimization Areas to Consider:
Route selection strategy (which reutes exchange)
Client-to-route assignment decisions
Proximity/disstace calculations between routes or clients Handling of capacity constraints
Diversity generation in offspring solutions
# Essential Fields and Methods for CVRP Crossover
* **ProblemData Key Methods:**
'numlocations() Returns 'size_t' total number of locations (depots elients)
'numClients() Returns 'size_t' number of client locations
"centroid()" Returns 'std::pair<double, double> center of all client
locations
'client (idx) Returns " with coordinates (x, y)
Route Key Methods:**
(centroid() Returns 'std::pair<double, d center of route's elient locations
vvchicccType () Returns "ehiiceeTpp'' (size_t) vehicle type index
(bogin() 'end() Iterator support for visiting clients in route
'sszc() Returns 'sizc_t" number of clients in route
-visits Returns "std::vectto<llieet>> all client indices in route
order
**Rute Construction:
'ooue (data, visits, vehicleType) Constructor taking 'std: ivector< Client> visits and vehicle type
18

---

Published as a conference paper at ICLR 2026
**lien Iteration:**
Routcs are iterable containers of 'Client' (size_t) indices
Use range-based for loops: 'for (client route) to access all
clients in route
Client coordinates: 'data.client (c) x', 'data.client (c)
A.4 RFTHGS PSEUDO CODE
Algorithm 1: RFTHGS:
Reinforcement Finetuning For Refining HGS
Input:
Initial policy Taw full instance set I, instance batch size B, clipping parameters
Slower; Supper, group size G Output: Optimized policy m
Initialize & fold:
.N do foreach iteration ...
Draw a random subset B с I with [B] = B:
foreach instance I B do
Generate operators with LLM Construct prompt for CrossOver operator optimization; Sample G operators (o,,..., ,a) ~ Te [g);
// Stee 1:
// Step 2: Evaluate with PyVRP
foreach operator o, do
Run PRP solver with o substituted on instance set B; Obtain objective value and compute reward r;;
end
// Step 3: Compute advantages (21.72)...,
Normalised reward for each operator: A., = mean(r).
std(r)
// Step 4: Update policy with DAPO foreach loken position L do
r0) + Ta(o; 4, 9,0c).
Tea (or (,02)) Elmin(r:(0)) A., cl), 1 Slower, 1 Eupper) Ar)]:
Lpolicy
end
0 + 0 + a V L
end
end
A.5 OPTIMIZING OTHER OPERATORS USING RFTHGS
The RFTHGS framework is not only suitable for optimizing the crossover operator but also be applied to optimize other operators (or modules) influencing solution quality within HGS. In this section, demonstrate the application of RFTHGS optimize the subpopul i (subp) operator in HGS. The primary function of this operator is to manage population control by deter- mining which individuals eliminate based on cost and diversity metrics, ensuring the population size remains within defined thresholds.
To rigorously test the generalizability of framework, we applied RFTHGS to the Capacitated Vehicle Routing Problem with Time Windows (CVRPTW) variant. is important to emphasize that the configuration of the RFTHGS framework remained consistent with the experiments described in the main manuscript, where the POMDP formulation, training algorithm, base model, and reward calculation were identical.
Experimental Setup For training, generate dataset comprising 50 randomly generated CVRPTW instances with problem sizes ranging from 100 to 300 customers. For the testing phase, generated 100 new random instances for each problem size, extending up 1000 customers.
19

---

Published as a conference paper at ICLR 2026
The performance of the RFTHGS-optimizze subp operator CVRPTW instances is summarized Table 5, where the relative gap to the original HGS, calculated as (CostaPтииs Cost /earcss//Cotttii HGS. is reported. A negative value indicates that the RFTHGS- optimized operator outperforms the baseline HGS.
Results and Analysis
Table 5: Performance of the optimized subpopulation operator on CVRPTW instances. represent the relative gap (%) compared to the baseline HGS.
Values
instance size
100
200
300
500
1000
HGS-subp (%)
0.000
0.000
0.000
0.000
0.000
RFTHGS-subp (%)
-0.088
-0.213
-0.299
-0.289
-0.224
As shown in Table 5, the optimized subp operator consistently outperforms the standard HGS across all problem sizes. Furthermore, the operator demonstrates strong generalization capabilities, maintaining superior performance on much larger problem sizes (500 and 1000) that were not seen during training, achieving relative gaps of -0.289% and -0.224%, respectively.
Mechanism of Improvement An analysis of the code generated by RFTHGS reveals that the enhanced subp operator modifies the fitness evaluation logic. The LLM introduced a third ranking metric based on the number of routes in a solution. This new metric is weighted and combined with the existing cost and diversity rankings to determine individual survival during parent selection.
This modification proves effective because it introduces a selective pressure that favors solutions utilizing fewer vehicles. By biasing the population towards more compact configurations, the algo- rithm indirectly promotes lower total costs, as fewer routes inherently reduce the cumulative distance associated with frequent returns to the depot. These results confirm that RFTHGS is a framework capable of optimizing diverse components within mature solvers across different problem variants.
A.6 TRAINING COST ANALYSIS
We provide a breakdown of the computational resources for the RL training of our 14B model. The total training duration was approximately 98 GPU hours per device. Analyzing the cost per training step reveals total duration of roughly 660 seconds, which is composed of 360 seconds for LLM training (inference, backward propagation, and updates) and 300 seconds for operator evaluation (execution and reward calculation).
Since reward computation constitutes a significant portion of the training cycle, future implementa- tions can mitigate this bottleneck by adopting fully asynchronous training paradigms and utilizing lightweight sandbox environments, such as 1lm-sandbox (vndee, 2025). These tools enable the isolation of code execution CPU-only nodes, allowing for massive parallelization and reduced GPU idle time.
A.7 AST-BASED ANTI-PLAGIARIS! IMPLEMENTATION DETAILS
This mechanism determines whether two LLee-geerrteed C++ codes are structurally equivalent by comparing their Abstract Syntax Trees (ASTs) after normalization.
The process begins by invoking the Clang compiler frontend as a subprocess on each source file. Clang is called with the -asl-dumy=json flag, which outputs the full AST in JSON format.
Once the raw AST is obtained, we traverses the tree depth-first, locating all function definitions that have actual body. For each function, it constructs simplified dictionary retaining only the function name, type signature, parameter list, and body, discarding all metadata such as source locations, node IDs, and mangled names.
The function body is then recursively simplified. After simplification, the functions undergo alpha- renaming. This ensures that two programs with identical structure but different identifier choices produce the same normalized AST. Finally, the two normalized ASTs are serialized to JSON with deterministic key ordering and compared via string equality.
20

---

Published as a conference paper at ICLR 2026
This mechanism helps the framework determine whether the LLM has made genuine changes the code, helping to prevent reward hacking.
A.8 COMPARISON WITH COMPASS
We compare with COMPASS Chalumeau al. (2023), another SOTA neural combinatorial op- timization (NCO) baseline, published at NeurIPS 2023. Our implementation ran on CUDA 12.8 while the original COMPASS was developed for CUDA 11, which may lead to a performance gup relative to the reported results; we did not find critical bugs in our reproduction.
Table reports results on CVRPLIB X instances (sizes 100-200). Our method maintains superioir performance compared to the COMPASS baseline.
Table 6: RFTHGS vs. COMPASS on CVRPLIB X instances.
Instance
Obj. COMPASS
Gap COMPASS (%)
Obj. RFTHGS
Gap RFTHGS (%)
Optimal
X-n101-k25
42674
54.67
27597
0.02
27591
X-n106-k14
34259
29.95
26546
0.70
26362
X-n110-k13
15310
2.27
15084
0.75
14971
X-n115-k10
16544
29.79
12747
0.00
12747
X-n120-k6
14897
11.74
13392
0.45
13332
X-n125-k30
71921
29.50
55825
0.51
55539
X-n129-k18
29578
2.20
29105
0.57
28940
X-n134-k13
23375
114.13
11050
1.23
10916
X-n139-k10
15366
13.07
13612
0.16
13590
X-n143-k7
52516
234.50
15743
0.27
15700
X-n148-k46
47049
8.29
44036
1.35
43448
X-n153-k22
28583
34.70
21337
0.55
21220
X-n157-k13
18697
10.79
16925
0.29
16876
X-n162-k11
44795
216.84
14175
0.26
14138
X-n167-k10
24145
17.46
20894
1.64
20557
X-n172-k51
63857
40.02
45939
0.73
45607
X-n176-k26
59699
24.86
48370
1.17
47812
X-n181-k23
28537
11.61
25734
0.65
25569
X-n186-k15
53322
120.84
24299
0.64
24145
X-n190-k8
19590
15.37
17294
1.85
16980
X-n195-k51
73445
66.07
44681
1.03
44225
X-n200-k36
85408
45.80
58862
0.48
58578
Average
39253.05
$1.57
27420.32
0.70
27220.14
21


Reviews:
[{"review_id": "f7NawGxXSG", "reviewer": "Reviewer_DLLe", "summary": "This paper introduces RFTHGS, a reinforcement learning framework for fine-tuning a 14B parameter LLM to generate optimized crossover operators for the Hybrid Genetic Search (HGS) algorithm applied to the Capacitated Vehicle Routing Problem (CVRP). The approach uses a multi-tiered, curriculum-based reward function that progressively guides the LLM through three stages: generating compilable code, producing executable operators, and finally creating components that exceed expert-designed baselines.", "questions": "1. Why optimize only the crossover operator? What happens with mutation/selection?\n2. Can you provide results on other VRP variants to justify the broad claims?\n3. What is the total training cost, and how does cost-benefit compare to manual design?\n4. What specific code changes did the LLM discover, and why do they improve performance?\n5. Why train only on small instances if the goal is large-scale optimization?", "limitations": "", "rating": 4, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "1. Novel Contribution: First demonstration that a small fine-tuned LLM (14B) can generate operators outperforming expert-designed components in a state-of-the-art CVRP solver.\n2. Strong Performance: Achieves -0.12% to -0.33% gap reduction over HGS baselines and outperforms trillion-parameter models (GPT-4o, GPT-o3, GPT-o4-mini) in both compilation success rate (16/16 vs. 3-9/16) and solution quality.\n3. Good Generalization: Successfully generalizes from training instances (n<400) to large-scale problems (n=1000) and across different iteration budgets.\n4. Well-Designed Method: The multi-tiered reward with continuous feedback (Equation 3) and incremental compilation for efficiency are effective design choices.", "weaknesses": "1. Severely Limited Scope:\n- Only optimizes ONE operator (crossover) for ONE problem variant (CVRP)\n- Title/abstract claim applicability to \"VRP\" but no experiments on VRPTW, PCVRP, or other variants\n- No demonstration that the approach works for other operators (mutation, selection, etc.)\n- This fundamentally limits the practical impact and undermines generalizability claims\n2. No Computational Cost Analysis:\n- Missing total GPU hours for RL training\n- No analysis of cost for evaluating operators during training (HGS runs on 30 instances per iteration × 1000 iterations × batch size)\n- Unclear if performance gains justify the substantial training investment\n- Makes it impossible to assess practical viability\n3. Shallow Analysis of Discovered Operators:\n- Section 6 mentions human experts analyzed operators but provides no concrete details\n- No explanation of WHAT modifications were discovered or WHY they work\n- Appendix C analysis is referenced but not included\n- Missing opportunity to extract design insights that could inform future operator development\n- Example output in A.4 shows reasoning but doesn't explain actual performance gains\n4. Questionable Experimental Design:\n- Training only on n≤400 instances while claiming to solve large-scale problems (n=1000)\n- Only 30 training instances seems insufficient for RL training stability\n- No variance/error bars reported across multiple runs\n- Validation set selection process not described\n5. Incomplete Baselines:\n- No comparison with automated algorithm design methods beyond LLMs (e.g., automated parameter tuning, grammar-based hyper-heuristics)\n- NCO baselines (POMO, AM) are construction methods, not improvement methods like HGS\n- Missing recent learning-based metaheuristic operator design work\n6. Technical Gaps:\n- Reward values (-1, -0.8, -0.7) appear arbitrary with no sensitivity analysis\n- AST-based anti-plagiarism: similarity threshold and detection details unclear\n- Incremental compilation: 75% reduction not validated across all operators\n- DAPO choice not justified; no ablation vs. standard PPO", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "EoAKvgL5w1", "reviewer": "Reviewer_tVaA", "summary": "This paper tackles the problem of designing crossover operators for the Hybrid Genetic Search (HGS) algorithm. It introduces RFTHGS which prompts a small LLM to code potential crossover operators and fine tunes the LLM with RL given a reward based on if the code compiles, if it is plagiarized from examples and its task performance. They show results equivalent and in most cases better than state-of-the-art.", "questions": "I do not see where the meaning of underlined values is explained for table 1. I have assumed that they mean values are within one standard deviation of the best value.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "The idea of generating code with an LLM and fine tuning it with RL for the purpose of improving human designed heuristics is interesting and this paper has shown that it can be useful. It is impressive that the method clearly generalises well to larger unseen task.\n\n\nThe reward design is clearly well thought out to provide both necessary training speed and improving results without copying existing solutions.\n\n\nThe results are comprehensive, and compare against a wide range of existing benchmarks. The fact that this achieves state-of-the-art performance in most cases is impressive.", "weaknesses": "My first concern is that it seems RFTHGS is not significantly better than the previous SOTA (HGS-PyVRP). As given in Table 1, HGS-PyVRP is always underlined when it is second best. This is not explained in the text, but I would assume it means within 1 standard deviation. Thus the results may not be statistically significant, this needs to be clarified in the text.  \nAnother concern along the same lines of comparing to HGS-PyVRP, is that the chosen generated crossover mechanism is never compared to the human designed heuristics and so it is hard to know how novel the new crossover algorithm is. It would be useful to have a discussion around the differences so that readers can know if these are trivial changes or significant algorithmic innovations.\n\n\nThe ablation study on reward design is lacking. It would have been interesting to see the impact of each part of the reward, for example how does the LLM perform without the plagiarism penalty?\n\n\nWhile the benchmark is comprehensive it seems to be lacking a SOTA NCS method in “Combinatorial Optimization with Policy Adaptation using Latent Space Search” (COMPASS) [1]\n\n\nMy final concern is the lack of generality of the method. It would have been interesting to see if the method works generally for defining heuristics in other evolutionary algorithms e.g defining new mutation strategies in Genetic algorithms, new velocity update rules in Particle Swarm Optimization, or novel pheromone update rules in Ant Colony Optimization.  \nAlong with this HGS is somewhat specialized to VRP problems and when testing on other algorithms it would be interesting to see the performance on other combinatorial optimisation benchmarks like traveling salesman and job shop scheduling.\n\n\n[1] Chalumeau, Felix, et al. \"Combinatorial optimization with policy adaptation using latent space search.\" Advances in Neural Information Processing Systems 36 (2023): 7947-7959.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "ovYCefojqM", "reviewer": "Reviewer_QZKW", "summary": "This paper introduces RFTHGS, a reinforcement learning framework that fine-tunes a 14B parameter LLM to generate crossover operators for the Hybrid Genetic Search (HGS) algorithm, specifically targeting the Capacitated Vehicle Routing Problem (CVRP). The authors employ a multi-tiered reward function that progressively guides the LLM through stages of producing compilable code, executable operators, and ultimately components that outperform human-designed ones. They also implement an operator caching mechanism using Abstract Syntax Trees to prevent plagiarism and promote diversity during training. Experimental results on CVRPLIB benchmarks demonstrate that the fine-tuned LLM generates crossover operators that significantly outperform expert-designed operators in HGS, with improvements holding from small instances (100 nodes) to large-scale problems (1000 nodes). The method also surpasses various baselines including neuro-combinatorial approaches and prompt-based methods using commercial LLMs like GPT-4o. This work demonstrates that smaller, specialized LLMs can be effectively fine-tuned to produce solver components that exceed the performance of human-expert designs in state-of-the-art optimization solvers.", "questions": "Besides the concerns in the weakness part, I also have the following questions:\n\n1. Why do the other prompting-based methods like ReEvo have so poor performance? Are the prompting-based baselines (ReEvo) actually integrated to evolve the crossover operator within the HGS framework, or were they used to evolve standalone heuristics?", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The authors present their work with exceptional clarity, following a logical progression from problem formulation through methodology to comprehensive experimental validation. The paper effectively uses figures (particularly Figure 1's pipeline visualization) and maintains consistent terminology throughout.\n2. The method achieves substantial improvements over expert-designed operators in HGS and consistently outperforms all baselines including state-of-the-art neuro-combinatorial methods (POMO, RF-POMO) and commercial LLMs (GPT-4o, GPT-o3) across problem sizes ranging from 100 to 1000 nodes, showing excellent scalability.\n3. This work pioneers the use of reinforcement learning to fine-tune language models specifically for generating optimization solver components, demonstrating that LLMs can learn to produce code that not only compiles and executes correctly but actually surpasses human expert designs in a state-of-the-art solver like HGS.\n4. The paper introduces several technical innovations including incremental compilation (reducing compilation time to 25% of full recompilation), an anti-plagiarism cache using Abstract Syntax Trees to encourage diverse operator generation, and a well-designed multi-tiered reward function that effectively guides learning through progressive stages from syntax correctness to performance optimization.\n5. The work demonstrates that a fine-tuned 14B parameter model significantly outperforms trillion-parameter general models (GPT-4o series), with Table 2 showing perfect compilation rates (16/16) compared to poor rates for GPT models (3/16-9/16), and the GPT models failing to produce any functional improvements despite generating syntactically correct code, highlighting the superiority of task-specific fine-tuning over general-purpose capabilities.", "weaknesses": "In general, I believe this is a good paper and lean toward acceptance. If the authors could address the following concerns, the paper could be stronger:\n\n1. The paper exclusively evaluates on the CVRPLIB X benchmark set, which, while comprehensive, represents only one distribution of CVRP instances. Evaluating on additional distributions would strengthen the generalizability claims. For example, testing on synthetic CVRP settings with controlled characteristics (clustered vs. uniform customer distributions), other CVRPLIB instance families (A, B, E, F sets), or real-world logistics datasets would provide more robust evidence of the method's effectiveness across diverse problem structures.\n2.  The evaluation is confined to optimizing a single operator (crossover) within a single solver (HGS). Expanding to additional settings would make the contributions more compelling. For instance, applying the framework to other state-of-the-art solvers like LKH-3, OR-Tools, or Concorde, and targeting different operators such as local search moves, construction heuristics, or selection mechanisms would demonstrate the broader applicability of the RL fine-tuning approach for solver component design.\n3. The method appears to require substantial domain-specific engineering, including detailed operator API specifications, incremental compilation setup, integration with solver codebases, and carefully crafted reward functions specific to each operator type. These requirements may create significant barriers for researchers wanting to apply this approach to new problem settings or different solvers, potentially limiting the practical impact of the work. The authors could strengthen the paper by providing a more general framework or toolkit that abstracts away some of these implementation complexities.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "waInpCex27", "reviewer": "Reviewer_CpHG", "summary": "This paper introduces RFTHGS, a novel Reinforcement Learning (RL) framework for fine-tuning a small LLM to produce high-performance crossoyer operators for the Hybrid Genetic Search (HGS) solver to solve the capacitated vehicle routing problem (CVRP). It presents a framework for automated heuristic design. The novel combination of a curriculum-based reward (compilability, executability, performance), an AST-based anti-plagiarism cache, and the use of the solver itself as the evaluation environment is a significant methodological contribution.It provide strong, empirical evidence that a small, accessible (14B) LLM, when fine-tuned via RL, can generate core algorithmic components that exceed the performance of human-expert designs within a SOTA CO solver.The strong generalization from small to large-scale problems is particularly impressive and demonstrates that the LLM has learned a genuinely robust and effective heuristic.", "questions": "1.Please provide a pseudo-code or a qualitative description of one of the  LLM-generated crossover operators.\n\n2.Please provide more details on the total computational cost.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "The paper proves that specialized, fine-tuned small LLMs are superior to prompted large LLMs.It is innovative and strongly supported by the data.The multi-tiered curriculum-based reward function solves the sparse reward problem in code generation. The AST-based anti-plagiarism mechanism is a great solution to prevent reward hacking and ensure the generation of novel, diverse operators.It offers a practical, accessible path for continuously improving SOTA solvers.", "weaknesses": "While the LLM is small, the RL training loop appears computationally intensive.The paper does not quantify the total training cost. In addition, the paper demonstrates that the LLM-generated operator is better, but not why. An analysis of the generated SOTA operator's code is needed.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)