HeuriGym: An Agentic Benchmark for LLM-Crafted Heuristics in Combinatorial Optimization
Authors: Hongzheng Chen
Yingheng Wang
Yaohui Cai
Hins Hu
Jiajie Li
Shirley Huang
Chenhui Deng
Rongjian Liang
Shufeng Kong
Haoxing Ren
Samitha Samaranayake
Carla P Gomes
Zhiru Zhang
Venue: ICLR 2026 Poster (2026)
Keywords: Benchmark
Large Language Models
Combinatorial Optimization
Code Generation
Agent
Automatic Heuristic Generation
Primary area: datasets and benchmarks
TL;DR: An agentic benchmark and framework for LLM reasoning capability in combinatorial optimization problems

Abstract: While Large Language Models (LLMs) have demonstrated significant advancements in reasoning and agent-based problem-solving, current evaluation methodologies fail to adequately assess their capabilities: existing benchmarks either rely on closed-ended questions prone to saturation and memorization, or subjective comparisons that lack consistency and rigor. In this work, we introduce HeuriGym, an agentic framework designed for evaluating heuristic algorithms generated by LLMs for combinatorial optimization problems, characterized by clearly defined objectives and expansive solution spaces. HeuriGym empowers LLMs to propose heuristics, receive evaluative feedback via code execution, and iteratively refine their solutions. We evaluate nine state-of-the-art models on various problems across domains such as computer systems, logistics, and biology, exposing persistent limitations in tool use, planning, and adaptive reasoning. To quantify performance, we propose the Quality-Yield Index (QYI), a metric that captures both solution pass rate and quality. Even top models like GPT-o4-mini-high and Gemini-2.5-Pro attain QYI scores of only 0.6, well below the expert baseline of 1. Our open-source benchmark aims to guide the development of LLMs toward more effective and realistic problem-solving in scientific and engineering domains.

Published as a conference paper at ICLR 2026
HEURIGYM AN AGENTIC BENCHMARK FOR LLM-CRAFTED HEURISTICS IN COMBINATORIAL OPTIMIZATION
Hongzheng Chen
Yingheng Wang
Yaohui Cai
Hins Hu
Jiajie Li
Shirley Huang Chenhui Deng
Rongjian Liang'
Shufeng Kong'
Haoxing Ren Samitha Samaranayake'
Carla P. Gomes' Zhiru Zhang'
1 Comell University
2 Harvard University
NVIDII
{hzchen, yingheng 8cs cornell.edu. {yc2632, zh223, 214222,,,14251 /ccone7 @cornell.ede
ABSTRACT
While Large Language Models (L..Ms) have demonstrated significant advance- ments in reasoning and agent-based problem-solving. current evaluation method- ologies fail to adequately assess their capabilities: existing benchmarks either rely closed-ended questions prone to saturation and memorization, subjective comparisons that lack consistency and rigor. In this work, we introduce HeuriGym, agentic framework designed for evaluating heuristic algorithms generated by LLMs for combinatorial optimization problems, characterized by clearly defined objectives and expansive solution spaces. HeuriGym empowers LLMs propose heuristics, receive evaluative feedback via code execution, and iteratively refine their solutions. We evaluate nine state-of-the-ar models on various problems across domains such as computer systems, logistics, and biology, exposing persistent limi- tations tool use, planning, and adaptive reasoning. To quantify performance, we propose the Quality-Yiec rate and quality. Even top models like GPT-o4-mini-hiig and Gemini-2.5-Pro attain QYI scores of only 0.6, well below the expert baseline of 1. Our open-source benchmark aims to guide the development of LLMs toward more effective and realistic problem-solving in scientific and engineering domains.
Index (QYT), metric that captures both solution pass
INTRODUCTION
Recent advancements in Large Language Models (LLMs) have significantly expanded their ca- pabilities in complex reasoning and agent-based problem-solving, enabling applications ranging Novikov et al. (025) to dynamic decision-making
from automated code generation (Li et al.
2022
systems (Schick et al.) 2023) Yao et al. 2023). Yet existing evaluation frameworks struggle to rigorously assess the full spectrum of these emergent abilities, often failing capture the demands of real-world tasks that require iterative reasoning, creative algorithm design, and adaptive tool use.
This shortcoming leaves critical gap in understanding whether LLMs can move beyond pattern recognition to exhibit genuine boblinn-ollvig ingenuity in practice.
Current evaluation paradigms fall into two categories with distinet limitations. (1) Ground-truth- based objective benchmarks rely on closed-fon foom questions (e.g., multiple-choice mathematics problems) that have become susceptible to rapid performance saturation. Widely used benchmarks such as AIME (Online 2025). HumanEval (Chen et al., 2021b), and GPQA Diamond (Rein et al.) 2024) now exhibit ceiling effects, with state-of-the-art models achieving over 80% accuracy (OpenAl
2025 Alibaba 2025 Mind 2025). Even emerging evaluations like Humanity's 's Last Exam (HLE) (Phan et al. (225),, initially proposed as rigorous PhD-level test, saw performance leap from 3% to 25% within months of release (OpenAl 2025). These benchmarks face dual crisis: their static question banks risk data contamination as models ingest newer training data, while their closed-ended nature fails to reflect real-world problem-sol where solutions neither unique predefined.
"Core Contributor

---

Published as a conference paper at ICLR 2026
(2) budge-preffreece-baseed subjective evaluations, such as Chatbot Arena (Chiang al. 2024).
take a different approach by assessing model quality through pairwise comparisons by humans or LLM-based proxies (Zheng et al. 2023). These benchmarks support wide range of plausible outputs, making them better suited for open-ended tasks. However, this flexibility introduces high variance: everyday communication tasks inherently subjective, and judgments often prioritize superficial factors like response structure or emoji usage over substantive reasoning quality (Singh et al. 2025
Zhang al. 2024). While recent efforts to automate evaluation with LLM-as-a-judee systems show promise, their reliability remains inconsistent across domains (Krumdick et al. 2025), particularly for technical tasks requiring specialized expertise.
To address these limitations, we introduce HeuriGym, new evaluation paradigm with an agentic framework centered on combinatorial optimization problems, which naturally combine well-defined objectives with large solution spaces. Rather than relying on well-known benchmarks such as SAT or
TSP, we assess whether L.LMs can produce high-quality solutions novel yet foundational problems
spanning computer systems (Cai et al. 2025: Moffitt & Fegade, 2025). scientific reasoning (Chen
ct al. 202 2016), computational biology (Dauparas et al.) 2025 Wijsman (012)). logistics (Li &
Lim
2001| Graves et al. 1993), and electronic design automation (Hofmann et al. 2025 Cong &
Zhang. 2006). They are ideal for benchmarking LLMs because they resist memorization due to their
computational hardness, offer clear metries for quantitative evaluation, and reflect real- world use cases where optimal solutions are tractable only for small instances. Since no single heuristic dominates across all problems or instances (Wolpert Macready [197), the search space rich and diverse. Tackling these challenges requires not only algorithmic knowledge but also heuristic reasoning, tradcoff navigation, and creative croblem-solving-- skills that are still underexplored in current LLM evaluations. Our framework extends beyond conventional static evaluations by implementing an interactive agentic loop: LLMs generate heuristic algorithms,, receive execution feedback from a code environment, and iteratively refine their solutions. This process mirrors practical engineering workflows and enables deeper evaluation of multi-step reasoning, tool use, and instruction following.
Our benchmark systematically evaluates LLMs across four dimensions: (1) tool-augmented reasoning through integration with external libraries, (2) multi-step planning in decomposing complex problems into executable sub-tasks, (3) instruction fidelity in adhering to problem constraints, and (4) iterative refinement based on runtime feedback. The framework uniquely probes practical creativity- the ability to adapt textbook algorithms or invent novel strategies for large-scale instances where exact methods like integer linear programming (ILP) may fail.
To capture both the number of feasible solutions and their quality relative to expert performance, we introduce a unified metric-the Quality- Index (QYI)-which ranges from 0 (all outputs incorrect low-quality) to 1 (expert-level performance). Empirical results reveal substantial performance gap: across nine diverse optimization problems. even ofate-of-the-art LLMs such as GPT-o4-minihhigh (OpenAl 2025) and Seemini-2.5-Pa (DeepMind) 2025) achieve QYI scores around 0.6, underscoring their limited effectiveness in realistic -oollig-solving settings. These findings highlight the limitations of current benchmarks, which fail to capture the complex, real- world demands of computational problem-solving-wwee where success requires integrating theoretical understanding, tool proficiency, and adaptive reasoning. The contributions of this work are threefold:
An open-source benchmark suite of nine combinatorial optimization problems that evaluates LLMs' multi-step reasoning capabilities through realistic programming tasks.
An end-to-end agentic framework supporting LLM solution generation, automated veri- fication, quantitative evaluation with well-defined metrics, and iterative refinement. The resulting system can also serve as testbed for exploring more advanced prompting tech- niques and evolutionary strategies.
A comprehensive empirical study of cutting-edge LLMss uncovering their current limitations and offering actionable insights for the development of next- next-generation models and agents.
2 RELATED WORK
LLMs for Combinatorial Optimization. Recent LLM- based combinatorial optimization (CO) meth- ods follow two main paradigms. The first emphasizes formalization. -troonnattnnnnnngggg natural language into structured optimization problems. This direction was initiated by the NL.4Opt Competition (Ra-
2

---

Published as a conference paper at ICLR 2026
Table 1: Comparison with other recent benchmarks.
Well-Defined Objective
I Large Solution Space
Agentic Setting
Evaluation
Subjects
Benchmark
Metries
Frontier
Humanit'ss y Last Examn
Knowledge
(HLE) (Phan al. 2025
X
X
Accuracy
HumanEval (Chen all
2021b
K
X
PASS
BigCodeBench (Zhuo cl a
2025
X
X
PASS @
Software
Engineering
LiveCodeBench Dam et 2025
X
X
PASS @
SWE-Bench (Jineeee al
X
PASS
CommitO Zhao et 2025
X
Pass rate
Performance
Engineering
KernelBench Ouyang el
2025
X
X
FAST.
Daily-Life Tasks
Chatbot Arena (C hrang et al.
2024
X
X
ELO PASS
r-Bench Yao ct al. 2025
V
NPHardEval
al. 2024
X
Accuracy
Combinatorial
GraphArena Tang
X
X
Accuracy
Optimization
ALE-Bench [majuku et al. 2025
V
ELO
HeuriGym (Thss work)
V
SOLVE, @i, QY1
mamonjison et al. 2023), with follow-up work improving Nomain-specifti model training (iia et al.
2023[JIANT et al. 2025 Li et al.
2025) and prompting strategies (Yang al. [2024] AemaaiTTeshnizi
et al. 2024 Ikassov et al. 2024). While effective on benchmarks, these methods struggle to scale due
their reliance exact solvers (Gurobi, 2025) The second paradigm focuses on heuristic discovery. FunSearch Shomera-Paredee et al., 2024) and AlphaEEvolve (Novikov et al.) 2025) use LLMs with evolutionary search to generate novel heuristics, but require evaluating thousands of candidates. Recent approaches YYe et al. 2024 Liu et al. 2024b Dat et al. 2025) improve efficiency via meta- heuristic templates, but still limit LLMs to filling in small portion of the algorithm. LLM4AD (Liu et al. 2024c) offers a platform for evaluating such template-basee methods. In contrast, HeuriGym removes reliance on templates or scaffolds. It tasks LLMs with generating complete, self-contained optimization programs, including custom data structures and end-to-end pipelines-bette reflecting real-world CO challenges, where success depends on uncovering problem-specifi structure and designing bespoke algorithms (Wolpert & Macready 1997).
Evaluation on LLMs. As shown in Table existing LLM benchmarks expose key limitations. Many focus on closed-ended tasks in domains like mathematics (Online 2025), programming (Chen
ct al.
2021b Zhuo et al.) 2025 Liu et al. 2023), and specialized knowledge (Rein et al. 2024 Phan
et al.
2025 Hendrycks et al. 2021), with fixed ground-truth that are prone data contamination (see $
$ In contrast, open-ended benchmarks (Chiang et al.) 20224] Ouyang et al. 2025) encourage
diverse outputs but often lack clear objectives, resulting in inconsistent evaluations. Benchmarks like NPHardEval (Fan et al. 2024) and GraphArena (Tang et al., 2025) assess exact solutions to small
NP-hard instances, limiting real-world relevance where heuristic solutions are often preferred for scalability. Our benchmark instead accepts any feasible solution that satisfies constraints, enabling broader evaluation of algorithmic reasoning. It tasks L.LMs with synthesizing executable code, using external libraries, and refining solutions through execution feedback, mimicking realistic workflows. ALE-Bench (Imajuku et al. 2025) and CO-Bench (Sun et al.) (025) are concurrent efforts that focus on score optimization for classic CO problems using metaheuristics, whereas HeuriGym targets practical, high-impact CO problems from scientific and engineering domains, requiring LLMs to discover moblem--seciffe structure and design tailored heuristics. Unlike ALE-Bench's 's ELO-style scoring, we propose fine- fine-graine preegrained SOLVE, E @i and QYI metrics that reveal which stage agents fail at and how close they are expert solutions in multi-round reasoning setting, detailed in 3.2
3 HEURIGYM: AN AGENTIC FRAMEWORK FOR HEURISTIC GENERATION
In this section, we introduce our agentic framework for evaluating LLM reasoning via iterative heuristic generation, along with benchmark metrics for quantitative assessment.
3.1 OVERVIEW
As illustrated in Fig. our framework presents a formal problem description to the LLM, which is then prompted generate complete heuristic algorithm conforming standardized function signature.
3

---

Published as a conference paper at ICLR 2026
Description
Heuristic
Final
Prompt
Generate
Alpecittin Jegiteminiaaioo Ion ded ao2vee Liroue Failes ats, dol coluttion, fitel wbat)
electronio design
1800,
LLM
corsiiraatio
Consudter custere data LLoN groph I0Br9
Stage Execution
the specatos schedai ing problem
to to thnd resetale dohedule de Uin 20
Loge
Compller linterpreter
Demo
stat vininizes the wvezell Lubancy 128
min magifi +.)
Feedback
Stage Solution Gen.
Solution File
prosided
with the following atructure: 2on
Stage Verification
Condraints
Verifirr
II Output Tomat
output abould peovide the esscution ochedule of grograe, For excpple,
Cost
Evaluator
Figure 1: Overview of the HeuriGym agentic framework for heuristic program generation, execution, and verification. We use operator scheduling as an example for the problem description.
It is subsequently compiled (for C++) interpreted (for Python), and verified for correctness and performance. Crucially, the framework incorporates feedback loop: execution logs, verification outcomes, and evaluation costs from small demonstration set appended back the prompt, enabling iterative refinement the LLM-generated solution.
3.1.1 PROBLEM DESCRIPTION
As shown on the left of Fig. [ we use operator scheduling (Cong & Zhang 2006). classic optimization problem in electronic design automation, as an example. Each benchmark task is accompanied by a structured problem description with three main parts: (1) Background: Introduces the optimization context and key terminology to help the LLM understand the problem setting. (2) Formalization: Defines the optimization objective and constraints using mathematical notation (e.g.. minimizing latency under hardware resource constraints), guiding the LLM toward objective-orienter algorithm design. (3) Input/Output Format: Specifies the structure of input/output files, providing clear expectations for parsing and execution. More details the problem set can be found in
3.1.2 PROMPT DESIGN
Effective prompt engineering is crucial for leveraging LLMs' capabilities (Wei et al. 2022, Sahoo al. 2024). We construct both system- and user-level prompts, tailored to each problem. A complete prompt example is provided in Appendix
System prompt. The system prompt includes machine configuration details (e.g., CPU cores, memory limits), available libraries with version numbers, and task-specific constraints such as execution timeouts. This environment specification instructs the LLM to avoid relying on unrealistic assumptions producing inefficient solutions that violate runtime limits.
User prompt. the initial iteration, the user prompt includes the problem description and a code skeleton with predefined function signature. As shown in Fig. the L.MM is only provided the interface-function name, input path, and output path- without hints on data structures or algorithmic approache, contrasting with prior work (Romera- (Romera-Paredes et al.) 2024 Liu et al. 2024b Ye al.) 2024) that often handcrafts partial implementations or restricts the design space. Here, LLMs must reason about the problem holistically: parsing inputs, constructing internal representations, and designing and implementing heuristics from scratch.
3.1.3 FEEDBACK LOOP
To emulate a few-shot in-context learning setup (Dong et al. 2024 Liu et al. 2022 Wu et al. 2023
partition the dataset into small demonstration set (around five instacces and larger evaluation set. Demonstration data is used during the refinement loop to provide timely, example-bar based feedback to the L.M;; the evaluation set is withheld until the model stabilizes its performance.
4

---

Published as a conference paper at ICLR 2026
Each problem includes a domain specific verifier and evaluator. The verifier ensures constraint satisfaction (e.g., dependency preservation in operator scheduling), while the evaluator calculates the cost based on the given problem objective. If the verifier fails, diagnostic messages are recorded.
After each iteration, we log the LLM- solution, execution trace, verification result, and the objective score. These logs appended to the prompt with the demonstration data in the next iteration, enabling the LLM to learn from past attempts and incrementally improve its output.
3.2
METRIC DESIGN
Traditional LLM benchmarks predominantly rely on the PASS@k metric (Chen et al., 202 Zhuo
et al. 2025 Jimenez al., 2024), which measures the probability of generating a ground-truth solution within the top-k samples. While PASS @ is effective for single-turn tasks with deterministic ground truths, it falls short in capturing the iterative reasoning and problem-solving abilities required in our multi-round agentic setting. Specifically, it does not reflect whether the LLM can understand problem constraints, debug based on feedback, or iteratively refine its solutions over multiple attempts.
To better evaluate LLMs in this complex setting, we introduce a new metric, denoted as SOLVE, @i, which tracks the LLM's ability solve constrained problems within iterations:
N 1 (pass stage s in the first i-th iteration).
SOLVE, @i =
N ==1
where N is the total number of instances that are fed to LLMs as inputs, and s E {I. II. IIII denotes the pipeline stage that the solution must pass, each representing key milestone agentic reasoning:
Stage I: Execution.
The generated program must compile interpret correctly with all necessary libraries included, and successfully perform basic I/O operations (e.g., reading and writing files).
Stage II: Solution Generation. The program produce non-empty output within the predefined timeout and adhere to the expected output format.
Stage III: Verification. The solution must satisfy all problem-specific constraints, as checked by a problem-specific verifier.
However, SOLVE, @i only indicates whether feasible solution is eventually produced through the iterative process- it does not account for solution quality. To address this, we additionally define separate metrics for quality and yield follows:
N
QUALITY
min
1,
YTELD = N
1=
where c. and c₁ represent the cost (i.e., the optimization objective) of the LLM-generated and expert- provided solutions, respectively, and N is the number of instances that pass verification (Stage III) in one iteration. We adopt the capped version of quality, which checks whether the LLM matches expert performance (up maximum of 1), though where the LLM outperforms the expert. We define a unified metric, the Quality Yield Index (QYT), as the harmonic mean of quality and yield. This formulation, analogous the F-score (Van Rijsbergen 1979), penalizes imbalanced values more strongly than the arithmetic mean:
uncapped version also be used measure cases
(1+ (P) QUALITY YIELD (B2 QUALITYY + YIELD
QYI
where I controls the relative importance of YIELD compared to QUALITY. By default, we use 3 = 1 in our evaluation. QYI captures both success rate and the relative quality of solutions, enabling holistic evaluation of an LLM's agentic reasoning capabilities, including its capacity for long-horizon planning and iterative refinement. Additionally, we define a weighted QYI by averaging QYI scores with weights proportional to the number of instances in each task, to provide an aggregate measure of overall LLM performance. Nevertheless, we continue to report per-problem QUALITY and YIELD to enable clearer inspection of tradeoffs across individual tasks.
5

---

Published as a conference paper at ICLR 2026
Table 2: Existing combinatorial optimization problems in our HeuriGym benchmark.
Domain
Problem
References
Difficulty
Electronic Design
(EDA)
Uperator scheduling
Technology mapping
Inlmani all (2025
et
(2024 Cong Zhang
Chen
2006
2004
Global routing
Liann CT al. 2022
Liao el at
2020
Compilers
E-graph extraction
Cai et (2025
Willsey cl
2021
Intra operator parallelism
Mollil & Fepade
Zheng 1999
2022
Computational
Protein sequence design
Dauparis di:
4425
Kleinberg
Biology
Mendelian ervor detection
Liinerre 2025
SSanclleez al
2008
Logistics
Aitline crew pairing
Korte & Yorke-
Apcarwal
Pickup and delivery w/ time windows
Tanguuchi ét 2025
& Lim
2001
***
4 BENCHMARK CONSTRUCTION
This section outlines the construction of our combinatorial optimization benchmark, detailing the principles behind problem selection and providing an overview of the resulting problem set.
4.1 PROBLEM SELECTION CRITERIA
Our primary goal is to evaluate an LLM's capacity for reasoning rather than its ability to regur- gitate well-known algorithms. To this end, we intentionally exclude ubiquitous problems such as TSP (Robinson) 1949) and SAT (Schaefer) 1978)-problems that are so widely studied and frequently included in public datasets that they are likely memorized during pretraining. Instead, focus problems that meet the following criteria:
Limited exposure in the literature. For each candidate problem, perform Google Scholar search and retain only the most-cited paper has fewer than 1,000 citations (as of May 2025). This 1,000-citation cutoff is practical criterion to exclude heavily standardized textbook CO problems that are almost certainly present in LLM training corpora, while still preserving well-defined, peer- reviewed problems. This empirical threshold increases the likelihood that the LLM must genuinely reason and design new heuristics rather than rely on cached or widely publicized patterns.
Clear natural-language specification with well-defined objectives. Each problem must be clearly expressible using plain language without the need for visual aids. We encode mathematical objectives ыЛTX to eliminate ambiguity, ensuring the LLM receives well-specified instructions.
Large solution spaces. We focus on problems that admit vast solution spaces with many feasible out- puts, encouraging creative exploration and reasoning rather than narrow pattern recognition (Hughes) et al. 2024); in our benchmark, single instance can present search spaces orders of magnitude larger than those of most existing benchmarks.
Scalable data instances. Each problem includes two disjoint sets of instances: small-scale demonstration set and large-scale evaluation set, differing by at least an order of magnitude. The demonstration set supports few-shot prompting and iterative refinement, while the evaluation set is reserved for final performance testing, as discussed in $ 3.1.3
Reproducible expert baselines. Reference implementations are included in the benchmark repository to ensure fair comparisons future studies. These expert baselines are often problem-specife heuristic methods, representing the best-known results from the literature, and are denoted QYI 1.0 in our evaluation to highlight the performance gap. If some problems can be expressed as mixed- integer programs, additionally provide formulations that be with commercial solvers like Gurobi. These are just for understanding the gap between heuristic solutions and the optimal solution on small-scale instances.
We prioritize domains with real-world impact, where even small gains yield significant societal industrial benefits. Many selected problems remain open, with heuristics far from theoretical bounds offeringf a compelling testbed for LLMs.
4.2 DATASET STATISTICS
The initial release HeuriGym spans nine optimization problems across scientific and engineering domains, covering fundamentally different types such covering, scheduling, and routing in Table

---

Published as a conference paper at ICLR 2026
Each problem provides five demonstration instances that capture different constraint requirements and provide representative guidance, and dozens of large-scale evaluation instances, totaling 218 (distribution in Appendix G A small number of demonstration instances is a practical consideration
to reduce evaluation time per iteration and also aligns with few-shot L.LM usage (Wei et al. 2022 Dong al. 2024). The effectiveness of the demonstration set is further analyzed in $15.2
For each instance, the search space is enormously large, growing combinatorially with problem size. This yields many distinct valid solutions rather than a single ground truth, making exhaustive search entirely infeasible. In several tasks, the search space far exceeds even astronomical scales (e.g., 1065,000 for intra- antraooperator parallelism), and when combined with hard constraints and non-linear objectives, the resulting problems cannot be handled by commercial solvers and are substantially more challenging than typical closed-form optimization benchmarks. All datasets are derived from realistic sources and real-world applications, enhancing the benchmark's 's practical relevance. Notably, most problems are NP-hard and feature complex constraints, resulting in a compact yet highly challenging problem suite. In addition, we reserve hundreds of instances as private test sets for future release. A detailed description of each problem is provided in Appendix D and empirical results ($ ( show that the benchmark remains substantially difficult for state-of-the thr LLMs.
To ensure clarity and correctness, we adopt a inntan-i--thh-loop process for problem specification. A human annotator first drafts the natural-language description, then uses weaker LLM such as DeepSeek-V3. V3 (Liu et al. 2024a) to highlight any unclear or ambiguous statements. The human and model iteratively resolve discrepancies until the specification is unambiguous and fully aligned with the intended semantics. Importantly, this procedure is soiely for refining the problem descriptions but not for improving solver performance. The assumption is that if a weaker model can successfully validate the specification's 's unambiguity and coherence, a stronger model will inherently possess sufficient understanding. The full prompt template used for refining problem descriptions is provided in Appendix C.4
Each problem includes task-specific verifier and evaluator to assess solution pass rate and quality. A separate reviewer ensures the expert solver reproduces published results and passes both checks.
Looking forward, we plan to extend HeuriGym along two axes: (1) breadth, by incorporating additional combinatorial optimization problems from underexplored scientific domains; and (2) depth, by scaling existing problems to larger instance sizes and tighter constraint settings. Community contributions are welcome, provided new problems satisfy the selection criteria articulated above.
EVALUATION
To evaluate the reasoning capabilities of LLMs CO problems, we benchmark nine prominent models released in late 2024 and mid-2025 ($ E.T) These models represent the current state-of-the- in general- purpose LLMs and rank among the top entries OpenRouter (OpenRouter 2025) and Chatbot Arena leaderboards (Chiang et al. 2024). We exclude smaller models due to the complexity of the benchmark tasks. All evaluations
conducted via official APIs ensure reproducibilitty We adopt the agentic workflow in Fig. [ constraining each model to generate Python programs that solve the given problems under fixed resource limits: a maximum of 8 CPU cores and problem-speeifie timcouts. We also allow the models to access the given Python libraries for external tool use. Full details of the experimental settings and results of each problem can be found in $ Notice the main goal of these experiments is not to present an optimal pipeline, but rather to establish a baseline for current LLM capabilities and demonstrate that HeuriGym serves as a common testbed on top of which more advanced prompting strategies and sophisticated agentic workflows can be developed. We further discuss these potential improvements in 6
5.1 OVERALL PERFORMANCE
For the overall evaluation, we fix the generation temperature at 0, following standard practice in recent LLM benchmarks (Ouyang et al., 2025 Yao et al. 2025 Phan et al., 2025). This ensures deterministic outputs and eliminates randomness across runs. Notably, OpenAl's 's o-series models only support a fixed temperature of 1.0 (OpenAl 2025). We measure the multi round performance using the SOLVE, @i i metric, where E [1,5, 10, indicates the number of iterations allowed.
7

---

Published as a conference paper at ICLR 2026
Table 3: Overall SOLVE, @i metric models on the whole HeuriGym benchmark.
SOLV VFm @
SOLVEI @5 @
SOLVE)
Model
@ I0
@
@ 10
@ 10
@
@
DeepSeek-V V
46.8%
42.7%
14.2%
87.6%
83.0%
66. 1%
100.0%
100.0%
90.8%
DeepSeek-RI
73.4%
72.9% %
44.0%
88.1%
88. 88.1%
60.6%
100.0% %
100.0%
71.6%
Gemini-2.5-Flash
67.4%
58.3%
25.2%
83.9%
79.4%
56 66.4%
100.0% %
100.0%
72.9%
Gemini-2.5-Pro
65.1%
64,2%
20.2%
89.4%
89.0%
42.7%
100.0% %
100.0%
51.4%
35.8%
33.5%
6.0%
84.9%
74.3%
8.3%
85.3%
85.3%
13.3%
LLaMA-3.3
33.9%
33.9%
20.6%
78.4%
78.4%
40.4%
99.5%
99.5%
61.9%
Qwen3-2358
45.9%
45.4%
38.5%
86.2%
83.0%
56.0%
100.0%
100.0%
70.6%
Claude S.nnoeet.
60.1%
58.7%
9.2%
97.7%
97.7%
41.3%
100.0% %
100,0% %
60.1%
GPT-o4-mini-high
74.8%
69.7%
53,2%
100,0%
100.0%
93.1% %
100.0% %
100,0%
100.0%
0Y1-0.40
QYL-0.50
0.8
QYL=0.80
Qualtty
0.6
Gemini-2. 2.5-
GPT--l
Gamini-2.5 X DeepSank-R1
GPT-ol-mini
Claude-2.7-5 3.7-5
Gemini-2.55PP Pr
0.6
Qwen3 2358
Swond-2350
0.4
Claude 3.7-5 onnet
0.4
LLaMa-d-tavaii
DegpS
NA- 3.3-708
LLaMA Maverick
(UA
0.0
0.2
0.4
0.6
0.8
1.0
10
Yield
Estimated API Cost (5)
cost Figure 2: Quality- Yield Index and estimated API cost of different models.
As shown in Table 3 most LLMs fail to solve large fraction of test cases within a single attempt, as reflected in the SOLVE score. Increasing the number of iterations generally improves performance across all models. For instance, the SOLVE success rate rises from 53.2% to 74.8% for GPT-o4-mini-higg as increases, underscoring the importance of iterative refinement in improving LLM-generated solutions. Among all models, -ii-oo-mini-high and DeepSeek- R1 demonstrate high success rates across multiple iterations, highlighting their stronger program repair capabilities.
To assess solution quality, we compare the final LLM- LLM--enerrtee programs expert-designed solutions using the weighted QYI metric defined in $ 3.2 As illustrated in Fig.2, substantial performance gap remains: even the best-performing model, Gemini-2.5-
achieves QYI of only 0.62, indicating that its solutions are, average, just 60% as effective expert-crafted ones. Several models, such as LLaMA-3.3 and -LaMA-4-Maverts their limited effectiveness these tasks. We also estimate the API cost for each model and find that Gemini-2.5-Flast offers the best cost-efficiency relative to its achieved QYI.
produce results with QYI scores below 30%, highlighting
We further compare state-of-the-an open-source evolutionary frameworks under the same setting. We fix the outermost evolutionary loop to 10 iterations and use a population size of 10; in total, each method therefore produces hundreds of candidate programs due to multi-step reasoning within each iteration. Further increasing the population size leads to context-length overflows for most of the problems inside HeuriGym. All frameworks in our benchmark, as the base LLM. The initial candidate program is generated directly from the problem description. As shown in Table 4 these frameworks perform poorly, often worse than the baseline model. Their main weakness is the lack of incorporating program execution feedback (errors and verification results, $ $3.113), and their search process breaks context across iterations, causing the system to repeatedly patch the same flawed initial program without real progress. This highlights both the complexity of our benchmark and the need for LLMMs to reason more deeply about problem-specific strategies. These frameworks originally only target toy problems under 20 lines of code (e.g., TSP, bin packing), while our benchmark typically requires 300+ lines, making strategy discovery essential rather than relying on prebuilt metaheuristics. These results also underscore the importance of better prompt design and context engineering. Simply appending all sampled programs to the prompt does not scale to our benchmark. limiting the population size per iteration. Moreover, improved mechanisms are needed to consolidate and reconcile feedback from different sampled candidates that refinements do not conflict with one another.
Gemini-2.5-PPrt the best-performing model
8

---

Published as a conference paper at ICLR 2026
Table 4: Performance of evolutionary frameworks.
Table 5: Ablation study on pickup and delivery with time windows.
Frameworks
SOLVEm @
QYT
Gemin1-2.5-Pro
0.6514
0.6170
# of Demos/
QYI
HSEvo
# of Feedback Rounds
0,5000
0.4491
Dat al.
2025
5/10
0.4196
Rv
3/10
0.2829
0,4771
0.4486
Ye ct al.
2024)
0/10
0.2351
EoH
5/5
0.3330
0.4954
0,4492
Liu et al.
2024b)
5/1
0.2350
Errer Importämer
Gemini-2.5-Flash
Propurtion
Syntaxtarer
Rurkimetrror
Yield (Y)
0.6
0.6
DeepSeek-V3 (Y)
Cleude-3.7 (Y)
04
TimcostBrarr
Verificattantirrr
Claude (Q)
Gemini 2.5 Flash Flass (Q)
(D)
DeepSeek-V3 (Q)
aprat mict
Deeplank-st
Glade Soudee--7--smnne 7-5mnnet
Qwen3-22 BespheikoV
Loaks, Maveettl
---355
LLaMe-3.3) 3-700
0.2
0.2
0.5
Tem Temperature
Figure 3: Error classifications.
Figure 4: Quality- Yield tradeoff.
To identify common failure modes, we analyze and categorize the most common error types produced by the evaluated models, as shown in Fig. 3 These include: (1) Hallucinated APIs: using nonexistent or outdated library calls. (2) Incorrect algorithmic logic: flawed implementation even when the
general approach is reasonable. (3) Constraint misunderstanding: ignoring or misinterpreting problem constraints. (4) Timeouts: no output or the execution time exceeds the given constraints. Additional
error cases and examples are listed in Appendix E.8
5.2 ABLATION STUDY
To assess the robustness and sensitivity of LLM performance under different settings, conduct set of ablation experiments with full details in Appendix E
Temperature. We evaluate three representative models across the QYI spectrum using temperatures TE {0.0.0.5 1.0]. Fig. shows that higher T increases diversity and quality but lowers yield due to more invalid outputs ($ E.4). Greedy decoding (T 0) has maximum yield with suboptimal quality, while stochastic sampling (7 = 1) achieves better quality at the cost of solving fewer problems. This reveals fundamental trade-off between quality and yield that future L.MMs must address.
Few-shot demonstrations. We assess the impact of in-context examples by comparing zero-shot, half-shot, and full-shot prompts. Due budget constraints, these experiments are conducted on few representative models. Specifically, we evaluate Gemini-2.5 Comini-2.5-Pras Pro on the pickup and delivery problem one of the most challenging tasks our benchmark (full results in $ E.5). As shown in Table 5] providing more informative demonstrations significantly boosts the overall performance, especially for tasks involving unfamiliar domains or requiring long- nogghorizon reasoning.
Feedback rounds. To evaluate the role of iterative refinement, we vary the number of feedback rounds given to LLMMs (1, 5, and 10), keeping the temperature fixed at The results in Table 5 show that later iterations frequently fix logic errors or constraint violations from earlier attempts, underscoring the value of multi-round reasoning. We provide further analysis in $ 5.3 and E.6
5.3
CASE STUDY
We present a case study on technology mapping (Chen Cong 2004) to highlight both the promise and current limitations of LLMs, where the task is to cover a logic network with K-input lookup tables (LUTs) while minimizing their total count; we fix K = 6. As an expert baseline, we use ABC (Berkeley) 22005. a of-te-of-the-an logic synthesis tool that leverages optimized cut enumeration and dynamic programming (DP)-based covering. We find that top perforrmn LLMs, such as GPT- 04-mini-high and Gemini-2.5-Po 5-Pro, can mimic similar heuristic strategies and iteratively refine them

---

Published as a conference paper at ICLR 2026
Quality Yield
static proning Lower ylakd. sigher quality
0.8
Nalve E-UUT masping Pars alf low quality
Sooe
0.4
wl aggressine prusing Deat baitans
DP ui pruning) TimesutError
Number Iterations
Figure 5: One iterative example of GPT-o4-mini-high on the technology mapping problem.
through feedback. Fig. 5 shows GPT-o4-mini-high explores a range of approaches over multiple iterations, evolving from naive mappings to sophisticated DP-based heuristics with pruning. It finally converges on a strategy that effectively balances yield and quality, achieving the highest QYI. The full generated programs across those iterations are listed in $
Nonetheless, a substantial gap remains between L.MS and expert tools (~60% of expert performance), due to the latter's extensive use of domain-specific optimizations and efficient implementations. This suggests that while LLMs can learn and refine heuristic algorithms, they are not yet capable of generating solutions with expert-level performance in real-world complex optimization tasks.
6 DISCUSSION AND LIMITATION
To guide future LLM development, we identify two key challenges: (1) forrectness-reduciie hallucinations (e.g., incorrect API usage) and improving costuuctoonfoolooinng to strictly satisfy problem constraints; and (2) Performance iifiormancee-aviiatiinne -nvigatinn enormous search spaces to discover high- quality solutions. Potential enhancements such as longer context windows and advanced search strategies like iterative best-of-N ($ E.7 may help address these challenges.
While our benchmark and framework offer a promising foundation for evaluating LLMs on combina- torial optimization problems, several limitations remain that suggest directions for future work.
First, all experiments are run in Python, which eases adoption but incurs execution overheads. We report preliminary C++ results in $ E.9 but full integration remains challenging due to reliance domain-specific libraries and the difficulty of generating efficient, correct, and parallel C++ code.
Second, our current agentic pipeline only uses standard configuration baseline and does not incorporate advanced prompting or multi-agent strategies. There are several promising directions for improving context efficiency, such using summarization (Sahoo al. compression (Chuang et al., (024). Automatic prompt engineering further expands the design space, with approaches based on gradient descent (Pryzant et al.. (023) or genetic algorithms (Guo et al.) 2024) for searching optimal prompts. In addition, multi-agent designs in which different agents handle different components of decomposed problem have shown strong potential for tackling more complex tasks (Zhou et al. 2025 Gottweis et al. 2025).
2024) prompt
Third, the iterative self-refinement process in our agentic workflow can be interpreted as a form of test-time scaling (TTS), analogous to compute-optimal scaling strategies (Snell et al. 2024). This
perspective creates opportunities to incorporate techniques such as best-of-N sampling (Stiennon
et al. 2020), beam search (Xie et al. 2023), evolutionary algorithms (Novikov et al. [0225] Ye et al. 2024), and external autotuner (van Stein et al., (025),, especially with increased iteration budgets. Furthermore, with a robust verifier in place, our framework provides natural platform to investigate self-verification capabilities (Kumar et al. 2024). promising avenue toward greater LLM autonomy.
Finally, evaluation pipeline currently relies formally defined and computationally efficient proxy metrics. While these metrics are useful for benchmarking, they may not fully reflect real- world performance. This gap is especially apparent in (1) scientific domains, where solution quality must ultimately be validated through physical experiments, and (2) engineering domains like EDA, where quality must be confirmed through time-consuming backend synthesis. A key challenge for future research lies in bridging this gap while managing the latency introduced by longer feedback loops.
We believe HeuriGym can as shared testbed and foster interdisciplinary collaboration.
10

---

Published as a conference paper at ICLR 2026
ACKNOWLEDGEMENT
We would like to thank Aaron M. Ferber, Delia Qu, Wenting Zhao, and Cunxi Yu for their constructive feedback the initial proposal of this project. We are also grateful Anthony Agnesina and Wen- Hao Liu for providing the initial global routing benchmark, and to Niansong Zhang for generously providing OpenRouter credits. The Cornell authors are supported part by ACE, one of the seven centers in JUMP 2.0. Semiconductor Research Corporation (SRC) program sponsored by DARPA; NSF Award #21 18709; the A12050 Senior Fellowship, a Schmidt Sciences program; the National Science Foundation (NSF); the National Institute of Food and Agriculture (USDA/NIFA); and the Air Force Office of Scientific Research (AFOSR).
ETHICS STATEMENT
In the development of this work, we have been guided by the ICLR Code of Ethics, aiming to con- tribute positively to the field of combinatorial optimization and human well-being. We acknowledge that, like any new dataset, the content presented could misused. Our intention for this work is to advance the field of combinatorial optimization for societal benefit, and we encourage the research community to consider and mitigate potential negative consequences in future applications.
We recognize that the large language models utilized may reflect existing biases. By primarily using publicly available models, we aim to be transparent and have avoided using private user data. The use of LLMs is listed in Appendix We advocate for continued research into identifying and mitigating biases in LLM-crafted heuristics to ensure fairness and prevent discrimination.
REPRODUCIBILITY STATEMENT
Our framework, dataset, and benchmark are released under an open-source license and are available in the supplementary material during the review process. All models accessible via the public API through OpenRouter Onaierr(ppRRRteerrr (2025). The models included in our benchmark are listed in Appendix E.I with detailed experimental settings provided in Appendix E To support reprodueibility. we also provide step-by- step-by-step instructions in our repository.
11

---

Published as a conference paper at ICLR 2026
REFERENCES
Luca Aceto, Jens A Hansen, Anna Ingólfsdóótti,, Jacob Johnsen, and John Knudsen. The complexity of checking consistency of pedigree information and related problems. Journal of Computer Science and Technology, 19:42-59, 2004.
Divyam Aggarwal, Dhish Kumar Saxena, Thomas Bäck, and Michael Emmerich. Real-world airline crew pairing optimization: Customized genetic algorithm versus column generation method. In International Conference on Evolutionary Multi-Criterion Optimization, pp. 518-531. Springer, 2023.
Ali AhmadiTeshnizi, Wenzhi Gao, and Madeleine Udell. Optimus: scalable optimization modeling with (mi)lp solvers and large language models. In Proceedings of the 41st International Conference on Machine Learning (ICML), 2024.
Christoph Albrecht. Global routing by new approximation algorithms for multicommodity flow.
IEEE Transactions on Computer-Aided Design of Integrated Circuits and Systems, 20(5):622-632, 2001.
Alibaba. Qwen3: Think deeper, act faster, 2025. https: https://qwenlm.giie http://www....iii....././..... qwen3/
Benjamin D Allen and Stephen L Mayo. An efficient algorithm for multistate protein design based on faster. Journal of computational chemistry, 31(5):904 9164499966111 2010.
Luca Amarú, Morre-Emmanue Gaillardon, and Giovanni De Micheli. The epfl combinational benchmark suite. In Proceedings of the 24th International Workshop on Logic Synthesis (/WLS), 2015.
Xavier Ambroggio and Brian Kuhlman. Computational design of single amino acid sequence that can switch between two distinct protein folds. Journal of the American Chemical Society, 128(4): 1154-1161, 2006.
Ranga Anbil, Rajan Tanga, and Ellis L. Johnson. A global approach to crew-pairing optimization. IBM Systems Journal, 31(1):71-78, 1992.
Roberto Baldacci, Enrico Bartolini, and Aristide Mingozzi. An exact algorithm for the pickup and delivery problem with time windows. Operations research, 59(2):414-226 2011.
Jayanth R Banavar, Marek Cieplak, Amos Maritan, Gautham Nadig, Flavio Seno, and Saraswathi Vishveshwara. Structure-based design of model proteins. Proteins: Structure, Function, and Bioinformatics, 31(1):10-20, 1998.
Lalch Behjat, Anthony Vannelli, and William Rosehart. Integer linear programming models for global routing. INFORMS Journal on Computing, 18(2):137-150, 2006.
Russell Bent and Pascal Van Hentenryck. A two-stage hybrid algorithm for pickup and delivery vehicle routing problems with time windows. Computers Operations Research, 33(4):875 893, 2006.
Berkeley. ABC: A System for Sequential Synthesis and Verification, 2005. URL http:// www. www..
http: eecs www.eeey.eeee.e.oooooo.e.ee
r >alarmi/abe/
Yaohui Cai, Kaixin Yang, Chenhui Deng, Cunxi Yu, and Zhiru Zhang. Smoothe: Differentiable e-graph extraction. In Proceedings of the 30th ACM International Conference on Architecturl Support for Programming Languages and Operating Systems (ASPLOS), Volume 1, 2025.
Andrew Canis, Jongsok Choi, Mark Aldham, Victor Zhang, Ahmed Kammoona, Jason H Anderson, Stephen Brown, and Tomasz Czajkowski. Legup: high-level synthesis for fpga-based processorlac- celerator systems. Proceedings of the 19th ACM/SIGDA international symposium on Field
programmable gate arrays, pp. 33-36, 2011.
12

---

Published as a conference paper at ICLR 2026
RC Carden, Jianmin Li, and Chung- the optimal solution. IEEE Transactions on Computer-Aided Design of Integrated Circuits and Systems, 15(2):208-25) 216, 1996.
Chung-Kuan Kuan Cheng. A global router with theoretical bound on
Chong- Yun Chao and Earl Glen Whitehead. On chromatic equivalence of graphs. Theory and Applications of Graphs. pp. 121-131, 1978.
Deming Chen and Jason Cong. Daomap: A depth-optimal area optimization mapping algorithm for fpga designs. In IEEE/ACM International Conference Computer Aided Design 2004. ICCAD-2004., pp. 752-759. IEEE, 2004.
Di Chen, Yexiang Xue, Shuo Chen, Daniel Fink, and Carla Gomes. Deep multi multi-species embedding. arXiv preprint arXiv: 1609.09353 2016.
Di Chen, Yiwei Bai, Sebastian Ament, Wenting Zhao, Dan Guevarra, Lan Zhou, Bart Selman, R Bruce van Dover, John M Gregoire, and Carla P Gomes. Automating crystal bystal-structure phase mapping by combining deep learning with constraint reasoning. Nature Machine Intelligence, (9):812-822, 2021a.
Hongzheng Chen and Minghua Shen. ddee-reinforeemen--eemingg based scheduler for fpga hls. In 2019 IEEE/ACM International Conference on Computer-Aided Design (ICCAD), pp. 1-8. IEEE, 2019.
Hongzheng Chen, Cody Hao Yu, Shuai Zheng, Zhen Zhang, Zhiru Zhang, and Yida Wang. Slapo: A schedule language for progressive optimization of large deep learning model training. In Proceedings of the 29th ACM International Conference on Architectural Support for Programming Languages and Operating Systems, Volume 2. pp. 1095-1111, 2024a.
Hongzheng Chen, Niansong Zhang. Shaojie Xiang, Zhichen Zeng. Mengjia Dai, and Zhiru Zhang Allo: A programming model for composable accelerator design. Proceedings of the ACM on Programming Languages, 8(PLDI):593 620, 2024b.
Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde De Oliveira Pinto, Jared Kaplan, Harri Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, et al. Evaluating large language models trained on code. arXiv preprint arXiv:2107.03374 2021b.
Jianyi Cheng, Samuel Coward, Lorenzo Chelini, Rafael Barbalho, and Theo Drane. Seer: Super- optimization explorer for high-level synthesis using e-graph rewriting. Int'l Conf. on Architectuaal Support for Programming Languages and Operating Systems (ASPLOS), pp. 1029-1044, 2024.
Wei-Lin Chiang, Lianmin Zheng, Ying Sheng. Anastasios N. Angelopoulos, Tianle Li, Dacheng Li, Banghua Zhu, Hao Zhang, Michael I. Jordan, Joseph E. Gonzalez, and lon Stoica. Chatbot arena: an open platform for evaluating Ilms by human preference. Proceedings of the 41st International Conference Machine Learning (ICML), 2024.
Yu-Neng Chuang, Tianwei Xing, Chia- Yuan Chang, Zirui Liu, Xun Chen, and Xia Hu. Learning to compress prompt in natural language formats. In Kevin Duh, Helena Gomez, and Steven Bethard (eds.), Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume I: Long Papers), pp. 7756-7767, Mexico City, Mexico, June 2024. Association for Computational Linguis- tics. doi: 10. 10.186533v1/222 120p///2222l21ll/////// 429. URL http://///ccaaatoooooooooog//// naac1-long 4297
China Graduate Mathematical Modeling Competition. Problem f, 2021.
Jason Cong and Yuzheng Ding. Flowmap: An optimal technology mapping algorithm for delay optimization in lookup-table based fpga designs. IEEE Transactions on Computer-Aided Design of Integrated Circuits and Systems, 13(1):1-12, 1994.
Jason Cong and Patrick Madden. Performance-driver global routing for standard cell design. In Proceedings of the 1998 International Symposium on Physical Design, pp. 73-78, 1998.
Jason Cong and Zhiru Zhang. An efficient and versatile scheduling algorithm based on sde formulation.
In P Proceedings of the 43rd annual Design Automation Conference (DAC), pp. 433-438, 2006.
13

---

Published as a conference paper at ICLR 2026
Jason Cong, Bin Liu, Stephen Neuendorffer, Juanjo Noguera, Kees Vissers, and Zhiru Zhang. High- level synthesis for fpgas: From prototyping to deployment. IEEE Transactions on Computer-Aide Design of Integrated Circuits and Systems, 30(4):473 2011. doi: 10.1109/TCAD.201
21 2100592.
Timothy Curtois, Dario Landa-Silva, Yi Qu, and Wasakorn Laesanklang. Large neighbourhood search with adaptive guided ejection search for the pickup and delivery problem with time windows. EURO Journal on Transportation and Logistics, 7(2):151-122, 2018.
Steve Dai, Gai Liu, and Zhiru Zhang. scalable approach exact resource- conosree-onntrrineed scheduling based on joint sdc and sat formulation. In Proceedings of the 2018 ACM/SIGDA International Symposium on Field- Progranmable Gate Arrays, pp. 137-146, 2018.
Pham Vu Tuan Dat, Long Doan, and Huynh Thi Thanh Binh. Hsevo: Elevating automatic heuristic design with diversity-driven harmony search and genetic algorithm using llms. In The 39th Annual AAAI Conference on Artificial Intelligence, 2025. https://github com/ catphamvn/
HSEVO
Protein Database. Resb protein data bank (resb pdb), 2024.
Justas Dauparas, Gyu Rie Lee, Robert Pecoraro, Linna An, Ivan Anishchenko, Cameron Glasscock, and David Baker. Atomic context-conditioned protein sequence design using ligandmpon. Nature Methods, pp. 1-7, 2025.
Google 2025.
DeepMind.
Gemini 2.5: Our most intelligent
model, nd/
https://bleg www.gooooooooo.oooo ogy/ o gemi ni -wo-mmomdlliiiieee tnokeell-tiinkkinn-updddtte ma ma 2025
Guy Desaulniers, Jacques Desrosiers, and Marius M Solomon. Column generation, volume 5. Springer Science & Business Media, 2006.
JM Deutsch and Tanya Kurosky. New algorithm for protein design. Physical review letters, 76(2): 323, 1996.
Ken A Dill, Sarina Bromberg, Kaizhi Yue, Hue Sun Chan, Klaus M Ftebig, David P Yee, and Paul D Thomas. Principles of protein folding-a perspective from simple exact models. Protein science, 4(4):561-602, 1995.
Qingxiu Dong, Lei Li, Damai Dai, Ce Zheng, Jingyuan Ma, Rui Li, Heming Xia, Jingjing Xu, Zhiyong Wu, Baobao Chang, Xu Sun, Lei Li, and Zhifang Sui. A survey on in-context learning. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, 2024.
K Eric Drexler. Molecular engineering: An approach to the development of general capabilities for molecular manipulation. Proceedings of the National Academy of Sciences, 58755552555 1981.
Jiangsu Du, Jinhui Wei, Jiazhi Jiang, Shenggan Cheng, Dan Huang. Zhiguang Chen, and Yutong Lu. Liger: Interleaving intra-and inter-operator parallelism for distributed large model inference. In Proceedings of the 29th ACM SIGPLAN Annual Symposium on Principles and Practice of Parallel Programmingg pp. 42 42-54,270 2024a.
Yuanqi Du, Arian R Jamasb, Jeff Guo, Tianfan Fu, Charles Harris, Yingheng Wang, Chenru Duan, Pietro Liò, Philippe Schwaller, and Tom Blundell. Machine learning- aided generative molecular design. Nature Machine Intelligence, 6(6):589-66 604899664.666 2024b.
Yvan Dumas, Jacques Desrosiers, and Francois Soumis. The pickup and delivery problem with time windows. European journal of operational research, 54(1):7-23 1991.
Issmail Elhallaoui, Daniel Villeneuve, François Soumis, and Guy Desaulniers. Dynamic aggregation of set-partitioning constraints in column generation. Operations Research, 53(4):632-645, 2005.
Lizhou Fan, Wenyue Hua, Lingyao Li, Haoyang Ling, and Yongfeng Zhang. NPHardEval: Dynamic benchmark on reasoning ability of large language models via complexity classes. In Wei Ku, Andre Martins, and Vivek Srikumar (eds.), Proceedings of the 62nd Anmual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), 2024.
14

---

Published as a conference paper at ICLR 2026
Amir Kafshdar Goharshady, Chun Kit Lam, and Lionel Parreaux. Fast and optimal extraction for sparse equality graphs. Proceedings of the ACM on Programming Languages, 8(0OPSLA2): 2551-2577, 2024.
Juraj Gottweis, Wei-Hung Weng, Alexander Daryin, Tao Tu, Anil Palepu, Petar Sirkovic, Artiom Myaskovsky, Felix Weissenberger, Keran Rong, Ryutaro Tanno, al. Towards an co-scientist. arXiv preprint arXiv:2502. /8864, 2025.
Glenn W Graves, Richard D McBride, Ira Gershkoff, Diane Anderson, and Deepa Mahidhara. Flight crew scheduling. Management science, 396))733-774, 1993.
Qingyan Guo, Rui Wang, Junliang Guo, Bei Li, Kaitao Song, Xu Tan, Guoqing Liu, Jiang Bian, and Yujiu Yang. Connecting large language models with evolutionary algorithms yields powerful prompt optimizers. In The Twelfth International Conference on Learning Representations, 2024.
URL BBKLLtttp:://peneeveee... net htt.:/eww..nn/toommmoooo/CCCaAa
Gurobi.
Gurobi optimizer, 2025,
htts:://www.uuett com/ mtom/soluue solut itutsiionss
Mark A Hallen and Bruce R Donald. Comets (constrained optimization of multistate energies by tree search): A provable and efficient protein design algorithm to optimize binding affinity and specificity with respect sequence. Journal of Computational Biology, 23(5):311-321, 2016.
Mark C Hansen, Hakan Yalcin, and John P Hayes, Unveiling the iscas-85 benchmarks: A case study in reverse engineering. IEEE Design & Test of Computers, 1999.
William E Hart. On the computational complexity of sequence design problems. In Proceedings of the first annual international conference on Computational molecular biology, pp. 128-136, 1997.
Dan Hendrycks. Collin Burns, Saurav Kadavath, Akul Arora, Steven Basart, Eric Tang, Dawn Song. and Jacob Steinhardt. Measuring mathematical problem solving with the MATH dataset. In Thirty-fifth Conference on Neural Information Processing Systems Datasets and Benchmarks Track (Round 2), 2021.
Sin C Ho, Wai Yuen Szeto, Yong-Hong Kuo, Janny MY Leung, Matthew Petering, and Terence WH Tou. survey of dial-a-ride problems: Literature review and recent developments. Transportation Research Part B: Methodological, 1 11:3995-42, 2018.
Matthew Hofmann, Berk Gokmen, and Zhiru Zhang. Eqmap: Fpga lut remapping using e-graphs. In 2025 IEEE/ACM International Conference on Computer-Aided Design (ICCAD), 2025.
Jiang Hu and Sachin S Sapatnekar. A survey on multi-net global routing for integrated circuits. Integration, 31(1): 2001.
Edward Hughes, Michael D Dennis, Jack Parker-Holder Feryal Behbahani, Aditi Mavalankar, Yuge Shi. Tom Schaul, and Tim Rocktäschel. Position: Open-endedness is essential for artificial super- human intelligence. In Proceedings of the 41st International Conference on Machine Learning. 2024.
C-T Hwang, J-H Lee, and Y-C Hsu. A formal approach to the scheduling problem in high level synthesis. IEEE Transactions on Computer-Aided Design of Integrated Circuits and Systems, 10 00::464475. 1991.
Zangir Iklassov, Yali Du, Farkhad Akimov, and Martin Takáč. Self-guiding exploration for combi- natorial problems. The Thirty-eighth Annual Conference on Neural Information Processing Systems (NeurIPS), 2024. URL http:pppww.eveeeeeeeeerrrrrrr/ea
Yuki Imajuku, Kohki Horie, Yoichi Iwata, Kensho Aoki, Naohiro Takahashi, and Takuya Akiba. ALE-bench: A benchmark for long-horizon objective-driven algorithm engineering. In The Thirty- ninth Annual Conference on Neural Information Processing Systems Datasets and Benchmarks
Track, 2025. /tl.:tttpp..//pperreieeee.eeeee (at
15

---

Published as a conference paper at ICLR 2026
Naman Jain, King Han, Alex Gu, Wen-Ding Li, Fanjia Yan, Tianjun Zhang, Sida Wang. Armando Lezar-L Koushik Sen, and lon Stoica. Livecodebench: Holistic and contamination free evaluation of large language models for code. In The Thirteenth International Conference Learning Representations (ICIR), 2025.
Jeppesen. Jeppesen crew pairing solution.
com/ 2021.
airline- crew
sowut
Caigao JIANG, Xiang Shu, Hong Qian, Xingyu Lu, JUN ZHOU, Aimin Zhou, and Yang Yu. Learning to define and solve general optimization problems from scratch. In The Thirteenth International Conference on Learning Representations (ICLR), 2025. URL https!
LLMOPT:
//openreview. net. oeteenrewiaaa.ee.fooooooooooooo
Carlos E Jimenez, John Yang, Alexander Wettig, Shunyu Yao, Kexin Pei, Ofir Press, and Karthik R Narasimhan. SWE-bench: Can language models resolve real-world github issues? In The Twelfth International Conference on Learning Representations, 2024. URL Jattps://operreviee
net feet/ooxee Eooummidd VTF 8yNQM66
John Jumper, Richard Evans, Alexander Pritzel, Tim Green, Michael Figurnov, Olaf Ronneberger, Kathryn Tunyasuvunakool, Russ Bates, Augustin Židek, Anna Potapenko, et al. Highly accurate protein structure prediction with alphafold. nature, 596(7873):583-55)))
2021.
Satwik Kamtekar, Jarad M Schiffer, Huayu Xiong, Jennifer M Babik, and Michael H Hecht. Protein design by binary patterning of polar and nonpolar amino acids. Science, 262(5140):1880-1665 1993.
Atoosa Kasirzadeh, Mohammed Saddoune, and François Soumis. Airline crew scheduling: models, algorithms, and data sets. EURO Journal on Transportation and Logistics, 6(2):111-137, 2017.
Jon M Kleinberg. Efficient algorithms for protein sequence design and the analysis of certain evolutionary fitness landscapes. In Proceedings of the third annual international conference on Computational molecular biology, pp. 226-237, 1999.
Johanna P Korte and Neil Yorke-Smith. An aircraft and schedule integrated approach to crew scheduling for point-to-point airline. Journal of Air Transport Management, 124:102755 2025.
Michael Krumdick, Charles Lovering, Varshini Reddy, Seth Ebner, and Chris Tanner. No free labels: Limitations of llm-as-a-judge without human grounding. arXiv preprint arXiv:2503.0506), 2025.
Aviral Kumar, Vincent Zhuang, Rishabh Agarwal, Su, John Co-Reyes, Avi Singh, Kate Baumli, Shariq Iqbal, Colton Bishop, Rebecca Roelofs, et al. Training language models to self-correct via reinforcement learning. arXiv preprint arXiv:2409.12911 2024.
Avery Laird, Bangtian Liu, NIKOLAJ BJØRNER, and Maryam Mehri Dehnavi. Speq: Translation of sparse codes using equivalences. ACM SIGPLAN Conf. on Programming Language Design and Implementation (PLDI). 2024.
Kit Fun Lau and Ken A Dill. Theory for protein mutability and biogenesis. Proceedings of the National Academy of Sciences, 87(2):638-642, 1990.
Chin Yang Lee. An algorithm for path connections and its applications. IRE transactions on electronic computers, BC-10(3):346335 365, 2009.
Dmitry Lepikhin, HyoukJoong Lee, Yuanzhong Xu, Dehao Chen, Orhan Firat, Yanping Huang, Maxim Krikun, Noam Shazeer, and Zhifeng Chen. Gshard: Scaling giant models with conditional computation and automatic sharding. arXiv preprint arXiv:2006.166 16668, 2020.
Haibing Li and Andrew Lim. A metaheuristic for the pickup and delivery problem with time windows. In Proceedings 13th IEEE international conference tools with artificial intelligence, pp. 160-167. IEEE, 2001.
Sirui Li, Janardhan Kulkarni, Ishai Menache, Cathy Wu, and Beibin Li. Towards foundation models for mixed integer linear programming. In The Thirteenth International Conference Learning Representations (ICLR), 2025. URL pettps://openeee net mere/foorumii.
6YENDA7J4G
16

---

Published as a conference paper at ICLR 2026
Yujia Li, David Choi, Junyoung Chung, Nate Kushman, Julian Schrittwieser, Rémi Leblond, Tom Eccles, James Keeling, Felix Gimeno, Agustin Dal Lago, et al. Competition-level code generation with alphacode. Science, 10916666))110221999 2022.
Rongjian Liang, Anthony Agnesina, Wen-Hao Liu, and Haoxing Ren. Gpu/ml-enhanced large scale global routing contest. In Proceedings of the 2024 International Symposium on Physical Design (ISPD), 2024.
Rongjian Liang, Anthony Agnesina, Wen-Hao Liu, Matt Liberty, Hsin- Tzu Chang, and Haoxing Ren. Invited: Ispd 2025 performance-driver large scale global routing contest. In Proceedings of the 2025 International Symposium on Physical Design, ISPD '25 pp. 252-256, New York. NY, USA, 2025. Association for Computing Machinery. ISBN 07984007129377 doi: 10.1145/3698344 3715706. UR//http:://dd...or//....0 1tt.l/or//....11/388881 3715700
Haiguang Liao, Wentai Zhang, Xuliang Dong, Barnabas Poczos, Kenji Shimada, and Levent Bu- rak Kara. A deep reinforcement learning approach for global routing. Journal of Mechanical Design, 142(6):06171,, 2020.
Zeming Lin, Halil Akin, Roshan Rao, Brian Hie, Zhongkai Zhu, Wenting Lu, Nikita Smetanin, Robert Verkuil, Ori Kabeli, Yaniv Shmueli, et al. Evolutionary-scale prediction of atomic-level protein structure with a language model. Science, 379(6637):1123-111 2023.
Aixin Liu, Bei Feng, Bing Xue, Bingxuan Wang, Bochao Wu, Chengda Lu, Chenggang Zhao, Chengqi Deng, Chenyu Zhang, Chong Ruan, et al. Deepseek-v3 technical report. arXiv preprint arXiv:2412.19437. 2024a.
Fei Liu, Xialiang Tong, Mingxuan Yuan, Xi Lin, Fu Luo, Zhenkun Wang. Zhichao Lu, and Qingfu Zhang. Evolution of heuristics: towards efficient automatic algorithm design using large language model. In Proceedings of the 41st International Conference on Machine Learning (ICML), 2024b.
Fei Liu, Rui Zhang, Zhuoliang Xie, Rui Sun, Kai Li, Xi Lin, Zhenkun Wang, Zhichao Lu, and Qingfu Zhang. Llm4ad: A platform for algorithm design with large language model. ArXiv, 2024c. URL
attps://eet //apsiv///xvv... org/ wtt. /21. 17287
Jiachang Liu, Dinghan Shen, Yizhe Zhang, Bill Dolan, Lawrence Carin, and Weizhu Chen. What makes good in-context examples for GPT-3? In Proceedings of Deep Learning Inside Out (DeeL1O 2022): The 3rd Workshop on Knowledge Extraction and Integration for Deep Learning Architectures, 2022.
Jiawei Liu, Chunqiu Steven Xia, Yuyao Wang, and Lingming Zhang. Is your code generated by chatGPT really correct? rigorous evaluation of large language models for code generation. In Thirty-seventh Conference Neural Information Processing Systems, 2023. URL. https //openreview.o net. wtwp:r/wwweeeeeeeeoorummmmmm
quuu0 OCu7
Mingju Liu, Yingjie Li, Jiaqi Yin, Zhiru Zhang, and Cunxi Yu. Differentiable combinatorial schedul- ing at scale. In Proceedings of the 41st International Conference Machine Learning (ICML), 2024d.
Panta Lučié and Dušan Teodorović. Metaheuristics approach to the aircrew rostering problem. Annals of Operations Research, 155:311-338, 2007.
Jacob Lundgren. Neuro-symbolic ai for pedigree analysis: neuro-symbolic question answering approach for pedigree analysis in hereditary cancer risk assessment, 2025.
Cristian Micheletti, Jayanth Banavar, Amos Maritan, and Flavio Seno. Protein structures and optimal folding from geometrical variational principle. Physical Review Letters, 82(16)33372, 1999.
Yaosen Min. Ye Wei, Peizhuo Wang, Xiaoting Wang, Han Li, Nian Wu, Stefan Bauer, Shuxin Zheng, Yu Shi, Yingheng Wang, et al. From static to dynamic structures: Improving binding affinity prediction with gased deep learning model. arXiv e-prints, pp. arXiv-2208, 2022.
17

---

Published as a conference paper at ICLR 2026
Alan Mishchenko, Satrajit Chatterjee. and Robert Brayton. Improvements to technology mapping for lut-based fpgas. In Proceedings of the 2006 ACM/SIGDA 14th International Symposium on Field Programmable Gate Arrays (FPGA), 2006.
Michael D Moffitt. Global routing revisited. In Proceedings the 2009 International Conference on Computer-Aided Design, pp. 805-808, 2009.
the
Michael D. Moffitt and Pratik Fegade. The asplos 2025 / eurosys 2025 contest intra-operator parallelism for distributed deep learning. In Proceedings of the 30th ACM International Conference on Architectural Support for Programming Languages and Operating Systems (ASPLOS), volume of ASPLOS 2025, 2025.
Christopher Negron and Amy E Keating, Multistate protein design using clever and classy. In Methods in enzymology, volume 523. pp. 171-190. Elsevier, 2013.
Greg Nelson and Derek C Oppen. Simplification by cooperating decision procedures. ACM Transac- tions on Programming Languages and Systems, 1(2):445 257, 1979.
Alexander Novikov, Ngån Vu, Marvin Eisenberger, Emilien Dupont, Po-Sen Huang, Adam Zsolt Wagner, Sergey Shirobokov, Borislav Kozlovskii, Francisco JR Ruiz, Abbas Mehrabian, al. Alphaevolve: A coding agent for scientific and algorithmic discovery. Technical report, Technical report, Google DeepMind, 05 2025. URL hups::/storage, googleapis .... 2025.
AoPS Online. American invitational mathematics examination (aime), 2025. https
//artofproblemsolllee ccm wtc.://ndee..ppp/mmmrrccaaa
Mathematics stt.tiiii_x__xmma_iiiioo
OpenAl. Introducing openai o3 and o4-mini, 2025.
index//
Lntroducing----andd--e
OpenRouter. Llm rankings, 2025.
Julian Oppermann, Andreas Koch, Melanie CeuueerOpperramnt and Oliver Sinnen. Ilp-based modulo scheduling for high-level synthesis. In Proceedings of the International Conference on Compilers, Architectures and Synthesis for Embedded Systems, pp. 1-10, 2016
Anne Ouyang, Simon Guo, Simran Arora, Alex I.. Zhang, William Hu, Christopher Ré, and Azalia Mirhoseini. Kernelbench: Can llms write efficient gpu kernels?, 2025. URI. htRlhtttp:://// pattts://arxiis
org/abs/2502. 10517
Debjit Pal, Yi-Hsiang Lai, Shaojie Xiang, Niansong Zhang. Hongzheng Chen, Jeremy Casas, Pasquale Cocchini, Zhenkun Yang, Jin Yang, Louis-Nos Pouchet, et al. Accelerator design with decoupled hardware customizations: benefits and challenges. In Proceedings of the 59th ACM/IEEE Design
Automation Conference. pp. 1351-1354 2022.
Pavel Panchekha, Alex Sanchez- James Wilcox, and Zachary Tatlock. Automatically improving accuracy for floating point expressions. ACM SIGPLAN Conf. on Programming Language Design and Implementation (PLDI), 50(6):1-11, 2015.
Alice C Parker, Jorge Pizarro, and Mitch Mlinar. Maha: A program for datapath synthesis. In 23rd ACM/IEEE Design Automation Conference, pp. 461-466. IEBE, 1986.
Pierre G Paulin and John Knight. Force-directed scheduling for the behavioral synthesis of asics. IEFE Transactions on Computer-Aided Design of Integrated Circuits and Systems, 8(6):661-679, 2002.
Long Phan, Alice Gatti, Ziwen Han, Nathaniel Li, Josephina Hu, Hugh Zhang, Chen Bo Calvin Zhang, Mohamed Shaaban, John Ling, Sean Shi, et al. Humanity's last exam. arXiv preprint arXiv:2501.14249, 2025.
Navin Pokala and Tracy M Handel. Energy functions for protein design: adjustment with protein- protein complex affinities, models for the unfolded state, and negative design of solubility and specificity. Journal of molecular biology, 347(1):203 2005.
18

---

Published as a conference paper at ICLR 2026
Reid Pryzant, Dan Iter, Jerry Li, Yin Lee, Chenguang Zhu, and Michael Zeng. Automatic prompt optimization with "gradient descent" " and beam search. In Houda Bouamor, Juan Pino, and Kalika Empirical Methods in Natural Language
Bali (eds.), Proceedings of the 2023 Conference Processing, pp. 7957-7968, Singapore, December 2023. Association for Computational Linguss- URL httpps::/aclaatthoollooooooo
tics. doi: 10. (0.18663/112223.emmmmmmm
emm emr.p-mmiin. 4947
main.
Samyam Rajbhandari, Jeff Rasley, Olatunji Ruwase, and Yuxiong He. Zero: Memory optimizations toward training trillion parameter models. In SC20: International Conference for High Performance Computing, Networking, Storage and Analysis, pp. 1-16. IEEE, 2020.
Rindranirina Ramamonjison, Timothy Yu, Raymond Li, Haley Li, Giuseppe Carenini, Bissan Ghaddar, Shiqi He, Mahdi Mostajabdaveh, Amin BamitalbiiDehkorrdi, Zirui Zhou, et al. N14opt competition: Formulating optimization problems based on their natural language descriptions. In NeurIPS 2022 Competition Track, pp. 189-203 203- PMLR, 2023.
David Rein, Betty Li Hou, Asa Cooper Stickland, Jackson Petty, Richard Yuanzhe Pang, Julien Dirani, Julian Michael, and Samuel R. Bowman. GPQA: A graduate-level google-proof q&a benchmark. In First Conference on Language Modeling, 2024.
Julia Robinson. On the Hamiltonian game (a traveling salesman problem). Rand Corporation, 1949.
Bernardino Romera-Paredess Mohammadamin Barekatain, Alexander Novikov, Matej Balog, M Pawan Kumar, Emilien Dupont, Francisco JR Ruiz, Jordan S Ellenberg. Pengming Wang. Omar Fawzi, et al. Mathematical discoveries from program search with large language models. Nature, 625(7995)4468 475, 2024.
Stefan Ropke and Jean- GranFFraçoois Cordeau. Branch and cut and price for the pickup and delivery problem with time windows. Transportation science, 43(3):267-266, 2009.
Stefan Ropke and David Pisinger. An adaptive large neighborhood search heuristic for the pickup and delivery problem with time windows. Transportution science, 40(4))455 2006.
Stefan Ropke, Jean-François Cordeau, and Gilbert Laporte. Models and branch-and-eut algorithms for pickup and delivery problems with time windows. Networks: An International Journal, 49(4): 258-272, 2007.
Sabre. Sabre crew pairing. efficient_ops, 2020.
Pranab Sahoo, Ayush Kumar Singh, Sriparna Saha, Vinija Jain, Samrat Mondal, and Aman Chadha. A systematic survey of prompt engineering in large language models: Techniques and applications. arXiv preprint arXiv:240..07927, 2024.
Marti Sanchez, Simon de Givry, and Thomas Schiex. Mendelian error detection in complex pedigrees using weighted constraint satisfaction techniques. Constraints, 13(1):130-155 154, 2008
Carlo S Sartori and Luciana S Buriol. study on the pickup and delivery problem with time windows: Matheuristics and new instances. Computers & Operations Research, 124: 124:105065, 2020.
Thomas Schaefer. The complexity of satisfiability problems. In Proceedings of the tenth annual ACM symposium on Theory of computing, pp. 216-226, 1978.
Timo Schick, Jane Dwivedi-Yo Yu, Roberto Dessí, Roberta Raileanu, Maria Lomeli, Eric Hambro, Luke Zettlemoyer, Nicola Cancedda, and Thomas Scialom. Toolformer: language models can teach themselves to use tools. In Proceedings of the 37th International Conference Neural Information Processing Systems (NeurIPS), 2023.
Thomas Schiex. Cost function httess//foorrelaa.a.na..ttaoooo
cost- ionstt-foatti---ibrbaa 2018.
Eugene Shakhnovich and AM Gutin. A new approach to the design of stable proteins. Protein Fingineering, Design and Selection, 6(8):793-800, 1993.
19

---

Published as a conference paper at ICLR 2026
Minghua Shen, Hongzheng Chen, and Nong Xiao. Entropy-directe scheduling for fpga high-level synthesis. IEEE Transactions on Computer- Design of Integrated Circuits and Systems, 39 (10):2588 2601, 2019.
Ziji Shi, Le Jiang, Ang Wang, Jie Zhang. Xianyan Jia, Yong Li, Chenean Wu, Jialin Li, and Wei Lin. Tap: Efficient derivation of tensor parallel plans for large neural networks. In Architecture and System Support for Transformer Models (ASSYST@ ISCA), 2023.
Eugene Shragowitz and Lynn Keel. A multicommodity flow approach to concurrent global routing. In Proceedings of the 24th ACM/TEEE Design Automation Conference, pp. 414-419, 1987.
Shivalika Singh, Yiyang Nan, Alex Wang, Daniel D' Souza, Sayash Kapoor, Ahmet Üstün, Sanmi Koyejo, Yuntian Deng, Shayne Longpre, Noah Smith, et al. The leaderboard illusion. arXiv preprint arXiv2250..20877 2025.
Gus Henry Smith, Zachary D Sisco, Thanawat Techaumnuaiwit, Jingtao Xia, Vishal Canumalla, Andrew Cheung, Zachary Tatlock, Chandrakana Nandi, and Jonathan Balkind. There and back again: A netlist's s tale with much egraphin'. arXiv preprint arXiv:2404.00770 2024.
Charlie Snell, Jaehoon Lee, Kelvin Xu, and Aviral Kumar. Scaling Ilm test-time compute optimally can be more effective than scaling model parameters. arXiv preprint arXiv:2408.033/ 2024.
Nadia Souai and Jacques Teghem. Genetic algorithm based approach for the integrated airline crew-pairing and rostering problem. European Journal of Operational Research, 199(3):674-683, 2009.
JIRI Soukup. Fast maze router. In Design Automation Conference, pp. 100-101, IEEE Computer Society, 1978.
Michael B. Stepp. Equality Saturation: Engineering Challenges and Applications. PhD thesis, University of California San Diego 2011. URL
com
docview/911049104/1
Nisan Stiennon, Long Ouyang, Jeffrey Wu, Daniel Ziegler, Ryan Lowe, Chelsea Voss, Alec Radford, Dario Amodei, and Paul F Christiano. Learning to summarize with human feedback. Advances in neural information processing systems, 33:3008-3021, 2020.
Shaojian Sun, Rachel Brem, Hue Sun Chan, and Ken A Dill. Designing amino acid sequences fold with good hydrophobic cores. Protein Engineering, Design and Selection, 1(12):205-11213, 1995.
Weiwei Sun, Shengyu Feng. Shanda Li, and Yiming Yang. Co-bench: Benchmarking language model agents in algorithm search for combinatorial optimization. ArXiv, abs/2504.04310,, 2025. URL
battps://arxx
wwt.//tgg///oooooo//tt///t..ae/t / 2504 04310
Jianheng Tang, Qifan Zhang, Yuhan Li, Nuo Chen, and Jia Li. Grapharena: Evaluating and exploring large language models on graph computation. In The Thirteenth International Conference on Learning Representations (ICLR), 2025. URL httpt://t//c..........t /ttp////eeeeeeeeeeeee/eeee net. www.forrmmim did= /
YIr9yCMzeA
E Taniguchi, T Yamada, M Tamaishi, and M Noritake. Effects of designated time on pickup/delivery truck routing and scheduling. WIT Transactions on The Built Environment, 36, 2025.
Samuel Thomas and James Bornholt. Automatic generation of vectorizing compilers for customizable digital signal processors. Int'l Conf. on Architectural Support for Programming Languages and Operating Systems (ASPLOS), pp. 19-34, 2024.
Ecenur Ustun, Ismail San, Jiaqi Yin, Cunxi Yu, and Zhiru Zhang. Impress: Large integer multiplica- tion expression rewriting for fpga hls. IEEE Symp. on Field Programmable Custom Computing Machines (FCCM), pp. 1-10, 2022.
Van Rijsbergen. Information retrieval: theory and practice. In Proceedings of the joint IBM/Uni- versity of Newcastle upon tyne seminar data base systems, volume 79, pp. 1-14, 1979.
20

---

Published as a conference paper at ICLR 2026
Niki van Stein, Diederick Vermetten, and Thomas Bäck. In-the-loop hyper-parameter optimization for basdd automated design of heuristics. ACM Trans. Evol. Learn. Optim., April 2025. ISSN doi: 10.1 145/3731567, 567. Just Accepted.
www.cxy..ibbaar.cor..oo. cornell
Alexa VanHattum, Rachit Nigam, Vincent T James Bornholt, and Adrian Sampson. Vectorization for digital signal processors via equality saturation. Ini'l Conf. on Architectural Support for Programming Languages and Operating Systems (ASPLOS), pp. 874-886, 2021.
Anthony Vannelli. An adaptation of the interior point method for solving the global routing problem.
IEEE transactions on computer-aidee aided design of integrated circuits and systems, 10(2):193-22) 203, 2002.
Jelena Vucinic, David Simoncini, Manon Ruffini, Sophie Barbe, and Thomas Schiex. Positive multistate protein design. Bioinformatics, 36(1):122-130, 2020.
Gang Wang, Wenrui Gong, Brian DeRenzi, and Ryan Kastner. Ant colony optimizations for resource- and constaicon operation scheduling. IEEE Transactions on Computer-Aided Design of Integrated Circuits and Systems, 2007.
Huzefa Rangwala. Long-context protein language model. bioRxiv, pp. 2024-10, 2024. Yingheng Wang, Zichen Wang, Gil Sadeh, Luca Zancato, Alessandro Achille, George Karypis, and
Yisu Remy Wang, Shana Hutchison, Jonathan Leang, Bill Howe, and Dan Suciu. Spores: Sum- product optimization via relational equality saturation for large scale linear algebra. Int'l Conf. on Very Large Data Bases (VLDB). 13(11), 2020.
Joseph L Watson, David Juergens, Nathaniel R Bennett, Brian L Trippe, Jason Yim, Helen E Eisenach, Woody Ahern, Andrew Borst, Robert J Ragotte, Lukas F Milles, et al. De novo design of protein structure and function with rfdiffusion. Nature, 620(7976):1089- 1100, 2023.
Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Fei Xia, Ed Chi, Quoc V Le, Denny Zhou, et al. Chain-of- prompting elicits reasoning in large language models. Advances in neural information processing systems, 35:24824 24837, 2022.
Ellen M Wijsman. The role of large pedigrees in an era of high-throughput sequencing. Human genetics, 131:1555-156,, 2012. 2012.
Max Willsey, Chandrakana Nandi, Yisu Remy Wang, Oliver Flatt, Zachary Tatlock, and Pavel Panchekha. egg: Fast and extensible equality saturation. Proc. ACM Program. Lang., 2021.
David H Wolpert and William G Macready. No free lunch theorems for optimization. JEEE transactions on evolutionary computation, 1(1):67 82, 1997.
Zhiyong Wu, Yaoxiang Wang, Jiacheng Ye, and Lingpeng Kong. Self-adaptive in-context learning: An information compression perspective for in-context example selection and ordering. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), 2023.
Ziyang Xiao, Dongxiang Zhang, Yangjun Wu, Lilin Xu, Yuan Jessica Wang, Xiongwei Han, Xiaojin Fu, Tao Zhong, Jia Zeng, Mingli Song, al. Chain-of-experts: When Ilms meet complex operations research problems. In The twelfth international conference on learning representations (ICLR), 2023.
Yuxi Xie, Kenji Kawaguchi, Yiran Zhao, James Xu Zhao, Min-Yen Kan, Junxian He, and Michael Xie. Self-evaluation guided beam search for reasoning, Advances in Neural Information Processing Systems, 36:41618-416500 2023.
AMD Xilinx. Vitis high-level synthesis (hls), 2025.
com htt.m//mccccmddee//d/ctttttee html
Yassine Yaakoubi, François Soumis, and Simon Lacoste-Julien. Machine learning in airline crew pairing to construct initial clusters for dynamic constraint aggregation. EURO Journal on Trans- portation and Logistics, 140:000020 2020.
21

---

Published as a conference paper at ICLR 2026
Chengrun Yang. Xuezhi Wang, Yifeng Lu, Hanxiao Liu, Quoe V Le, Denny Zhou, and Xinyun Chen. Large language models as optimizers. In The Twelfth International Conference on Learning Repre- sentations (ICLR), 2024. URL httptttts////peree///
Yichen Yang, Phitchaya Phothilimthana, Yisu Wang, Max Willsey, Sudip Roy, and Jacques Pienaar. Equality saturation for tensor graph superoptimization. Conf. on Machine Learning and Systems (MLSys), 3:255-268. 2021.
Chen Yanover, Menachem Fromer, and Julia M Shifman. Dead-end elimination for multistate protein design. Journal of Computational Chemistry, 28(13)212 2007.
Shunyu Yao, Jeffrey Zhao, Dian Yu, Nan Du, Izhak Shafran, Karthik R Narasimhan, and Yuan Cao. React: Synergizing reasoning and acting in language models. In The Eleventh International Conference Learning Representations, 2023.
Shunyu Yao, Noah Shinn, Pedram Razavi, and Karthik R Narasimhan. (SitauS) (aauuS)-beune -bench: benchmark for ( Jool-lundellee ( A uundetuunndrneeeeee ser interaction in real-world domains, In The Thirteenth International Conference on Learning Representations (ICLR), 2025.
Haoran Ye, Jiarui Wang. Zhiguang Cao, Federico Berto, Chuanbo Hua, Haeyeon Kim, Jinkyoo Park, and Guojie Song. Reevo: Large language models as hyper-heuristics with reflective evolution. In Advances in Neural Information Processing Systems (NeurIPS), 2024.
Kaizhi Yue and Ken A Dill. Forces of tertiary structural organization in globular proteins. Proceedings of the National Academy of Sciences, 92(1):146-150, 1995.
Xuanchang Zhang, Wei Xiong, Lichang Chen, Tianyi Zhou, Heng Huang, and Tong Zhang. From lists to emojis: How format bias affects model alignment. arXiv preprint 1rXiv:2409.11704, 2024.
Yihong Zhang. The e-graph extraction problem is np-complete. https://dffetttyyeeemsbblgggggh extraction.html, 2023.
Wenting Zhao, Nan Jiang, Celine Lee, Justin Chiu, Claire Cardie, Matthias Gallé, and Alexander M Rush. Commitò: Library generation from scratch. In The Thirteenth International Conference Learning Representations (ICLR), 2025.
Yanli Zhao, Andrew Gu, Rohan Varma, Liang Luo, Chien-Chin Huang, Min Xu, Less Wright, Hamid Shojanazeri, Myle Ott, Sam Shleifer, et al. Pytorch fsdp: experiences on scaling fully sharded data parallel. arXiv preprint arXiv:2304. /177, 2023.
Lianmin Zheng, Zhuohan Li, Hao Zhang. Yonghao Zhuang. Zhifeng Chen, Yanping Huang, Yida Wang, Yuanzhong Xu, Danyang Zhuo, Eric Xing, et al. Alpa: Automating inter-and {Intra- Operator) parallelism for distributed deep learning. In 16th USENIX Symposium Operating Systems Design and Implementation (OSDI 22), pp. 559-578, 2022.
Lianmin Zheng, Wii-Lin Chiang. Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric Xing, et al. Judging ilm-as-a-judge with mt-bench and chatbot arena. Advances in Neural Information Processing Systems (NeurIPS), 2023.
Han Zhou, Xingchen Wan, Ruoxi Sun, Hamid Palangi, Shariq Iqbal, Ivan Vulić, Anna Korhonen, and Sercan Ö Ank. Multi-agent design: Optimizing agents with better prompts and topologics. arXiv preprint wwXXi:25022.05333 2025.
Terry Yue Zhuo, Vu Minh Chien, Jenny Chim, Han Hu, Wenhao Yu, Ratnadira Widyasari, Imam Nur Bani Yusuf, Haolan Zhan, Junda He, Indraneil Paul, Simon Brunner, Chen GONG, James Hoang. Armel Randy Zebaze, Xiaoheng Hong, Wen-Ding Li. Jean Kaddour, Ming Xu, Zhihan Zhang, Prateek Yadav, Naman Jain, Alex Gu, Zhoujun Cheng. Jiawei Liu, Qian Liu, Zijian Wang, David Lo. Binyuan Hui, Niklas Muennighoff, Daniel Fried, Xiaoning Du, Harm de Vries, and Leandro Von Werra. Bigcodebench: Benchmarking code generation with diverse function calls and complex instructions. In The Thirteenth International Conference on Learning Representations (ICLR), 2025.
22

---

Published as a conference paper at ICLR 2026
APPENDIX
A The Use of Large Language Models (LLMs)
24
Updated Results in February 2026
24
Prompt Design
25
System Prompt
25
C.2
User Prompt
26
Prompts for Improvement Guidance
27
C.4
Refinement Prompt for Problem Descriptions
28
Example Problem Description
28
Problem Set
29
Operator Scheduling
29
D.2
Technology Mapping
29
D.3
Global Routing
30
D.4
E-Graph Extraction
30
D.5
Intra-Operator Parallelism
31
D.6
Protein Sequence Design
31
D.7
Mendelian Error Detection
32
D.8
Airline Crew Pairing
32
D.9
Pickup and Delivery Problem with Time Windows
32
Additional Experiments
33
Models
33
E.2
Experimental Settings
................................
33
Detailed Results on Each Problem
34
E,4
Ablation Temperature
36
Few-Shot Demonstration
36
E,6
Feedback Rounds
37
Iterative Best-of-N Sampling
37
E.8
Error Analysis
37
C++ Example
38
E. 10 Token Usage
39
Detailed Analysis of Case Study
39
Datasets
54
23

---

Published as a conference paper at ICLR 2026
A THE USE OF LARGE LANGUAGE MODELS (LLMS)
In this work, we design an agentic benchmark and evaluate the performance various L.MSs on combinatorial optimization problems. Model specifications
provided in Appendix E.1 and We ensure that all uses of LLMs are transparent, and our results can be fully reproduced from the submitted supplementary material. Beyond their use in experiments, we also employ LLMs to polish the writing. Importantly, we do not use LLMs to propose ideas design experiments.
experimental details are described in Appendix [
B UPDATED RESULTS IN FEBRUARY 2026
Gamini- 3-rr
OPT a-Coee
-
0.8
Quality
0.6
Weighted
0.6
kini-i2.5
Genini
Klml-X2.5
0.4
Qwinn 3.5-3978
0.4
MiniMax--2555
0.2
0.0
0.2
0.4
0.6
0.8
1.0
Yield
Estimated API Cost official pricing)
Figure 6: QYI and estimated API cost of the latest ten models in February 2026.
Table 6: Model performance the latest ten frontier models in February 2026.
Model
QYI
SOLVEII @5 (Yield)
SOLVEn @5
SOLVE, @
Quality
Gemini-3.1-Pro
0.878
91.7%
91.7%
100.0%
0.947
GPT-5.3-Codex
0.788
93.6%
100.0%
100.0 %
0.773
GLM-5
0.776
80.3%
93.6%
100.0% %
0.849
Claude- -uudee-puus-466
0,750
83.9%
99.5%
100.0% %
0.849
Gemini-3-Flaan
0.682
78.0%
87.6%
100.0% %
0.760
Grok-4
0.652
63.3%
78.4%
98.2%
0,868
Kimi-K2.5
0.593
71.6%
88.5%
100.0%
0.730
0.578
68.3%
88.1%
100.0% %
0,709
DeepSeek -3.2
0.307
33.9%
84.9%
100.0% %
0.626
M2.55
0.300
33.9%
62.4%
83.0%
0.681
We extend our evaluation to a broader set of frontier models released prior to February 2026, as summarized in Table Given the substantial performance improvements observed across models 2025, we limit each model to five iterative attempts in this experiment to generate alternative solutions. All models evaluated under the same timeout configuration described Appendix E.2
As shown in Table and Fig. 6 Gemini-3.1-Por leads all ten models with a weighted QYI of 0.878, followed by GPT-5.3-Codex. GLM-5, and Claude-Opus-44 forming a top tier separated from the remaining six models. Overall, these models demonstrate substantially stronger performance than those evaluated in May 2025 (Fig. 2). indicating marked improvement in frontier capability over the past year. The left panel of Fig. 6 shows that these four models cluster in the high-yield, high-quality region, near the QYI=0.8 contour, indicating a balanced tradeoff between coverage and solution performance. In contrast, Gemini-3-Flash, Grok-4, Kimi-K2.5, and Dwen3.5-391 and Minimax-M2.5 rank last by a substantial margin. The right panel of Fig. further contextualizes these results by incorporating estimated API cost. Although Gemini-3. 1-Pro achieves the highest QYI, it is also among the most expensive models, while SPT-5.33Codde. and GLM-5 achieve competitive QYI at moderately lower cost. Notably, GLM-5 lies close to the Pareto frontier in the cost-QY! plane, offering near-top-tien performance at substantially lower price than Gemini-3.1-Pro.
constitute mid tier, while DeepSeek-VS
24

---

Published as a conference paper at ICLR 2026
Table Model specifications with API names and official pricing (per million tokens).
Organization
Model
API Name
Price (SIn/$Out)
Google
Gemini-3.1 1-PPro
google/geminn--.111ee
2.00/12.00
OpenAl
GPT-5.3-Codex
cpt-5.3-codex
1.75/14.00
Anthropic
Claude-Opus-4.6
claude-opus-4-6
5.00/25.00
Zhipu AI
GLM-5
z-ai/glm-5
1.00/3.20
Google
Gemini-3-Flash
google//genii----flla-------eree
0.50/3.00
XAI
Grok-4
x-ai/grok-4
2.00/15.00
Moonshot AI
Kimi-K2.5
moonshotal/kimi-k2..
0.60/3 00
Alibaba
Qwen3.5-399
cwen3.5-397b-al1
0.55/3.50
Minimax
MinimaxMMa
minimax/minimax--ne m2
0.30/1.20 200
DeepSeck
DeepSeek-Ve V3.2
dcepseck-v3.2
0.28/0.42
The ordering by SOLVEm @5 partially diverges from weighted QYI: GPT-5.3-Codex achieves the highest coverage 93.6% (204/218), placing it above Gemini-3.1-Pro (91.7%). yet Gemini-3311PPP leads on weighted QYI because its solutions are of substantially higher quality when they are valid. Technology mapping illustrates this tradeoff sharply: GPT-5.3-Codex achieves 100% yield but at solution quality of only 0.253, whereas Gemini-3. 1-Pro covers 94% of cases at quality 0.937. Grok-4 represents the most extreme quality-over-coverag trade-off, achieving the highest average quality among models outside the top four (0.868) but the second-lowest yield (0.628), pulling its weighted QYI to 0.652 despite near-optimal solutions on problems it does solve.
Performance varies widely across problems, revealing a clear difficulty gradient. Operator scheduling, protein sequence design, e-graph extraction, and intra-op parallelism are the most tractable: nine or ten models achieve 100% SOLVEm @5 on cach, with DeepSeek -..oe being the sole exception on operator scheduling, where it solves only of 24 cases despite producing syntactically valid code in every attempt. At the opposite extreme, pickup delivery with time windows is the most challenging problem: only Gemini-3. 1-rro achieves full coverage (30/30 valid), six models score 0% despite universally reaching the Stage II, and no other model exceeds 53% coverage. Global routing exhibits a sharp bimodal pattern: Claude- Claude-Opus-44 Opus-4.6,
HPT-5.3-Codex and GLM-5 achieve 16-100% SOLVEn @ 5, while Kimi-K2.55 DeepSeek V3.2, Minimax-Mar M2.5, and Gemini-Flash each score 0%, with the failure occurring entirely at the ourput-format stage.
Failure mode analysis via the SOLVE, @ 5-to--OLVEE @ gap reveals qualitatively different break- down patterns across models: that nearly all code produces satisfies constraints when it runs; DeepSeek-V V3.2 2 loses 68.1 points on average, reflecting structural tendency to generate solvers that execute correctly but violate prob- lem constraints. Iterative refinement provides the largest gains between SOLVEm @ and SOLVEm @5: Gemini-3.1-PP 's SOLVEH @i pass rate rises from 24.8% to 87.6% over the first three iterations, 63-point gain attributable to foedback-driven correction, with only modest additional improvement to 91. .7%% by iteration five. In contrast, DeepSeek- -3.. V3.2 plateaus after three iterations, suggesting its failures reflect systematic algorithmic deficiencies that iterative prompting alone cannot resolve.
loses only 5.2 percentage points average, indicating
C PROMPT DESIGN
In this section, we detail the system and user prompts used by the LLM agent, as well as the auxiliary prompt employed to enhance our problem descriptions.
C.1 SYSTEM PROMPT
Each iteration of our benchmark begins with a task-agnostic system prompt that instructs the LLM to generate and iteratively refine executable heuristics for combinatorial optimization problems. This system prompt is followed by a task-specific problem statement and an input/output specification. The prompt includes placeholders highlighted in red that dynamically instantiated runtime for each task. For instance, (NUM CPU CORES) represents the CPU core limit for the task (default: 8), and (TIMEOUT) specifies the wall-clock time limit (default: 10 seconds).
25

---

Published as a conference paper at ICLR 2026
System Prompt
You are a world-class optimization expert and algorithmic problem solver. Your task is to develop a highly efficient solution to the following optimization problem. Please analyze the problem background, mathematical formulation, and I/O specifications with extreme rigor and attention to detail.
Your mission is to devise and implement the most performant algorithm possible, optimizing for both computational efficiency and solution quality. You should leverage your deep knowledge of algorithms, data structures, and optimization techniques to craft a powerful solution. You have complete freedom in your algorithmic approach. Think systematically and creatively. Your goal is to push the boundaries of what's possible within the computational constraints. Please strictly follow the instructions below.
1. A problem template is provided below. You only need implement the solve function. Do NOT modify the function signature including the data types of the input arguments. You are free use any data structures or algorithms within this function, but please make sure you have imported necessary libraries and modules, and defined required classes.
2. The evaluation machine has (NUM CPU CORES) CPU cores and sufficient mem- ory to run your program. The time limit for this question is (TIMEOUT) seconds. You are free to implement parallel algorithms where appropriate to maximize perfor- mance.
3. The Python version is 3.12. You may use any standard Python libraries and only the following third- party libraries:
numpy=2.2.55
networkx==3.4.2
pandas=2.22.3
4. Your response should consist of complete implementation of the 'solve' func- tion. Do NOT include any explanations, comments, additional text, or Markdown formatting.
5. You will receive execution feedback after the your program, including runtime metrics and correctness evaluation.
C.2 USER PROMPT
For each problem, the first iteration begins with the following user prompt, which introduces the task and its objective the LLM, along with program template that the model is expected complete.
User Prompt
# Problem Information (PROBLEM DESCRIPTION)
# Program Template
def solve iipuu_Fill: str, solution_file; file: str): тnp
Solve the optimization problem.
Please do NOT change the function name and arguments. Inputs should be read from input_file and outputs should be written to solution_ file. Input and output formats have been specified in the problem statement. u.*
raise NotImp.lemeteddEroor
"This is a placeholder implementation you need to fill in."
26

---

Published as a conference paper at ICLR 2026
C.3 PROMPTS FOR IMPROVEMENT GUIDANCE
Based on the feasibility of the final outputs, we issuc one of two improvement prompts in subsequent iterations. If any test cases fail, we provide the following prompt:
Improvement Guidance Case
# Feedback from Previous Iteration (Iteration (iteration These are the test cases and results from the previous iteration: ## Test Case I: (test_name)
**Input F
|content] * **Result***
(execution_message
## Test Case 2: [test_name]
**In File:* {content] **Result:**
inrecution_messagee
Improvement Guidance
The program failed to produce valid solutions for some test cases, Please fix the following issues:
1. Check for compilation
runtime exceptions.
2. Ensure the program handles all edge cases and meets the problem constraints correctly.
3. Verify that the input and output format match the expected format.
4. Make all required functions implemented correctly, and external forbid- den libraries are used.
5. If the program not able to produce valid solutions for any test case, please try to find the root cause and fix it.
6. If the program is able to produce valid solutions for some test cases, please try to improve the solution.
Otherwise, if all test cases pass verification, we issue the following prompt:
Improvement Guidance Case
Feedback from Previous Iteration (Iteration (iteration-1)
# Improvement Guidance
Please carefully observe problem structure and improve upon this program by:
1. Addressing any weaknesses in the previous approach.
2. Introducing more advanced efficient algorithms.
3. Focusing improving performance for test cases.
Your goal is to improve the solution for as many test cases as possible, with special attention those where the previous solution performed poorly.
27

---

Published as a conference paper at ICLR 2026
C.4 REFINEMENT PROMPT FOR PROBLEM DESCRIPTIONS
To ensure clarity and correctness in problem specification, we employ a human-in-the-loop process. Specifically, prompt weaker LLM flag any unclear ambiguous statements in the task description. The following prompt is used for this purpose:
Refinement Prompt for Problem Descriptions
If you were solve the programming task below, do you have any questions? Is there anything I should clarify before you begin writing code?
# Problem Description
[PROBLEM DESCRIPTION)
C.5 EXAMPLE PROBLEM DESCRIPTION
The following provides an example problem description for operator scheduling, For other problems, please refer our repository.
# Background
nigh-level level syathesis (BLS) an important stage in electronic design auzomation (EDA), simed tronslating high-level program specification (e.g., written in c/C+ BystemC) into cycle accurate hardwace implementation, After the program is parsed and analyzed, typically cransformed into intermediate representation known Control Dota Flow Graph (CDFG). This graph caprures the operations (e.g., arithmetic, memory accesses) and their control/data dependencies. The CDFG can further proceasec into Directed Acyelic Graph (D8G) to facilitate scheduling
optimizs.ion.
One of the challenges ILS is operater scheduLing, which determines the exact control - step for cycle) at which each operation is executed, while satisfying data dependencies constraints. gefinient scheduling plays critical rola optimizing + dealgn quality in terns of performance, area, and power.
and
# permalization
Conaider CDFG with $nš operation nodes $o_1s, where 51 Vin 0 \t1 2, Aidoss, n\Is, and precedence relation Siprecs on sos that
opecation dependencies. - $a_18 is asscciated with cycle delay $1_i Via \machbb(2)*45 and a rescurce type $r_i Vin R ME, 2, \idots, kMs,, Let $T M0, 1, 2, Vidots, LVJS represent the aet of (t_l, t_2, Vidots, t_m) where $t_i Vin rş denotes the star: time (c-step) of operation $o_i$.
operation
control steps (c-steps), and define schedule an SgS-tuple
A schedule 953 is feasible it satisfies all data dependencies: $\ferall i, vin
\pree \Righterrow t_i di \leg t_jš.
Let $58 denote the set of all fessible schedules, For a given schedule 6s8, let (tL be the number of operations that
5rš in contral step sts, and defina the total + usage of resource žr$ as $N_r - \sun_lt it Vin TF {t)ss),
Given a bound $6_r$ on the number of available instances for each resource type &r Min Rs, the operator scheduling problem - the overall latency $14, defined Simin_la Min 51 \max_lt Vin (t_ d_i)s,
find fessible schedule 85 mininizes
subject to the resource constraints R,
$\forall
(L) Vleg
Iaput
The input is provided in JSoN format with the follewing structure:
.*json
"name": "input",
"delay":
"mul"; 3,
"sub":
"rescurce":
"mul*: 2,
"sub":
"nodes": ['n'*, "mul"], [*n2*, "mui*), ["a3*, "sup*]
28

---

Published as a conference paper at ICLR 2026
"edges": [ Mal", "n3", "ine"), (Ma2%, "a3", "chs"]
"aame": Name of the input greph
"delay'; Kapa
resource Lype execution delay cycles "resouroe" Maps each resource type to the number of avsi functional unita represented "[node_id, essurce_type]] "dgee": 11at of edges, where each edge la represented as [ssurre_node,, target_node, edge_name]
"addes":
nodes,
Outpuc Format
The output should provide the ixecution achedule of the program, indicating the start eycle each operation. example, following output neans "nl" and start at 4 cycle 0, while "n3" starts at eycle 3:
n1:0
n2:0
n3:3
D PROBLEM SET
In this section, we provide more details on the problems included in Table For representative problem description used in the prompts, please consult our repository for additional details.
D.1 OPERATOR SCHEDULING
Operator scheduling is a critical stage in high-level synthesis (HLS) (Cong et al. 2011 Pal et al. 2022), the process of converting behavioral hardware descriptions into register-transfer transfer level (RTL) implementations. This task involves carefully assigning each operation to specific clock cycle while managing variety of constraints such as data dependencies, resource availability, and performance targets. The effectiveness of the scheduling process is vital, as it directly influences key design metrics including area, power consumption, and execution time, making it important focus in the field of electronic design automation (EDA).
Over the years, researchers have developed wide range of tethhiiquuu tackle the inherent challenges of operator scheduling in HLS. Exact methods, such as those based on integer linear programming (ILP) (Hwang al., 1991 Oppermann et al. (016), can provide optimal solutions but often suffer from scalability issues. As a result, many commercial and academic HLS tools (Xilinx 2025 Canis
et al. 201T) rely on heuristics achieve practical, near-optimal results. Traditional heuristic ap- proaches, including priority- fentir-funnttootnbnee based methods (Shen et al. 2019 Parker et al. 1986 Paulin Knight 2002). focus on balancing resource utilization with performance requirements. Notably, methods leveraging systems of difference constraints (SDC) enable efficient formulation that captures a rich set scheduling restrictions and casts the optimization objective into linear pro- gramming (LP) framework (Cong & Zhang 2006 Dai al.. 2018). More recently, the incorporation of machine learning techniques (Chen & Shen 2019; Liu et al. (024d) has further advanced the state-of-the-ane enhancing both scheduling efficiency and solution quality in the face of increasingly complex hardware designs.
D.2 TECHNOLOGG MAPPING
Technology mapping, in the context of logic synthesis for integrated circuits and field- programmable gate arrays (FPGAs), is the process of converting a logic network into an equivalent network of standard cells logic resources from specific technology library. The objective is to optimize key design metrics such as area, delay, and power consumption. a crucial step in the VLSI design flow and FPGA design flow, determining the actual physical implementation of design.
Here in our problem setting, we focus on area-optimal technology mapping for lookup table (LUT)- based FPGAs. Given input logic network, the goal is cover the network with K-input subgraphs,
29

---

Published as a conference paper at ICLR 2026
each of which can be implemented by a K-LUT, while minimizing the number of LUT's representing the circuit area.
The most widely adopted approaches are cut-based methods, which operate in two stages: cut enumeration and cut selection. In this approach, all feasible K -iinutt cuts-i.e., subgraphs with at most K inputs-are enumerated for each node in the boolean network. Then, a dynamic programming- based selection process chooses one cut per node to construct a full LUT cover of the circuit, optimizing for metrics such as area or delay (Chen & Cong 2004 Cong & Ding 1994, Mishchenko et al. 2006). A refinement of this approach is known as priority cut pruning, which retains only limited set of the most promising cuts per node rather than considering all possible cuts. This significantly improves scalability for large circuits and is widely implemented in tools such as ABC (Berkeley, 2005).
D.3 GLOBAL ROUTING
The global routing problem addresses the challenge of planning signal paths across a chip after logic placement, determining how a set of nets should traverse the layout to ensure connectivity while reserving space for detailed routing. Rather than producing exact wire geometries, global routing generates abstract paths through routing regions. This step must account for routing congestion, layer limitations, and timing criticality, while managing a growing number of nets modern designs like Very-Large-Scanl Integration (VLSI). The quality of the global routing solution plays critical role in determining the feasibility and effectiveness of downstream routing stages and can ultimately dictate the success or failure of physical design closure.
The problem has been studied extensively via sequential and II. II.--ased methods. Maze routing, introduced by Lee (2009), laid the groundwork for sequential approaches, with subsequent improve- ments such as the work by Soukup (1978). For multi-terminal nets, rectilinear Steiner tree methods were developed (Cong & Madden 1998). However, sequential routing lacks global coordination and often leads to congestion. ILP--base methods formulate routing as a 0-1 programming, concurrently optimizing over all nets with objectives like wire length and capacity constraints. While exact ILP solvers are computationally intensive, relaxation techniques such as randomized rounding (Carden) al. 1996) and multi-commodity network flow models (Shragowitz Keel, 1987 ( | (001))
have been employed. Interior-poin methods for solving the LP relaxation (Vannelli 2002 Behjat
ct al. 2006) have also proven effective for scalable and ncar-optimal routing.
Hu & Sapatnekar ) provided a comprchensive survey of global routing techniques for integrated circuits. MMffiit 2009) revisited the problem, offering a historical perspective and highlighting key open challenges that remain unresolved. More recently, to foster the development of advanced global routing methods, Liang et al. (2024) 2025) introduced an ISPD contest that encourages the use of GPU-based techniques to accelerate global routing.
D.4 E-GRAPH EXTRACTION
E-graph (Chao & Whitehead 1978 Nelson & Oppen 1979) is data structure that compactly represents set of expressions. Given an input program and set of rewrite rules, an e-graph is constructed by applying the rules to the program, generating new expressions, and merging equivalent expressions. It has been widely used to represent and explore the huge number of equivalent program space in tensor graph transformation (Yang et al. 2021 Chen al. (024b), sparse linear algebra optimization (Wang et al. (2020), code optimization (Laird et al. SSith et al. (024),, digital
signal processor (DSP) compilation /VanHattum et al. 2021 1 Thomas & Bornholt, 2024), circuit
datapath synthesis (Ustun et al. 2022 Cheng et al. 2024) and floating-pon -oainn arithmetic (Panchekhal ct al. 2015).
In e-graph, all functionally equivalent terms are organized in the same equivalent classes, known as e-classes. Nodes within each e-class that represent values or operators are called e-nodes. E-classes a partition of e-nodes, where each e-node belongs to exactly one e-class. Dependencies in e-graphs are directed, which point from e-nodes to their children e-classes, indicating the operator (e-node) requires the values (e-nodes) from the child e-classes to compute its value.
In e-graph extraction, an optimized term from an e-graph is extracted after rewrites, based on a user- defined cost model. The goal is to produce a functionally equivalent but improved implementation of
30

---

Published as a conference paper at ICLR 2026
the original input program. The e-graph extraction problem is proven to be NP-hard when common sub-expressions are considered (Stepp) 2011 Zhang 2023
Existing e-graph extraction methods include exact methods employing ILP (Cheng et al. 2024 Smith et al. 2024). Recently, there has been significant progress in employing heuristics for e- graph extraction. These include simple working- list method (Panchekha al., (013). a relaxation method utilizing gradient descent (Cai et al.) 2025), and specialized method tailored for sparse e-graphs (Goharshady al. 2024). The dataset used in evaluation for this work primarily comes from SmoothE (Cai et al. 2025).
D.5 INTRA-OPERATOR PARALLELISM
Intra- Sara--Operato Parallelism (IOPDDL), an emerging challenge introduced in the ASPLOS' contest track (Moffitt Fegade, 2025), addresses the complexities of distributed deep learning. Leading teams in this competition have predominantly employed metaheuristic approaches, distinguished by their unique pre-pro precceseessng and optimization strategies.
The effective distribution of large machine learning models across multiple hardware accelerators paramount for achieving desired performance in both training and serving applications (Zhnng
et al. ctt al. communication overhead, process made intricate by the vast number of operations and tensors involved. Specifically, for a given graph where nodes represent operations with distinct execution strategies (each possessing associated cost and memory usage), optimal strategy must be chosen for every node. The objective is to minimize the aggregate sum of node and edge costs, without exceeding a strict memory usage constraint across all devices at any point. The inherent diversity in topological and memory characteristics of ML models across varied tasks and modalities renders this problem especially demanding.
2022, Zhao et al. 2023) (Shi et al.] 2023 Rajbhandari et al.) 2020; Lepikhin et al.) 2020 Du
2024a| Chen et al. 2024a). This task necessitates sharding the computation graph to minimize
D.6 PROTEIN SEQUENCE DESIGN
Understanding how proteins fold into their native three-dimensional structures (Jumper al. 202
Watson et al. 2023 is central problem in structural biology (Min et al. 2022 Du et al.) (024bb). traditionally framed as a forward problem: predicting the structure a given amino acid sequence will adopt (Lin et al. 2023 Wang et al. 2024). In contrast, the protein sequence design or inverse folding problem starts from fixed target structure and seeks sequences that are likely fold into Many works have shown that this inverse formulation not only offers practical applications in protein
engineering but also deepens our understanding of sequence-structure relationships (Drexler 1981
Yue & Dill 1995 Shakhnovich & Gutin 1993 Deutsch & Kurosky 1996 Sun et al. 1995 Lau &
Dill 1990).
A common modeling approach treats sequence design as global optimization problem over the space of amino acid sequences. Methods developed by [Sun et al. (1995), Shakhnovich & Gutin (1993). and others define fitness function to select sequences with favorable folding properties. These functions are designed to balance positive design (low free energy in the target structure) with negative design (high energy in competing folds), promoting both thermodynamic stability and structural specificity. More recently, people have been working on multi-state design with more or less general fitness
functions (Pokala & Handel 2005 Ambroggio & Kuhlman, 2006; Allen & Mayo 2010 Negron &
Keating 2013; Yanover et al. 22007] Hallen & Donald 2016 Vucinic et al. 2020)
In our benchmark, we focus on the Grand Canonical (GC) model (Sun et al. 1995) of protein
sequence design. The GC model operates on (i) detailed diree-dimensiona geometric representation of target structure with n residues, (ii) a simplified binary alphabet distinguishing only hydrophobic (H) and polar (P) residues, and (iii) fitness function & that favors sequences with densely packed hydrophobic cores while penalizing solvent-exposed hydrophobic residues. Despite its simplicity, the H/P model has been shown to capture key qualitative features of real protein structures (Dill et al. 1995 Kamtekar et al. 1993). Several studies (Micheletti et al., 1999; Banavar et al.) 1998) have explored the correspondence between sequences optimized under the GC model and those observed in natural proteins. However, a key obstacle has remained: computing an optimal sequence for a given structure is computationally challenging. The brute-force enumeration over all 2n H/P
31

---

Published as a conference paper at ICLR 2026
sequences is infeasible for realistic protein sizes, and the algorithmic complexity of the problem was explicitly raised as an open question Haar (1997). An efficient algorithm that constructs an optimal sequence in polynomial runtime was introduced later (Kleinberg 1999) using network flow.
D.7 MENDELIAN ERROR DETECTION
Chromosomes encode individual's genetic information, with each gene occupying specific position known as a locus. At each locus, a diploid organism carries two alleles- one inherited from each parent-forming its genotype. When direct genotyping is not available, researchers rely on the observable traits or phenotypes, which represent sets of compatible genotypes. A group of related individuals, along with their phenotypes at locus, is organized into pedigree, where each individual is either a founder or has parents defined within the structure.
Due to experimental and human errors, pedigree data may contain inaccuracies. These errors classified as either parental errors (incorrect parentage, which we assume do not occur here) or phenotype errors, which can lead Mendelian errors. A Mendelian error arises when all genotype combinations compatible with observed phenotypes violate Mendel's i law that each individual inherits one allele from each parent. Detecting such inconsistencies computationally challenging; the number of possible genotype combinations grows exponentially with pedigree size, making full enumeration impractical. fact, verifying consistency has been shown be NP-complete (Aceto) et al.. 2004).
Error detection and correction are crucial for downstream tasks like genetic mapping or disease gene localization. However, existing tools are often limited by scalability issues, strong assumptions, or incomplete analysis. To address these limitations, soft constraint network framework for detecting Mendelian inconsistencies of required corrections, and suggesting optimal modifications. These problems naturally align with weighted constraint satisfaction and provide rich testbed for scalable and flexible inference in large, complex pedigrees.
proposed (Sanchez et al. 2008), estimating the minimum number
D.8 AIRLINE CREW PAIRING
The airline crew pairing problem is a well-established topic in operations research. involves constructing sequences of flight legs-known as pairings-that begin and end at crew base, cover all scheduled flights, and satisfy a variety regulatory and contractual constraints. The primary goal is to minimize total crew-related costs, such wages, hotel accommodations, and deadhead travel, while ensuring legality and operational feasibility. This problem is typically formulated set partitioning model and addressed using column generation and branch-and-price techniques (Desaulniers et al., 2006 Kasirzadeh et al. 2017). Foundational systems developed for carriers like American Airlines demonstrated the effectiveness of these methods scale (Anbil et al.) 1992). More recent innovations include dynamic constraint aggregation (Elhallaoui et al., 2005) and machine learning- based pairing generation (Yaakoubi al., 2020), which are now integral to commercial solvers such as Jeppesen (Jeppesen| (2021) and Sabre (Sabre 2020), capable of processing monthly schedules with tens of thousands of flights.
In addition to exact methods, heuristic and metaheuristic techniques such as genetic algorithms, simulated annealing, and local search have been explored improve scalability and reduce com- putation time, particularly for medium-sized instances disruption recovery (Lučié & Teodorović 2007; Souai & Teghem] 2009). These hybrid approaches aim to complement exact optimization methods by leveraging historical data and incorporating planner preferences, offering more flexible and adaptive solutions in practice.
D.9 PICKUP AND DELIVERY PROBLEM WITH TIME WINDOWS
The Pick-up and Delivery Problem with Time Windows (PDPTW), originally proposed by Dumas et al. (199]), is generalized from a classical NP-hard combinatorial optimization problem- Capacitated Vehicle Routing Problem (CVRP). introduces additional complexity through precedence constraints, requiring pick-up locations to precede corresponding drop-off locations, and service time windows at each location. The problem can be seen in many logistic and public transportation systems, with the primary objective of minimizing the total travel cost.
32

---

Published as a conference paper at ICLR 2026
Over the past three decades, a wide range of models and algorithms have been proposed address the PDPTW, with most falling into the category of heuristic or metaheuristic approaches. Prominent works include simulated annealing (Li & Lim 2001 Bent & Van Hentenryck] 2006), large neighborhood search (Curtois et al., 2018; Ropke & Pisinger 2006), and iterated local search (Sartori & Buriol 2020). In contrast, research into exact solution methods has been relatively limited, with the most effective approaches relying on the set partitioning formulation combined with the branch-cut-ande price algorithm (Ropke & Cordeau, 2009, Baldacci al., 201T). Ropke al. (2007) provided comprehensive survey of PDPTWW solvers developed up to 2007. Ho et al. (2018) later reviewed more recent advancements up to 2018, with particular emphasis on PDPTW variants for people transportation, referred to as the Dial-a-Ride problem. TTniguuhi et al. (2255) develop a mathematical model and applies heuristics (Genetic Algorithm, Simulated Annealing, Tabu Search) to analyze how time-window constraints affect urban pickup/delivery truck routing and scheduling.
To support algorithm development, several benchmark datasets have been created and maintained. The Li and Lim dataset (Li & Lim (001)) is widely used and includes instances ranging from 100 to 1000 locations. More recently, Sartori Buriol (2000) released larger-scale dataset generated from real-world spatial-tempora distributions.
E ADDITIONAL EXPERIMENTS
In this section, we provide more experimental results and analysis our benchmark.
E.
MODELS
The LLMs used in our experiments are listed in Table 5 All models were accessed via official APIs provided by their respective organizations, except for the Meta models, which accessed through the OpenRouter Router 2025) API.
Table Model specifications with API names and official pricing.
Organization
Model
API Name o4-mini-high
Price (SIm/SOut)
Type
OpenAl
Gii---hmin
1.1/4.4
Reasoning
Anthropic
Claude-3.7-Sonnet
2oaudo-37-somme-22225000
3/15
Reasoning
DeepSeek
DeepSeek-VV V3
dccpseck-char(0224)
0.27/1.10
Base
DeepSeck
DeepSeek-R1
deepsock-reasoner
0.55/2.19
Reasoning
Google
Gemini-2.5-Flaa
gemini-2 5-1eash--rreview-0--177
0.15/3.5
Reasoning
Google
Gemini-2.5-44 2.5-Pro
gemini-2.55 2.5- 5-pro-preview-055-06
1.25/10.0
Reasoning
Meta
LLaMA-3.3
meta-llama// Ll 3--a....-70B-Bnstrree
0.07/0.33
Base
Meta
LLaMA-4-Maveriik
meta Illamal.laamaa
Maverick- -
278-1288--Instruu
0.27/0.85
Base
Alibaba
Qwen3-235B
qwen3-2550-a22b
0.29/2.86
Reasoning
E.2 EXPERIMENTAL SETTINGS
By default, we constrain LLMs generate Python code for each problem and execute the code on CPU server, with each instance allocated 8 CPU cores. The timeout for each problem is specified in T
Table Timeout for each problem.
Problem
Timeout (sec)
Operator scheduling
Technology mapping
10
Global routing
300
E-graph extraction
10
Intra-op parallelism
60
Protein sequence design
10
Mendelian error detection
10
Airline crew pairing
Pickup and delivery w/ time windows
60
33

---

Published as a conference paper at ICLR 2026
E.3 DETAILED RESULTS ON EACH PROBLEM
We provide the detailed SOLVE, @i values for each problem in Tables 10 th [8] The variation in SOLVE, @i across different problems highlights the diverse levels of difficulty, as summarized in Table 2 For instance, the global routing problem remains unsolved by all evaluated LLMs even for generating a single feasible solution. the case of the pickup and delivery problem. the low SOLVEm @ 10 ratio also indicates that current LLMs struggle to consistently satisfy the problem's constraints.
Table 10: SOLVE, @i results on operator scheduling problem.
SOLVEnI @
SOLVE;
Model
@ 10
@
@ 10
GB5
@
@ 10
@
@
DeepSeek-V -3
100.0% %
100,0%
4.2%
100.0%
100,0%
100.0% %
100.0%
100.0 %
100,0 %
DeepSeek-R1
100.0% %
100.0%
100.0%
100.0%
100.0%
100.0%
100.0%
100.0
100.0%
Gemini-2.5-Fiaa
100.0% %
100.0%
0.0%
100.0%
100.0%
0.0%
100.0%
100.0%
0.0%
Gemini-225-Pro
100.0%
100.0%
100.0%
100.0%
100.0%
100.0%
100.0%
100.0%
100.0%
LaaMA-4-Maveridn
20.8%
0.0%%
0.0%:
100.0%
4.2%
0.0%
100.0%
100.0%
4.2%
LLaMA-3.3
100.0% %
100,0%
100.0%
100.0%
100.0%
100.0% %
100.0%
100.0% %
100.0 %
Qwen3-2355
100.0% %
100.0%
100.0%
100.0%
100.0%
100.0% %
100.0%
100.0 %
100.0 %
Claude-3.7 ---nneet
100.0% %
100.0%
0.0%
100.0%
100.0%
0.0%
100.0%
100.0 %
0.0%
GPT-o4-mini-high
100.0%
100.0%
100.0%
100.0%
100.0%
100.0%
100.0%
100.0%
100.0%
Table 11: SOLVE, @i results on technology mapping problem.
SOLVEm @S
SOLVEn
SOLVEy
Model
@0
@
@10
@
@
@ 110
@
@ @
DeepSeek-V5
0.0%
0.0%
0.0%%
100.0%
100.0%
100.0%
100.0%
100,0%
100.0%
DeepSeek-PP R1
87.1%
87.1%
77.4%
100.0%
100.0%
100,0%
100.0%
100,0%
100.0% %
Gemini-2.5 2.5-hlash
0.0%
0.0%
0.0%
93.5%
77.4%
67.7%
100.0%
100.0%
100.0%
Gemini-2.5-Pro
74.2%
74.2%
0.0%
100.0%
100.0%
0.0%
100.0%
100,0%
0.0%
LLaMA-4-Maverics
0.0%
0.0%
0.0%
100.0 %
100.0% %
0.0%
100.0%
100,0%
0.0%
LLaMA-3.3
0.0%
0.0%
0.0%
100.0% %
100.0%
0.0%
100.0%
100.0%
6.5%
Owen3-2358
0.0%
0.0%
0.0%
100.0%
87.1
0.0%
100.0%
100.0%
3.2%
Claude-3.7 3.7-ent
87.1%
87.1%
0.0%
100.0%
100.0%
64.5%
100.0%
100.0%
100.0%
GPT-o4-mini-high
100.0%
100.0%
45.2%
100.0%
100.0%
51.6%
100.0%
100.0%
100.0%
Table 12: SOLVE, @i results on global routing problem.
SOLVEm @
SOLVEn @
SOLVE]
Model
@10
@
@ @10
@
@ 10
@
@
DeepSeek-Ve V3
0.0%
0.0%
0.0 %
33.3%
33.3%
0.0%
100,0%
100,0%
[NH,0%
DeepSeek-R1
0.0%
0.0%
0.0 %
0.0%
0.0%
0.0%
100,0%
100,0
100,0%
2emini-2.5-Flaase
0.0%
0.0%
0.0 %
20.8%
0.0%
0.0%
100,0%
100,0%
100,0%
Gemini-2.5
0.0%
0.0%
0.0%
100.0%
100.0%
0.0%
100,0%
100,0%
0.0%
-LaMe-4-Maverick
0.0%
0.0%
0.0%
0.0%
0.0%
0.0%
0.0%
0.0%
0.0%
LLaMA-3.3
0.0%
0.0%
0.0%
0.0%
0.0%
0.0%
100,0%
100,0%
4.2%
Qwen3-235B
0.0%
0.0%
0.0%
0.0%
0.0%
0.0%
100.0%
100,0
100,0 %
Claude-3.7 7-Sonnet
0.0%
0.0%
0.0%
100.0%
100.0%
0.0%
100,0%
100,0
0.0%
GPT-o4-mini-high
0.0%
0.0%
0.0%
100.0%
100.0%
100.0%
100,0%
100,0%
100,0
Table 13: SOLVE, @i results on e-graph extraction problem.
SOLVEnI @
SOLVEJ GR.5
SOLVE
Model
@
@
@
@
DeepSeek-V3
4.3%
0.0%
0.0%
10W.0%
100.0%
82.6%
100.0%
100.0%
100,0%
DeepSeek-R1
100.0
100,0%
100.0%
100.0%
100,0%
100.0% %
100.0%
100.0 %
100.0 %
Gemini-2.5-Fiag
100.0% %
100.0%
0.0%
100.0%
100.0%
0.0%
100.0%
100.0 %
0.0%
Gemini-2.5-PP
100.0% %
100.0%
0.0%
100.0%
100.0%
100.0%
100.0%
100.0
100.0%
LLaMA-4-Maverics
0.0%
0.0%
100.0%
100.0%
0.0%
100.0%
100.0%
0.0%
LLaMA-3.3
39.1%
39.1%
0.0%
100.0%
100.0%
100.0%
100.0%
100.0%
100.0% %
Swven32335B
87.0%
87.0%
87.0%
100.0%
100,0%
100.0% %
100.0%
100.0%
100.0 %
Claude-3.7 3.7-Sonnet
39.1%
39.1% 1%
0.0%
100.0%
100.0%
100.0% %
100.0%
100.0 %
100.0 %
GP BPPT-4-mini-high)
100.0 %
100.0%
39.1%
100.0%
100.0%
100.0% %
100.0%
100.0
100.0 %
34

---

Published as a conference paper at ICLR 2026
Table 14: SOLVE, @i results on intra-op parallelism problem.
SOLV @5
SOLVEI
SOLVE
Model
@ 10
@
@ 10
@
@ @110
@
@1
DeepSeek-V3
82.1%
53.6%
35.7%
82.1%
53.6%
35.7%
100.0%
100.0%
100.0%
DeepSeek-R1
92.9%
92.9%
35.7%
92.9%
92.9%
35.7%
100.0%
100.0%
35.7%
demini-2.5-Flase
100.0%
100,0%
100.0%
100.0%
100.0%
100.0%
100.0%
100.0%
100.0% %
Gemini-2,5-Pro
82.1%
82.1%
0.0%
1%
82.1%
0.0%
100.0%
100.0% %
0.0%
-LaMe---Maverick
96.4%
96.4%
3.6%
100.0%
100.0%
3.6%
100.0%
100.0%
3.6%
LLaMA-3.3
75.0%
75.0%
3.6%
82.1%
82.1%
3.6%
100.0%
100.0%
100.0%
Qwen3-235B
75.0%
71.4%
67.9%
78.6%
75.0%
75.0%
100.0%
100.0
100.0%
claude-3.7-Sommee
82.1%
82.1%
71.4%
82. I
82.1%
78.6%
100.0%
100.0 %
96.4%
GPT-o4-mini-high
100.0%
100.0%
92.9%
100.0%
100.0%
100.0%
100.0%
100.0 %
100.0% %
Table 15: SOLVE, @i results on protein sequence design problem.
SOLVEm @55
SOLVEI @R:5
SOLVEJ
Model
@ In
@
@ 10
@
@ 10
@855
@
DeepSeek-V3 V3
83.3%
83.3%
83,3%
100.0%
100,0%
100.0%
100,0%
100.0%
100.0%
DeepSeek-R1
87.5%
87.5%
0.0%
100,0%
100.0%
0.0%
100,0 00000%
100.0%
0.0%
Gemini-2.55Flan
95.8%
95.8%
95.8%
100.0%
100.0%
100.0%
100.0%
100.0%
100.0%
Gemini-22.5PPo
100.0%
95.8%
0.0%
100.0%
95.8%
0.0%
100.0%
100.0%
4.2%
LLaMMA---Mavercck
83.3%
83.3%
0.0%
95.8%
95.8%
0.0%
100.0%
100.0%
4.2%
LLaMA-3.3
12.5%
12.5%
12.5%
95.8%
95.8%
95.8%
95.8%
95.8
95.8%
Owen3-235B
87.5%
87.5%
87.5%
100.0%
100,0%
100.0% %
100.0%
100.0%
100.0%
Slaude-3.7-Somre
58.3%
45.8%
0.0%
100.0 %
100.0%
0.0%
100.0%
100.0%
0.0%
GPT-o4-mini-high
91.7%
91.7%
91.7%
100,0%
100.0%
100.0%
100.0% %
100.0%
100.0%
Table 16: SOLVE, @i results on mendelian error detection problem.
SOLVEm @
SOLVER @
SOLVE
Model
@10
10
@
@
DeepSeek- V3
100.0%
100.0%
0.0%
100.0%
100.0%
0.0%
100.0%
100,0%
0.0%
DeepSeek-R1
100.0%
100.0%
0.0%
100.0%
100.0%
0.0%
100.0%
100.0%
0.0%
Gemini-2.5 2.5-Flahh
100.0%
10.0%
10.0%
100.0%
100.0%
100.0%
100.0%
100,0%
100.0%
Gemini-2.5-Pro
80.0%
80.0%
80.0%
80.0%
80.0%
80.0%
100.0% %
100.0%
100,0% %
LLaMA-4-Mavvrick
60.0%
60.0%
60.0%
60.0%
60.0%
60.0%
60.0%
60.0%
60.0%
LLaMA-3.3
55.0%
55.0%
55.0%
55.0%
55.0%
55.0%
100.0%
100.0%
100.0% %
Owen3-23555
55.0%
55.0%
0.0%
100.0%
100.0%
0.0%
1000.0%
100.0%
0.0%
Claude-3.7 3.tnnnet
100.0%
100.0%
0.0%
100.0%
100.0%
100.0%
100.0%
100.0%
100.0%
GPT-o4-mini-higg high
100.0%
50.0%
35.0%
100.0%
100.0%
100.0%
100.0%
100,0%
100,0% %
Table 17: SOLVE, @i results on airline crew pairing problem.
SOLVEnI 655
SOLVEH 6:55
SOLVE
Model
@:10
@
@ 10
6
@ 10
6
@
DeepSeek-V3
100.0% %
I00,0%
0.0%%
I00.0%
100,0%
100.0 %
100.0%
100,0 %
10000 %
DeepSeek-R1
100.0% %
100.0%
100.0%
100.0%
100.0%
100.0% %
100.0%
100.0 %
100.0% %
Gemini-2.5-FFi Flash
0.0%
0.0%
0.0%
0.0%
0.0%
0.0%
100.0%
100.0
14.3%
Gemini-2.5-Pro
0.0%
0.0%
0.0%
0.0%
100.0%
100.0%
100.0%
LLaMA-4-Man Maveerick
100.0%
100.0%
0.0%
100.0%
100.0%
35.7%
100.0%
100.0%
100.0 %
LLaMA-3.3
42.9%
42.9%
42.9%
42.9%
42.9%
42.9%
100.0%
100.0 %
100.0 %
Qwen3-2355 -35B
21.4%
21.4%
0.0%
100.0%
85.7%
0.0%
100.0%
100.0 %
0.0%
Mlaude-3.7-Sonee
100.0% %
100.0%
0.0%
100.0%
100.0%
0.0%
100.0%
100.0
0.0%
-in----mini-tigh
100.0%
100.0%
100.0%
100.0%
100.0%
100.0%
100.0%
100.0%
100.0%
Table 18: SOLVE, @ results on pickup and delivery with time windows problem.
SOLVEnI @
SOLVEn @
SOLVE;
Model
@10
@ @10
@ @10
DeepSeck-Vo
0.0%
0.0%
0.0%
80.0%
73.3%
73.3%
100,0%
100.0%
100,0%
DeepSeek-RI
16.7%
13.3%
3.3%
100.0%
100.0%
100.0%
100.0
100.0% %
100,0% %
96.7%
90.0%
6.7%
100.0%
100.0%
100.0%
100.0 %
100.0% %
100.0% %
Demini-2.5-Pro
30.0%
26.7%
13.3%
100.0%
100.0% %
100.0% %
100,0 %
100.0% %
100.0 %
LLaMA-4-Maverick
0.0%
0.0%
0.0%
100.0%
100.0% %
0.0%
100,0%
100.0% %
0.0%
LLaMA-3.3
0.0%
0.0%
0.0%
100.0%
100.0% %
0.0%
100,0% %
100.0 %
0.0%
Qwen3-235B
0.0%
0.0%
0.0%
100.0%
100.0% %
100.0%
100.0%
100.0 %
100.0%
Claude-3.7 3.7-Senet
0.0%
0.0%
0.0%
100.0%
100.0% %
16.7%
100.0%
100.0 %
100.0 %
GPT-o4-mini-high
3.3%
0.0%
0.0%
100.0%
100.0%
100.0%
100.0%
100.0%
100.0% %
35

---

Published as a conference paper at ICLR 2026
E.4 ABLATION ON TEMPERATURE
We evaluate various models across different temperature settings, T {0.0, 0.5, 1.0). For each model, we run 10 iterations problem and report the highest QYI achieved across these iterations as the final QYI score for that problem. The overall benchmark score is then computed as the arithmetic mean of QYI across all problems. Detailed results are shown in Tables 19 to 21
In general, improving the temperature can be beneficial quality as the model becomes more creative, but may harm yield as it may not follow the constraints strictly. Note that yield emphasizes the best iteration that achieves the highest QYI, whereas SOLVEm reflects the cumulative success rate across iterations; therefore, their values may differ. Additionally, the weighted QYI is not the harmonic mean of weighted yield and weighted quality, as is computed by aggregating metrics across different problems using weighted approach.
also report an uncapped version the weighted QYI metri which better reflects cases where LLM-generated programs outperform expert solutions on certain test instances. Improvements are underlined in the tables. While this variant achieves slightly higher scores for most models indicating occasional superior performance it also confirms that, in the majority of cases, LLMs still lag significantly behind expert solutions.
Table 19; Performance of different models on Temperature = 0.
Weighted QYI
Weighted QYI (Uncapped)
Model
Weighted Vield
Weighted Quality
(Capped)
Claude-3.7-Sonne
0.5963
0.4686
0.5034
0.5034
DeepSeek-R1
0.6972
0.5775
0.5498
0.5553
DeepSeek-V3
0.4587
0.3890
0.3707
0.3707
Gemini-2.5-Flash
0,6606
0,5281
0.5682
0.5753
Gemini-2.5-Pro
0.6468
0.6700
0.6170
0.6228
LLaMA-3.3
0.3394
0.3521
0.2951
0.2953
LLaMA-4-Maverick
0.3211
0.3383
0.2955
0.2955
Swen3-233BB
0.4450
0.4513
0.4355
0,4423
Table 20: Performance different models on Temperature 0.5.
Weighted QYI
Weighted QYT
Model
Weighted Yield
Weighted Quality
(Capped)
(Uncapped)
Mlaude337--oonne
0.6147
0.6468
0.5437
0.5451
DeepSeek-R1
0.5138
0.5751
0.4743
0.4812
DeepSeek-V3
0.3716
0,4645
0.3322
0.3322
Gemini-2.55Flas
0.4817
0,5700
0.4760
0.4828
Gemini-2.55Pro
0.4817
0,5609
0.4767
0.4789
LLaMA-3.3
0.3991
0.4407
0.4108
0.4108
SaaMA-4-Maverinn
0.3349
0,3712
0.3050
0.3646
Qwen3-2355
0,4128
0.4798
0.4269
0.4327
E.5 FEW-SHOT DEMONSTRATION
Table 22
highlights the impact of few-shot demonstrations LLM performance across the entire HeuriGym benchmark. Introducing only a small number demonstrations (c.g., three) can negatively affect solution quality and success rate, these examples may not be representative of the overall dataset, leading the model to overfit to them. However, providing a larger set of demonstrations can potentially improve QYI, as the model benefits from greater diversity and learn more generalizable patterns.
'The uncapped version of quality is computed as 1/Ñ E Ge/on, and the uncapped QYT is derived by substituting the original quality metric with this uncapped variant.
36

---

Published as a conference paper at ICLR 2026
Table 21: Performance of different models on Temperature
Weighted QУI
Weighted QYT
Model
Weighted Yield
Weighted Quality
(Capped)
(Uncapped)
Mlaude-3.7-oome
0.5138
0,5924
0.4828
0,4841
DeepSeek-Fi
0.5688
0,5625
0,5313
0,5383
DeepSeek-V3
0,4128
0,4188
0.3839
0.3841
GPT-o44mini-high
0.6927
0,6440
0.6089
0.6158
Gemini-2.5
0.4771
0.7688
0,5030
0,5047
Gemini-2.5-P P.--Pr
0.5229
0,4893
0.4921
0.4981
LLaMA-3.3
0.3028
0.3627
0.2868
0.2916
LLaMA-4-Maverickk
0.2982
0.3271
0.2667
0.2672
Qwen3-235B
0.5459
0.5228
0.5294
0.5364
Table 22: Impact of few-shot demonstrations on performance (Model: Gemini-2.5-Prop
of Demos
Weighted Yield
Weighted Quality
Weighted QYI
Zero-shot 0.5872
0.7159
0.3999
Half-shot
0.5092
0,6526
0.5361
Full-shot
0.6468
0.6700
0.6170
E.6 FEEDBACK ROUNDS
Table 23 shows that increasing the number of feedback rounds has a nuanced impact performance. While a moderate number of rounds (e.g., five) can enhance overall quality by guiding the model to refine its solutions, excessive feedback may lead to diminishing returns This suggests that too many rounds can overwhelm the model, making it harder to identify and prioritize the most critical information from the feedback.
degrade performance.
Table 23: Impact of feedback rounds on performance (Model: Gemini-2.5-Pro))
# of Feedback Rounds Weighted Yield
Weighted Quality
Weighted QYT
0,6193
0.7290
0,6253
0.6055
0.7313
0.6259
0.6468
0,6700
0.6170
E.7 ITERATIVE BEST-OF-N SAMPLING
To investigate the benefits of test-time search strategies, we sample candidate programs in each iteration, evaluate them, and return feedback for all k programs to the LLM. After fixed number of iterations, we select best-performing program from the entire pool process we refer as iterative best-of-N sampling. The total number of sampled programs is held constant across different values of This strategy allows the model to explore diverse candidate solutions in parallel and evolve the program based evaluative feedback.
As shown in Table 24 increasing & leads to better quality of results, indicating that aggregating feedback across multiple candidates allows the LLM to better explore the solution space and improve sampling efficiency by allocating computational budget toward more informative evaluations.
Table 24: Impact of iterative best-of-N sampling on performance (Model: Gemini-2.5-Pro..
# of Samples @ Iteration
Weighted Yield
Weighted Quality Weighted QYT
2@5
0.5688
0.7698
0.6160
1@ 10
0.6468
0.6700
0.6170
E.8 ERROR ANALYSIS
In the following, we present representative examples of common errors made by LLMs during heuristic generation. These errors highlight current limitations in code reliability and execution:
37

---

Published as a conference paper at ICLR 2026
Import error: This type of enror occurs when the generated code relies external libraries that are not available in the environment. the example below, the model attempts to import the ortools library, which results in a ModuleNctFourddI: that the model does not strictly follow the instructions given in the prompt.
Such errors suggest
le "operator_scheduu ing itwp:ichhhiiiiiggiiiiiii-i-----1 2.
tio44 /oolers py", line 2, <module>
from ortools sat.pythoo import cp_model ModuleNotFFouddrrre No module named 'ortools'
API misuse error: LLMs often misuse APIs due to a misunderstanding of library interfaces. In the following case, the model tries call random () directly from the random module, which is not callable.
File "intra_on sataa_opppaaaee_e
line 64,
in init_jitter
if len (ci) > and random () 0.1:
amaa0aaa
TypeError: 'module' object not callable
Syntax errors are common when the model fails to adhere basic language rules. In this example, there is an unmatched parenthesis in while loop condition, leading to a SyntaxErros Such mistakes typically indicate a lack of code completion validation in the generation process.
Syntax error:
File "crew "crree_paiieeee wwwpee/parring/eeeeeeeeeeeaaaeee ptwp:://ppppeeeeetttttttaaa////a
1ine 60
while len (used_legs) len (df)
SyntaxError:
unmatched ")"
Runtime error: Even syntactically and semantically correct code can fail at runtime. In this case, the model modifies a dictionary while iterating over it, which raises Runt immerrmor ror. This highlights the model's difficulty in reasoning about the actual executable code in long context.
File
"technol Loonnooooyyyoooo htt.ppphhhnnnmmmmmmmmmaaaaaaeetr ver
line 104, in technol ogy ogy_mapping
for successor in G.successors (node)
RuntimeError: dictionary changed size during iteration
E.9 C++ EXAMPLE
We conduct preliminary experiments on the technology mapping problem by modifying the prompt to instruct the LLM to generate C++ solution, using the provided function tem- plate: void se (const std::strings input_file, const. std: std::string& output_file).
Integrating C++ into our agentic feedback loop remains challenging due to dependencies on domain- specific libraries and the complexity of parallel execution. As a result, our preliminary experiment with C++ involves only a single iteration of prompting.
Table 25 presents a performance comparison between the Python solution with 10 iterations and the C++ solution with just one iteration. Although the C++ solution does not produce high-quality output in its initial attempt, already achieves better yield than the Python solution after 10 iterations unexpectedly strong outcome. Notably, the Python solution fails to generate any valid result in its first iteration. This is attributed to the significantly faster execution speed of C++ code, which enables it to avoid the timeout emrors frequently encountered by Python in this task.
We expect to see further performance improvement with C++ after we integrate into the feedback loop in framework.
38

---

Published as a conference paper at ICLR 2026
Table 25: Impact of C++ code technology mapping performance (Model: Gemini-2 -2. 5 S-pro).
Language Python C++
of Iterations
Yield
Quality
QУT
10
0.7419
0.6423
0.6885
0.7742
0.3493
0.4814
E.10
TOKEN USAGE
Table 26 presents example of token usage when running the complete HeuriGym benchmark across different models. Among them, Gemini-2.5-P8 consumes the most tokens for prompt and completion.
Table 26: Token counts from single run of HeuriGym across different models.
Model
Prompt Tokens
Completion Tokens
Claude-3.7-Somme
1911947
298369
DeepSeek-R1
1215661
702746
DeepSeek-V3
1536084
122058
GPT-o4-mini-higg
1063688
978891
G.mini-2.5-Flassh
2742487
332569
Gemini-2.5-Pro
2880737
455739
LLaMA-3.3
969887
77523
LLaMA-4-Mavann Mavverick
1497730
77080
Qwen3-235B
1089888
127447
F DETAILED ANALYSIS OF CASE STUDY
Here we include detailed analysis of the case study in Section 5.3 We append the entire code of each iteration and analyze the generated solution.
In the first iteration, the GPT-o4-mini-high model generates an algorithm based on cut enumeration and dynamic programming (DP)-based cut selection, which is the common approach used technol- ogy mapping tools. The core logic is highlighted in the following code snippet. However, it does not prune the cut enumeration space and computes all possible K K-feasible cuts for each node, which leads to exponential explosion in computation time. While this achieves high quality, it frequently triggers Timeout Error for larger datasets, resulting in low yield.
/teeation 1: base algorichm without proning the search space
def solve (input file: ste, selution_file: strl:
collections import deque class Node:
5
slocs ('name', ' 'fanin_names "Canine', 'Lanoute', "palterns_one' "patterns_sed
'const_val")
7
def init init__sellll (self,
name
9
self, fanii_nmee_ss []
felinfenins
self.fanouta = []
self.patterns_cne []
13
self.patterns_zero = []
self.const_va.
raw cin_sines_ open (inputt_____ file) read() sppidi)epl
lines
17
-
0
while len lines)
19
line raw raw_lines [1] rstripl!
or wtw.::iii....tttsiialaaaa
21
1=
continue
23
if line Line.ends<ithh""WW
mezged [ine[:-]].estripf]
25
1-
while (en [entrax_iinssl:
27
axt (t)]liiee(]]..etetll
39

---

Published as a conference paper at ICLR 2026
nxt ent..endwwtth W
29
nerged F nxt axt [2-1] (estrip()
31
elae:
nerged
33
35
Lines aipend (nerged)
37
linez, append
39
model model_name name
input_ List []
out 11st =
()
Idx
while 1áx 1en (Lines:;
45
inne[1ddx) [1dx] atrrpp
1dx 1:8tartsswihh1111
if 1.startax"th(".modss pasts 2.oplit if en (parta) 1:
mode-_nama [arts[
idx +- elif L'
parts (spl't()
for in parts(1:]:
input Hist. apt.a
modos:
nodes zeme) Node (name)
idx elif L.startswich(' .uctputst""
parts (.oplit
for name in [1:t:
cutpur_ (namc)
idx elif 1.otartswith (nnmmee
pasts -.split()
in_names parts[l: 1]
out_neme parts[-i]
if caddesoot__emms Node (out.nnmm) nodes node nodes
not in nodes:
fade_faaiinnnmes in_names[:]
- idx while (lnneiss)):
[ines
if not 1j or 1a. 1j.startswich(1P)
break
pettecns.agpendte.
1
if Zen (in_nemes) -- 0:
gatterns patterns 11";
mode.const_vel. -
else:
node, const_velet
else:
for pline in patterns:
parts2 pline. pline.apLit() ap
if len (parts2) == 1: maak [a]ts2[0]; bit '1' else:
mask parts2[0]; bit parts2 [1] if bit == '1': node. pat.tattttte_eeeeeee appedd (mask)
else:
node.patterns_zeat zew.aaapeneo...ppeeeeee... (mask)
1dx =
elif 1.stertswichh end"))
100
break
101
102
idx +2 node nodes.values
104
node, faning = ()
fanin_names:
105
if fa in nodes:
node, fancuts - app.end (nodes [fn])
feninss
108
node sodes.valuesde for in node. node. fanins:
110
40

---

Published as a conference paper at ICLR 2026
112
reacheble - ze
deque()
114
for out in List:
in nodes:
115
dq. (otp
while de:
118
u dq.pep
120
continue
reachap_e add
122
for in nodes [u] [u]-
in reachable:
124
cq.append (f.name)
Aodes [name:node ues ()
node
itema if reachable]
126
for node in nodea.va valaa.aaleesse
fanins if
nodee]
128
indeg [nane: [f for in fanouts if f.name in nodes]
Canins)
130
dq deque |ame for name, in indeg. itema () iz == 0J)
Lopo_names
132
while de:
133
dq.popleft
134
aypo_names-appent for in nodes [ul. fenouts:
136
indeg name] if indeg name] 0:
138
6q 6q.append (w. name)
topt_list
[nodes(nand) for tapo_names]
140
5
141
daf prune_cuts (cset):
142
cuts (cset)
143
144
for in cuts:
False
146
for é in cuts:
is not and d. d.issubset (c):
148
skip True
break
150
if not skip: (cpend
151
152
return res cuts 11 for n in topo_list:
153
Cut Cnumezation
134
155
if not fanins:
136
cuts[n..amee] [frozenset [n.name])]
157
else:
138
cuts_n None
159
for E in fanins:
cuts futs.f.name)
161
if cuts_n is None:
162
outa_n of[:]
163
else:
newset set for in cuts_n: for eZ in ef:
165
167
cl c2 X:
169
newset.addle outa_n grune_cute (newses)
171
if cut.s_n is None:
cuts_n
173
cuta_set set (cuta_n)
outs add (frozenset ([a-name]) nammen.mama] = prune_ccts (ct)s_det)
175
cost 11
177
best_cut = ()
178
INE 10*
179
for n in topo_lists 11st:
DP-baaed Cut Selection
faninss::
181
cost name) =
best_cut nane]]
183
alse:
185
hva TNF
nammee.name]
187
if an frozenset n[n.nam)]
189
191
+- cost.gett (m, 0)
193
if eval bval:
41

---

Published as a conference paper at ICLR 2026
evai;
195
if bcut is None:
Erozenset name [anina]]
197
beut E1
199
For in beut:
cost.get (m,
201
bval - -1 na.e]
203
best_cut na.nane] beut
mapping_nodes
205
dq daque output_List:
207
if out in beat and best_cut [out:] is not None:
dq. append
209
while de:
()
211
if u in nsppingg_oddes
continue
213
mappl mapping ag_ best_cutjju]
215
if c:
for in m in nodes:
217
nodes [m]
219
if not in input_ 11st and comst_ss__val is None:
220
dq.eppend
221
mapping_toppe [n for in topo_Iist if nyname in mapping_nodes)
open (so_ution_fflll coln_ ".write(" model -mocel
223
224
f.writa innputss '4' (:oiin(inuur__Httt)) S.write(' .outpuus 14' (opoppptttt".opiioooooooooo..ei ".joiiloutpor__list):: An"") for mapping_topor
225
226
227
leaves best_eut (n.nmme)
inputs (laaves k len (inputs)
229
230
N 1<ck
231
fuil_mask (1<<20
=
233
for in range (k): 0
234
235
for j in renge (N):
236
if (1x
257
= (1<<j)
238
masks.append
239
vis set ()
240
queue2 [ame]
241
while queue2:
242
queue2. pppp))
243
in vis or in inputs: continue
244
245
vis.add for nodes Senins:
246
247
fn.name not vis:
248
queue2. append (fn. name)
249
cone_nodes- val [ name enumerate (Inputss))
[nodes [name] for name in topo_names if name An vis]
250
251
252
val name: macks(i]
253
in cone_nodess
254
if v.const_va is not None:
255
full_mask if v.const_val=- else C
256
else:
257
v-patterna__eet 2ero:
258
zero_mask = for in v.pattezns_aero. full_mase
250
260
261
for idx, ch in enumerace vm = val [v. fan [1dx]
262
264
$= vm elif chamt0':
266
f= [-vm) sful mask
267
zero_mask val [v.ngme]
==
268
270
one one_mask mask
271
in v.patterns_one:
272
full idx, enumerate vm val fanin_names [idx]]
273
274
275
1
276
5 Vm
42

---

Published as a conference paper at ICLR 2026
277
elif chmmf0':
278
5- I-vm) afull_mask
one_maax
280
val [V. [eeeemm] oon_maak
root mask na-neme]
282
E.write (' .nanes 1-. ".join (iaputs)+1 "in.name (An")
range (N)
264
if (coot_mas pooot_msskoo
11. joinn(' (1>91)61 else for range (k))
286
E.Mzite (liner" 1An'1
Wwrite (encAnn) cclooee
288
In the second iteration, the model overcorrects by adopting naive 6-LUT mapping strategy that trivially assigns each node separate LUT. While this approach passes all test cases within the timeout, it produces poor solution quality. Specifically, it redundantly maps each non-primary-inge node as an individual LUT by directly copying inputs to outputs, resulting excessive and inefficient resource usage.
iteration very naive mapping stzategy. Simply copy the input to output.
2
def salve (imput_filee sts, selution_fila: strl: raw = open (1 feeuttffferrree htte
marged 0
while [on()
line rawl1, ratrri()
if line line. (tt.rsipll.
10
continue
if line. heieeedddwwihhhh ('W):
Line(:-1] (sstri)) r)trip())
+-
while len (Fex)
nxt rew [ewlil.Fee cstrrip
accum + nxt Ie- rstrip)
else:
+
break
merged
append append (accum)
else:
mezged. (linel
1- with open (solucion 'x') as
for in merged:
f.write '(n')
In the third iteration, the model proposes sophisticated strategy that first applies static pruning to the cut enumeration space limiting the number of cuts per node to 20 - followed by dynamic programming- baaeed cut selection. This approach improves solution quality, though full yield is still not achieved. Additionally, truth table computation is optimized using bitwise operations.
Eteration 3: statle pruning capping the numbez of cut's node to M 20; efficient - hitwise operations
3.
def solve (input_ stc, solution_filer stal:
from collections import deque
5
rew raw_lines open (input_ raae))rraa))))) read() readdi).eliitlines ()
lines - []
while Lem raw_l'neat rstripp(
if not line or wtt.siippt..ttttsiiil.......
continue
Iine.endsuitt(""N)
accum Linel: 11] retripp)))
+=
while Ien fraw fenfraalines// linest: exc raw Liwal (t).sst(ppp)
18
if axt. enddendddithhh ( accum
nxt [xt])))))rratipp
43

---

Published as a conference paper at ICLR 2026
21
elae:
accum
23
25
Linez append
27
linez append
29
class Node:
name):
self.name nama name
sel_._anin_names
self self.patterns_on
selL.pallerns_eerr
35
self self.conat_ None
seli.Ianins []
Sell.fanouse = ()
model_ name
Inpot_ 1st [)
output_lise
1od8a ()
idx
while 16x len inest:
1 lines
if not
idx continue
if t.stortawsthh..ma
parts spp if (parts) >: modei_name parts[]]
idx
elif "ntartswiich".iipute
(.
for name in parts[l:]:
input_ perut_list..ppe itt.....dd.eeddna.........
name not in nodes:
nodes
(name)
idx elif ssstertswich "outputut"..u
parts E.split()
for in parls
(neme)
1dx 1-
elif 1. s.stertswihh .names"):
pazts s.split
in_names [arts[1:-1]
partsl
if out_name not nodea:
nodes[out__emn)
Node (out_name)
node nodes [out_nnou_
node. fanin_nemss in_names[:]
patterns []
- idx
while (linea) and [3)
Lines striipp
1 Setternss.apeeddie
if en (in_names) =s 0:
gatterns patterns f1's
node, won.ccontttttt. vpl =
else:
mode.const_ne =
else:
for pline in patterns:
parts2 pline. apine.
88
if len (parts2) =s 1:
parts2 [0];
else:
[a]ts2[0];
parts2
if bit == '1':
node.patterns_cne.coo append
else:
nade www.ppatttss_errooooe (mask)
1dx =
elif a.stertswichh endd))
98
break
100
idx +-
nodes. (odes.values()) ():
102
node. fanins = [nodes [nm] for nm in fedde.faainnameee if zm in nodes]
44

---

Published as a conference paper at ICLR 2026
node, []
104
for node in nodes. v)
fanins:
106
f.fanouts ...anouts.epperdddat append
reachable set
108
stack [am for nm in output_list if an. in nodes;
110
u stack.papt)
112
reacheble. add
nodes [u] [an_
114
if in nodes and p not in reachable:
app
115
modea [nmand for nm, rd in nodes. if nm in reachable) vodes.values()
118
for in na.fanina if n.name in nodes] ad. canouss Indeg = [nms len Ind. faninst for nm, nd in nodes. Itema [1]
nd. fanfaning
fanouss
in nodes]
120
deque [Jom nm, indeg, iLems topo []
0]}
122
while de:
124
u dg.popleft
topo.appenddu!
126
for in nodes (u)- indeg name] if "ndeg namaee] =* D:
128
éq.append
130
tapo_naćes nodea [nodes [m]] for nm in topo]
131
name_ta_idx Ind.name:i for i, in enumerate (tapo_nadess)
132
idx_to_node topo_nodes
133
idx_ro_name [nd.name for nd in tepo_ncdes]
134
lenttopo_nodees
135
2__idx (nama_to_idd) [nm] for in input_ if in rama_to_iday self_mask [1 i for range (NI)
136
138
for range
140
for i, nd in enumerate (tepo_nneeddd
141
if FI_idx rd.conat_vaa not None:
142
cuts[i] [seif_maskli]
143
continue
144
fan_idxs [name_to_idx(x) for x nd, namen_nemes if x neme_co_ide) c_iist None for in fan_idxs:
145
146
147
feuts cuts[f]
148
if c_list None:
149
c_iist [cuts[iK]
130
else:
151
Rewsel set()
for [
153
for in [cuts[:M]; a if u.bit_count K: newset.addio
154
155
157
if newset:
Lat sorted Inewset, key-lambda x: x. x.bit_count (1)
159
e_iss [st[:M]
161
c_iist
162
not c_llat:
163
break if c_sist None:
164
165
1_1ist []
set (c_list) (sal [1]]
167
1st2 sorted key=lambda xt x.bit_count ())
169
cuts(1) 1st2
INE 1G++18
171
cost [0]+N
[None]
173
for nd in enumerate (toponnddss)
Fi_idx or const_val None:
175
cost [1] =
bestM[á]
177
continue
179
hm Hone cuts(il:
181
if em -- www. [1]]
183
185
while m:
45

---

Published as a conference paper at ICLR 2026
187
3 _eb.bit__ lengtht
cost. m 1sb
189
191
if eval beste:
193
bm - cm
195
197
for in nan.fain_nme panin_namess name_to Lome_to_ide idx:
199
3 name_to_ [dx
self_mask[j]
201
a cost
203
bestc a +
[i] bestc
205
beatM[1 on
mapping
207
stack thame_to_ldx(_____ for nm output_Mast it om An name_to_ide
208
visited sat()
209
while stack:
stack.pop()
211
if u in viatted:
continue
213
visited.add (u)
214
bm bestM[u;
215
if bm is None or hm --- maalf_mskkkppoee [u]
continue
217
magping.add(u))
hm while m:
219
220
1sb - 1sb bib.biit_length() -
221
A= 1s5
223
if not in visited and j not in Pi_idx and topo_nedes col.tonst_va is None: stack.cpperddj]
224
225
mapped = 1i for in range (N) if in mapping]
226
with open (soolliioon
'e') E:
227
f.write .model E.write (" in.iptss * " list 1 'An") w.wrice('.outeee for in mapped:
model_name name "An')
228
229
laat 'An')
230
231
nd tepa_nodes(s bm bestM[i]
232
233
leaves 15 for j in range (NI if Con >> 3) inputs [idx_to_name[]] [j] for in leaves. (inputs)
234
235
236
Np <c ful C Np)
237
238
masks
239
for in range (K):
240
241
for range (Mp)
242
if 52 ti mn
243
244
masks. eppend (mm)
245
val [1
246
for t. in enumerate (leaves):
247
valli] masks[t]
248
vis set (leaves)
249
stack2 - [1] while stack2:
250
251
stack2.pop
252
if u in vist
253
continue
254
add
p topo_nodes fanins: pi hame_to._ idx, get (p.name)
256
pi is stack2 ap
and pi
258
cone [3
range (N) if
and j not lsaves]
260
for j in sortad tcone, kay-lambda name_to_ide (ttoooooooooo name)] _opo_nodes!]]
261
262
if v.const_ val is not None: vallj:
263
v.const_val.
264
else:
265
if v.patterms__erto
266
zn =
267
p w.patterns_zero:
268
am full
46

---

Published as a conference paper at ICLR 2026
E, enumerste (p)
270
Ea fenin_nemmes [t.
p] naame_to_idx__________
272
if Pj is None:
274
break vallp]]
276
if ch 111:
278
elif ch -- 101;
(- fuil
280
zm 1 mm val[j] (~2m) cull
282
else:
284
for p. An v-patterma_ones
285
for t, ch in enwmarate (p)
288
pj name name__to fo__ddx.getttt.. (fn)
pi None:
200
mm break vm [a]
291
202
293
if ch 11':
204
am 6- vm
295
olif ch == 10";
205
an 6- 1-vm) 6 fu")
297
208
[3] cm
299
vi]
300
f.write(" .names
'.join (imputs)
namname + 'An')
301
for range
302
(root >> })
303
bits 11 ("'' if 13 alse '0' for ronge (k))
304
f.write (bits l\m')
305
f.write (merit""..eddn"))
306
In the fourth iteration, the limit per node is increased from 20 30, enabling broader solu- tion exploration and potentially improving quality. Additionally, the algorithm reduces redundant computations by caching precomputed scores for each cut.
iteration ds exploze lazgez solution spacez reduce zedundant computations
2
solve camput_files str, solution_files stel:
from collections import deque
raw (smput___iii read() (eg.lar.rred()..lttllllt.e splitlines lines []
while 1. < (rew)
rax raw"i]. sstrip()
10
if not or 1. "tttipssssaartsit(((""))
12
continue L.endewith('
14
scc - xstip))
15
while i len(rew)
cew csliipp
18
if mi. E..endswiihh('V
11 estript)
20
22
ace +l
24
break
Lines append (acc)
25
else:
Lines. (2)
28
+=
Node:
30
slots_ I'name', 'faninnnames', "pacterna_one", patterna_zero' 'oonat_val', 'fanin
+
s", "'noouss)""
init__see (self,a)
32
name - n
fanin_nens
34
sell. sell.palleens_one - []]
self.patterns_zers
const_v__ None
47

---

Published as a conference paper at ICLR 2026
self. []] self. fanouts - []
38
model 1
40
inputs []
outputs L:
42
aodes ()
idx
44
while idx len
(idk).
46
if not ::
1- it thstartsw" (mmodeee)))
pazta =.8p1it
50
if en (partaa))1 model parts[1]
idx elif 1.atartswlthh". .tnpsttt
52
pazte 2.8p1it for nm in [1:)s
inputa (om)
%6
nn not in nodea: modea [nm] Node [nm)
idx alif 1.atartakithi" "oupputtht".rrrttt"""
58
parts -.split
for nm (1:ts(1::]:
outputs. appendd idx elif 1.atartewith ('aamens))
parts sspl(t()
parta(i:-l];
parts[-1]
if outp not in nodes: nodes (outp) Node toutpl nd nodoss
fadin_nnnms inps[:]
idx+_
while 1en (lines) and not lines (t
72
: lines [j] stripp
pata.cppend(s)
j +-
if not
pats and pats [0]=(t'' nd.const_val.
else: nd. const_v__val =
78
else:
for in pats:
sp st.sp
if (sp)
mask spl0]; bi: '1'
else:
misk, sp[0].sp[i]
if bit==!l'; ad. pdl.peteenss_te append (mask)
else: wed.patterns_eeeee www...prrr........ (maak)
1dx
elif s.startewthh""endt break
idx 1-
for nd in vales..aluues):
nd. Fanins "nodes [[m] for in ad. namen_names if in nodes]
for nd in (vlues))ll
for in nd. fanins:
f.Ianouts.appen (nd)
reachable = set
st [nm for nm in outputs if nm in nodes]
while at:
100
u ptp if u in reachable: continue
101
102
seachable.addie wat.fffnnnnnaannenn
103
for in nodes (u]
104
if in nodes and p not reachable:
105
st.append
aodes (amend
nm,
nodes.items
reachable)
107
for nd in andes.valuedl ()
faninnins :f
nd.
f.name nodes]
109
nd.fanouts = [f for in nd. fanouta if f.name in nodes) indeg
am, nodes. modes.items items (1) da dacuei [mm for nm, d in indag, items if daw0)) topo while de:
111
113
u dq.popleft ))) topo.append for in nodes[u] fe]_uensuts:
115
indeg(w. name]
117
if indeg w. ==0m eg.eepedd nanee
topo_nades (nodes [nn] nm topo)
119
N len (topg__ooodde))
48

---

Published as a conference paper at ICLR 2026
name_t ind.name:1 nd enumerate (tapo__ndess)
121
idx__o_ [ad. name for nd in tepo_nedes]
tnane_to _o__idx [nm]
inputs name_to_ida
123
cost [0]
beatM [None]
125
X 5
127
cuts [[] for in range (N)
1, if i in PI or nd. const_vel is not None:
(Lopo_nodes):
129
cost[i] [i]
131
bestM( [] None
cuts[i] [[1ssi, 0}]
133
continue
_name_to_idx(mm
Canin_nnnaee if name_to_idde
135
e_ 1st None
137
fout [f)
if (Icuta) feuts [CM]a[:M]
139
if 1_t is None:
c_list [cuts.copy
141
else:
142
143
for n1, 30' in o 1st::
144
for m2, ac2 in fouts:
145
m if m.bir__count(
m2
146
147
if m in newn: continue
148
149
while t:
1sb
leb.bit. bengheegghhh>
151
SC cost [3]7 t Am isb
153
not nеиm:
154
c_list ()
135
break
itoma sorted (newm. items key=lambda x(1))
157
c_list items [:X]
if c_list is None: l_list
139
fb
160
scf 0
161
for in fans: Ib (1<cf) sef cost
162
163
164
if fb.bit_count <-
165
if not any im=-Eb for m. in c_list);
166
c_pisteppeend (Eb, scf))
167
beste 10.+10y on None for in e_list: +
168
170
if beste:
171
besto V;
172
if bm is None:
bm Ib; beste set cost[i] bester bestM[i]
174
cuta[i] sorted list. key-lambda x: x(1.) :MM mapping set
176
177
st
[nm] for outputs name_to_idk [dx]
178
vis set while st:
180
u st.pop())
181
if u via: continue
182
vis.add (t)
183
bestMlu.
184
if bm is None: continue mapping.add(ur.
186
t hm while
187
188
150 = -ty lab. bit (ongth t >e 1sb
or topo_mades consttvvv t.pp_ttoess....contttvall mapped sorted (mapping))
J if not in vist: st.append
continue
190
191
192
with open (salution ff tw') as f.write .model
'An')
194
+ ".jointinpaten "An') E.writel'. ('rtpput. for in mapped:
(.writee',
(.join(au 'An')
196
197
topa_nodes[i]
198
bm bestM[il
leaves for j range (N) (11))j)a11
200
iaputs_1 [idx_to_pane [j] for in leaves] (imputa__])
201
202
Ap lcck
49

---

Published as a conference paper at ICLR 2026
203
ful: (-<<up)
204
masks []
range
206
mm
range (
208
if (fost)sl: mm 1663
maaks.apgend (mm)
210
Vel - !} enumerate :leaves):
212
val[j] nasks[t]
(leaves)
214
stk [1]
while atk:
215
6 atk-pop()
vie2:
218
vis2.add (u)
Lopo_nodes fasins:
220
p] name_to
if pj is _ro_ddx.get (p.name)
pj vis2:
222
st aappedd (p:)
Lopo False]
224
cone [3 for in range (N) if in v1a2 and 5 not in leaves]
for ] sortad/cone)
226
v
v.const_val.
None:
228
val val full if v.conat_va valla_ else
else:
230
if v.patterns_aerro
251
232
for p in v.patterna_ re full
233
234
for t, in enumerate (p)
235
fn v.famin_namez Pj name name_ttm__ae setee_t__ixx_eeeeeee (fn)
236
if pi None:
238
mm D5 break
239
val(pp]
240
if chem!1's 1 mm vm
241
alif ches 10'; 6= (-vm) 6full
242
zm 1- mm
243
val[j] (>2m) &full
244
else:
245
246
for p in v.patterns_one: full
247
248
for t, ch in enumerate (p):
249
Ia faninnnamess [t.
250
Pj name to tt_.ic..eiiet_eeeeee (En)
251
if pj is None:
252
D; break
253
val[pp]
254
ch=="l's mm
255
elif ch 10!= 6- (-Vm) afull
256
om 1- mn val[i]
257
258
root val[i]
259
f.write(" nemee 1 ".join (inpu:s___ for 1 in range (Hp):
nd. namname 'An")
260
261
if (root>> a1:
262
pat " join( (1ast)a1 else 1D" for in range (K)
263
f.uritelpat lAn')
264
E.writer (end\n")
265
In the fifth iteration, introduces dynamic pruning method by adapting the cut limit based on eircuit size mize-more cuts for smaller circuits (higher quality), and fewer for large circuits (better yield).
iteration 5: dynamic pruning of enumesation
def solve (input_ files ste, solution_"lle: strl: collections import deque
raw open (Laput__[_lll read()) splitlines [
7
(raw))
9
1 raw :striio))
or ..atrrip...trrrsstttthe
11
continue
if 1. c.
5ec [[==]] rstrip()
50

---

Published as a conference paper at ICLR 2026
15
while [en
18
if al, d..endswihhh' cstript)
20
1 1-
22
acc
24
break
agpend
26
else:
Linee append (1)
28
+=
Node:
30
slots ('name', , 'fantn_names' "pacterns_one", [patterns_zero" 'const_val' 'fanti
s', "Eanouts')
selr.namp - n
selt.tenin_nanes
self selT.patterns_ong -
sel..palle:ns_zets
self.conat_
None
senin []
38
fanouta = []
model 11
Inputs []
outputs
42
nodes ()
idx
while 16x (=iest:)
1 linos [ino strrpp(
if not -: continue
idx
if L.atartsw(thh' (mmdeee)
parts 1.split
50
if (parta)) >. 1: model [1]
idx elif 1.stertswich(' iiinptts"'.
parts (.split
for nm in parts[l:];
inputs.oppend
an not in nodes: modes [nm] Mode (nm)
idx elif L.stertswich( cuupptt::")
parts s.plitt)
for nm parts(
apppudd
idx elif 1.stertswith nnamees))
parts 2.split() inps parts([:-1]> if outp not nodes: nodes(outp] bode (outp) nd [outs fddinannnnemme inps[:] =
parts[-1]
idx
while Len (lines) and not lines [j] startswith
a Lines stripp!
(ppend
i
if not inps:
nd.const_val
pats and pata(0] 121 else
else:
for pt in pets:
st if len(sp) 1: mask, bit sp(0), 11
else:
mask, bit sp[0), sp[1]
if bit "I' nd. nd.pacterne one www.onn..eee...... append (maak)
else: patpers zero-appenc (mask)
idx
elif w.startwwfhh(..edd break
else:
for nd in nodes.values (
fonins [nodes [nm]
fanin_names nades]
for nd in aades.valueal
fanins:
f.fanouts. (.fanoots-appeeddnn
se5
51

---

Published as a conference paper at ICLR 2026
[nm outputs while st:
96
(t.
100
if u. in reachable: continue
reachable -u]__iein__eeeee Seenin_names:
102
for in nodes
p reachable;
104
st. st.append(p)
godes [amend
nodes.ilems()
reachable)
106
for nd in nodes. ():
nodes]
108
fanouts = [f for in nd. "anouta if t.name in nodes] (nd.
am, nodes. itema (1) dq = decue nm for nm, d in indag. ttmms () 1f 6 an G]}
indeg
110
Lopa
112
while de:
dq.poplefL Lopo.append
114
for in nodes [u] [u]-fsnouts:
indeg name]
115
if "ndeg waname] =* D: dq.append (M. name)
Lopo_nodea [nodes [ns]
topo]
118
len (topop _odess
name_ta_idx ind.nama: i for nd in (topo_rodea) idx_to_nand _am_onnmme [ndename for nd in node_]
120
(wana_ntnnattaaeeeeeeeeeeeeeaaee [nm] for inputs if in name_to_idx) cost [0]
122
bestN [None]
124
if <= 1500:
126
M 64
olif 3000:
128
X 48 olif <= 5000:
130
X 32
131
alse: N -
132
for ronge
134
def prune_cuts (items, limit):
itame_sarted cortod key-lambda (x[1), x[0] _bit__contt ()))
136
p= []
137
for in items_sorted;
138
dom False for pa,
139
pe:
140
psc <= se and (pm ml
pm:
141
dem True
142
break
143
if not dom:
144
pr. pr.append (m, sc))
145
if lea (pz] >- limit: break
147
return pr
for 1, nd in enumerate (tosoonnddss)) nd. const_vel is not None:
149
if cost [i]
151
bestM[i] None
152
outali] [O] << 1, 0}]
153
continue
154
fana
ttame_to_dde _name_to_iix([mm) [nm] for nddinann_names if am in name_to_ion 1dx]
155
fane. sort (key-lambda len (cuts (x]))
156
l_iss None
157
for in fans:
fout outa[E]
159
if not fouts:
c_list
161
break fouta fouts :M]
162
163
if list is None:
164
c_ltst Ecuts.copy ()
165
else:
newm !]
167
for m1, so1 in c_ litt: m2,
169
m1 m2
m.bit_count 52 = sal 5c2
171
prev newm.get (m)
173
if prev is None a2 < prev: nexm(m)
175
if not nеиm:
c_st [1]
177
break
list prune_cuts
(1),
179
if not
list:
52

---

Published as a conference paper at ICLR 2026
um
181
for E in Feas:
183
use -- cost[E]
0 l_t [um,
185
um 0; use
in rans:
187
um - (1 << fI
189
if um..it_ccontt () <- and all (m !- um for in c_litt):
e apppppa (un, usc)1
191
e_ 1st prrune_cut cut. 1o_ 1ist, M)
beste 100xx8;; sel fm (1 << 1)
193
m,
c_list:
195
if -= aml continue
197
sc bester
199
bests = vy hm =
201
bm um
202
beste u
203
cost [1] beate
204
bestM
205
cut. [1] = c_ 1st
2016
mapping
207
st [name_to, 1dx [nm] for nm in outputa if nm in name_to_1ad 1xx] via act
208
209
while st::
st.pop()
211
if u vis: continue
vig.add bm bestMlu:
213
214
if bm None: continue
215
mapping.add(u))
hm while t:
217
1s5
219
= 1sb 1sb.bit_length((
220
~- 18b
221
if in P1 or topa_todes constt_vv not None: continue
222
if not in vie: st appendd))
223
224
mapped sorted with (po
225
'w') as f:
226
E.write modell model 'An1
227
f.write ('.imputs E.write ('.ouppus magped:
linputss) 'Aa')
228
". (outtttutsps) 'An')
229
230
nd [ope_nodes bm bestM[i] leaves for zange (N) if Con, 59 input ti_dd_t__naeellle [1] for An leaves]
231
232
233
234
len (Inputs_
235
Mp ful: Np) masks
236
237
238
for in (k)ge(():
239
240
for in range (Mp)
241
if (1 29 tl mm 1= <x
242
243
maaks. aapend (mm)
244
val []
245
for L, 1 enumerate (leaves)
246
vel val(s) = naska[t]
247
seen set (leaves)
248
stk [1]
250
while stk: stk.pop!)
252
if u in saen: continue
seen.add
254
cone-append
p tope_nodes
256
E] name name_to_i _o___o________________________ so_idx.ger (p.name)
257
Fj is not
FJ seen:
258
(ppen
cona. sort
260
for j in cone:
261
tepo_nodes[[]
262
if v.const_val is not None:
53

---

Published as a conference paper at ICLR 2026
vall3: if v.const_vai.
264
else:
if v.patterns_
266
in -
p v.palteras_
268
mm full
enumerate
270
fa v. waniianinnerre [t.
p] it_idttood_id___oeeee get
272
if pj is None:
274
break
vallp]]
276
if ch 15::
278
elff ch - 1011
(-ym) full
280
zm 1 mm
val[J] (>2m) Cull
282
else:
284
for p. An v-patterna__on full
285
285
for t, ch in enuuerater (p1
287
fn waw.i_nnameeeeee
288
p] name to_ pp..dx..geettt.. (fn)
289
if pi None:
200
mm break vm [p]
291
202
293
if ch '2':
294
mm 6- vm
295
olif ch == 10";
296
mm 6- (-vm) & fuil
298
val[j] cm
vo]
300
E.write(" .names for ] ranga (MP):
1.join (inpuzs_1)
nd. name 'An')
301
302
(root >> 3) 1:
303
" join if 30 L) else 1g" for in range
304
E.write/pat 1An'1
305
E.weite (end\n")
306
G DATASETS
We summarize the original data sources for each problem in Table 27 and the number of data instances in Table 28 All datasets are derived from real-world applications. We further partition or transform them into standardized input formats, ensuring the inclusion of both small-scale instances for demonstration purposes and large-scale instances for evaluation. For detailed data organization, please refer our repository.
Table 27: Datasets used our benchmark.
Problem
Original Data Source
Operator scheduling
EXPRESS Wang et al. 2007
Technology mapping
EPFL. Ammru of al. 2013 and ISCAS8S
Hansen et al.
1999
Global routing
ISPD'24 Contest (Liang
2024
E-graph extraction
SmoothE (Cai
2025
Intra-op parallelism
ASPLOS 24 Contest Motiit & Fegade 2025
Protein sequence design
Protein Data Bank (PDB) (Daaahuse 2024
Mendelian error detection
Cost Function Library (Sanchez 2008
Schiex 2018
Airline crew pairing
China Graduate Mathematical Modeling Competition 21 Competition, 2021
Pickup and delivery w/ time windows
MetaPDPTW Tim. 2001
54

---

Published as a conference paper at ICLR 2026
Table 28: Number of instances of each problem in HeuriGym.
Problem
# of Instances
Operator scheduling
24
Technology mapping
31
Global routing
24
R-graph extraction
23
Intra-op parallelism
28
Protein sequence design
24
Mendelian error detection
20
Airline crew pairing
14
Pickup and delivery time windows
30
Total
218
55


Reviews:
[{"review_id": "1c0BbM4PXK", "reviewer": "Reviewer_NKuf", "summary": "This paper introduces HeuriGen, a benchmark for evaluating LLMs on combinatorial optimization tasks that require generating and refining executable heuristics through an agentic feedback loop. Covering nine diverse problems across science and engineering, the framework measures both solution feasibility and quality using the proposed Quality–Yield Index (QYI). Results on nine leading LLMs show large performance gaps, revealing current models’ limited capabilities in adaptive reasoning and real-world problem solving.", "questions": "1. I have one question not directly related to the proposed contributions: in section 3.1.1, the authors mention that the optimization object and the constraints will be presented as mathematical notations. I saw some papers about LLM for heuristic generation, but they only do this by text description. Based on your knowledge and your experimental experience, do you feel that using math notations instead of pure text descriptions will bring some advantages?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "1. I personally appreciate that this benchmark supports not only Python but also C++. Most other heuristic-generation benchmarks are limited to Python, so this cross-language support is valuable for real-world deployment scenarios and makes the benchmark more practical for diverse research settings. The released codebase is also clearly structured and of good quality, which will be appreciated by the community.\n\n2. The problem selection process is interesting and reasonable. I was interested by the use of Google Scholar search and citation statistics as part of the selection criteria.\n\n3. Choosing CO as the core evaluation domain for LLMs is strongly motivated. LLM for CO heuristic generation is a spot topic recently. This choice provides a solid foundation for evaluating the true problem-solving ability of LLMs in realistic, structured scenarios.", "weaknesses": "1. One concern is about the token usage. In the feedback loop: “After each iteration, we log the LLM-generated solution, execution trace, verification result, and evaluation score.” I feel that it appears token-inefficient and may not scale well to larger problems or longer iterations. Appendix E.9 shows multimillion-token runs, confirming substantial computational overhead. A more sustainable design might summarize or structure feedback (e.g., key errors, constraint metrics) rather than concatenating full logs, preserving the agentic realism while improving efficiency and model focus.\n    \n2. Another concern is about the fairness of the QYI design. I do like the current F-score-like design, neat and clear. But my *intuitive* feeling is that the current design puts the “quality” and “success rate” at equal importance, while in some contexts, achieving feasible solutions is more critical (at least in the beginning iterations), while in others, solution quality should dominate. Moreover, the binary definition of yield ignores near-feasible outputs and can disproportionately penalize partial success.\n    \n3. The authors mention that “we intentionally exclude ubiquitous problems”. However, I have a different opinion that this complete omission of such tasks may reduce interpretability and comparability. Classical problems serve as essential anchors, providing intuitive performance references, allowing transparent quality validation, and enabling clearer analysis of the heuristics that LLMs discover. Including even a small subset of randomized or reparameterized classical problems would not compromise contamination control and would significantly enhance the benchmark’s scientific and pedagogical value.\n    \n4. While the introduction of a weighted QYI is conceptually appealing, averaging QYI values across heterogeneous problem domains may not be statistically or conceptually justified. QYI is a relative metric that compares each problem’s ratio to its expert baseline, so cross-task comparability is limited by differences in inherent difficulty and cost scaling. For instance, achieving a 0.8 QYI on an arduous routing task may represent stronger heuristic reasoning than a 0.9 QYI on an easier scheduling task, yet the weighted average treats them equivalently.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "djhRjIIMGN", "reviewer": "Reviewer_iG2L", "summary": "The paper introduces HeuriGym, an agentic benchmark for testing whether large language models can invent and refine heuristic algorithms for combinatorial optimization. Instead of closed-ended Q&A, models write Python solvers, run them, see feedback, and iteratively improve under verification, mirroring real engineering workflows. Tasks span nine realistic problems across EDA, compilers, computational biology, and logistics. Performance is scored by the Quality-Yield Index (QYI)—the harmonic mean of solution quality (relative to expert heuristics) and pass rate. Across nine state-of-the-art models, top systems such as GPT-o4-mini-high and Gemini-2.5-Pro reach only ~0.6 QYI versus an expert baseline of 1.0, revealing persistent gaps in tool use, planning, and constraint handling.", "questions": "Please see the questions.", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- It is a well-motivated benchmark that challenges the models in their critical agentic capabilities such as tool-augmented reasoning, multi-step planning, and instruction following.\n- The benchmark involves well-defined continuous objectives, large solution spaces, and agentic settings. They are suitable for benchmarking current fast-evolving LLMs.\n- The paper conducts an extensive evaluation and verifies that the benchmark is challenging for SOTA models.\n- The paper reveals current limitations that are insightful for future methodological development.", "weaknesses": "- In Table 4, you use the metric SOLVE@10. Does this mean only 10 generations are allowed for LLM+EA frameworks? This may be an unreasonable budget for EA frameworks, which typically require more iterations to achieve performance gains. Also, is it possible to incorporate feedback from your benchmark into these EA frameworks for a fairer comparison?\n\n- Is it possible to include black-box problems, as emphasized in ReEvo, to test LLMs’ generalizable reasoning capabilities without relying on internal knowledge?\n\n- How does this benchmark compare with other similar benchmarks, such as ALE-Bench?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "aXagzyPxBa", "reviewer": "Reviewer_ZY54", "summary": "It introduces a new benchmark named HeuriGym for evaluating heuristic algorithms generated by LLMs for combinatorial optimization problems. It also presents a new criterion with three stages for the benchmark. Experiments were conducted on around 10 LLMs and different important yet not so well-known combinatorial optimization tasks. It contributes to both algorithm development and LLM communities.", "questions": "1. The problem specifications are generated by prompting DeepSeek-v3 and iteratively refined until the description is unambiguous. However, the optimal prompt for different LLMs (even different versions within the same LLM series) can vary significantly, especially for complex problem-solving tasks. While acknowledging the difficulty of designing a universal prompt, the iterative approach on a single LLM to design the prompts should be reconsidered or further clarified.\n\n2. Please provide reasons for the timeout settings in Table 7. Why are different timeouts assigned? It appears that 10 seconds may be insufficient for many metaheuristics when implemented in Python.\n\n3. Testing evolutionary frameworks, such as EoH and ReEvo, with only 10 iterations might be misleading. Common practice typically involves hundreds of evaluations to effectively assess evolutionary algorithms. Ten iterations with single-point-based refinement (i.e., without a population or with a population size of 1) closely resemble a random search or local refinement. To ensure effectiveness, a population size of 10 or 20 is often used. The settings for assessing evolutionary frameworks should be reconsidered, incorporating more iterations and appropriate population sizes.\n\n4. Furthermore, I do not believe that designing a piece of an algorithm or the entire algorithm constitutes a key methodological improvement or difference in comparison with existing works, as it primarily involves letting the LLM generate longer code or complex metaheuristics. When the template and evaluation block are replaced with new settings, existing methods like EoH and ReEvo can readily be used for entire algorithm design, as evidenced in many recent works. While the benchmark's importance is not disputed, it is better to fit within the existing body of LLM-driven algorithm design research.\n\n5. The selected tasks seem to be either too simple or too hard. For instance, results for the operator scheduling problem show 100% success for most models, even without any iteration. Conversely, why do all LLMs fail on the global routing problem (i.e., 0.0% even after 10 iterations)? Is this due to hard constraints? Would results improve with more iterations?\n\n6. The authors argue that they focus on novel yet foundational problems, rather than commonly used combinatorial optimization tasks such as TSP and Bin Packing. However, establishing clear boundaries for task selection is challenging. Although the authors provide a methodology for selecting tasks based on Google Scholar searches, this approach can hardly guarantee the absence of pattern matching during heuristic design.\n\n7. Could you provide the details on the training and testing instances? Are they of the same distributions and sizes? How is the out-of-distribution performance of design algorithms, which is one of the main limitations of current automated heuristic design methods.\n\n8. A discussion and comparison with related platforms and benchmarks is suggested, such as LLM4AD and Co-Bench.\nLLM4AD: A platform for algorithm design with large language model. arXiv 2024.\nCo-bench: Benchmarking language model agents in algorithm search for combinatorial optimization. arXiv 2025.\n\nFrom my point of view, I think the capability of LLMs to continuously improve and learn from experience is more important than zero-shot generation from prompts for algorithm/heuristic design tasks. It is similar to how human experts design high-performance algorithms through trial and error. Moreover, the results of different LLMs are not robust given different initial prompts, while the performance improvement is always observed during evolution/iterative search. However, assessing this learning capability (or, more broadly, open-ended problem-solving) in a principled manner presents a significant challenge, given its dynamic and costly evolutionary nature.", "limitations": "", "rating": 6, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "LLM-driven heuristic design benchmarking is vital for both algorithm development and LLM communities, serving as an open-ended, challenging, and evaluative benchmark. \n\nThree stages with new criteria are designed for this benchmark.", "weaknesses": "Further clarification is needed regarding the methodology for designing the benchmark, specifically concerning the tasks, prompts, and evaluation procedures. \n\nAs a benchmark paper, more comprehensive results, including a greater number of iterations and diverse prompt strategies, are expected.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "teXYlWNSsc", "reviewer": "Reviewer_QPGo", "summary": "The paper introduces HeuriGym, an agentic framework designed to systematically evaluate large language models (LLMs) on heuristic generation tasks within combinatorial optimization. Its major contributions can be summarized as follows:\n\nThe authors propose a new agentic framework that formalizes the process of LLM-driven heuristic generation through iterative prompting, execution, and feedback. Unlike single-turn benchmarks, HeuriGym allows LLMs to iteratively refine their algorithms by integrating structured feedback such as execution logs, verification outcomes, and cost evaluations. This closed-loop setup mirrors human-like problem-solving and provides a rigorous testbed for assessing LLM reasoning capabilities in constrained optimization settings.\n\nThey also introduce a novel set of evaluation metrics beyond traditional pass@k, most notably the Solveₛ@i metric, which measures whether an LLM can solve constrained problems within a given number of iterations across execution, solution generation, and verification stages. To assess performance quality, they define the Quality-Yield Index (QYI), combining success rate and relative solution quality to reflect both feasibility and optimality.\n\nIn addition, the paper presents a comprehensive benchmark suite of nine combinatorial optimization problems across domains such as electronic design automation, compilers, computational biology, and logistics. These problems were carefully selected to avoid overfitting to well-known tasks (like TSP or SAT) and emphasize reasoning over memorization. Each benchmark includes demonstration and evaluation sets to support few-shot and iterative learning analysis.\n\nFinally, the authors conduct a large-scale empirical evaluation comparing several state-of-the-art LLMs—including GPT-4-mini, Gemini-2.5-Pro, Claude-3.7-Sonnet, and DeepSeek models—under the HeuriGym framework. The results show that while models like GPT-4-mini and DeepSeek-R1 achieve strong reasoning consistency, overall performance still lags behind expert-designed heuristics, highlighting the current limitations and future potential of LLMs for agentic reasoning in optimization tasks.", "questions": "1. In the demonstration set, only five small instances were used for each task. This sample size seems too limited to capture the variability and difficulty spectrum of real-world combinatorial problems. Could the authors clarify the rationale behind choosing such a small number of demonstration instances, and whether larger or more diverse demonstration sets were considered?\n\n2. The paper mentions applying a citation-based filtering criterion (less than 1,000 citations) when selecting benchmark problems. This threshold appears somewhat arbitrary and potentially large, as problems with hundreds of citations could already be well-studied. What is the justification for using this particular cutoff, and how does it effectively ensure that the benchmark tasks are not overexposed or memorized by LLMs?\n\n3. What does large solution space mean? It's not rigorously defined.", "limitations": "", "rating": 4, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "- **Well-presented and systematic framework:**  \n  The paper is clearly organized and methodologically rigorous, providing a formalized agentic framework (HeuriGym) that integrates problem definition, heuristic generation, verification, and iterative feedback in a coherent workflow.\n\n- **Goes beyond standard benchmark tasks:**  \n  Unlike prior works focusing on well-studied problems such as TSP or SAT, HeuriGym introduces diverse, domain-rich combinatorial optimization tasks (e.g., operator scheduling, protein sequence design, airline crew pairing), enhancing the benchmark’s generality and realism.\n\n- **Novel evaluation metrics capturing agentic reasoning:**  \n  The introduction of `Solveₛ@i` and the Quality-Yield Index (QYI) provides a more holistic evaluation than standard `pass@k`, accounting for feasibility, constraint satisfaction, and iterative refinement quality.", "weaknesses": "- **Lack of comparison with ILP solvers:**  \n  The paper claims that the proposed framework can solve larger instances than traditional ILP solvers such as Gurobi. However, no quantitative comparison is provided to substantiate this claim. For smaller-scale problems, it would be important to report the optimality gap or at least benchmark against Gurobi solutions to demonstrate relative performance.\n\n- **Missing evaluation against state-of-the-art heuristics:**  \n  While the framework aims to generate heuristics automatically, it remains unclear whether the resulting solutions improve upon or even match the performance of existing state-of-the-art heuristic algorithms in the literature. Without such comparison, it is difficult to gauge whether HeuriGym advances the frontier of heuristic design or merely replicates baseline performance.\n\n- **Marginal novelty in the agentic framework:**  \n  The proposed agentic framework, although well structured, shows limited conceptual innovation beyond existing approaches such as AlphaEvolve. The iterative feedback and refinement loop are largely similar in spirit, and the paper would benefit from introducing more distinctive mechanisms or theoretical insights that clearly differentiate it from prior work.\n\n- **Limited engagement with more advanced related work:**  \n  Recent studies have explored more sophisticated integrations of LLMs with optimization processes. For instance, *van Stein et al. (2024)* proposed an in-the-loop hyperparameter optimization approach for LLM-based heuristic design (*ACM Transactions on Evolutionary Learning*), which achieves a tighter coupling between LLMs and evolutionary optimization. Compared with such work, HeuriGym appears relatively simplistic, relying mainly on AlphaEvolve-style iteration without deeper algorithmic co-adaptation or automated search over LLM configurations.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)