QiMeng-NeuComBack: Self-Evolving Translation from IR to Assembly Code
Authors: Hainan Fang
Yuanbo Wen
Jun Bi
Yihan Wang
Tonghui He
Yanlin Tang
Di Huang
Jiaming Guo
Rui Zhang
Qi Guo
Yunji Chen
Venue: NeurIPS 2025 poster ()
Keywords: Large Language Models
Neural Compilation
Compiler
Code Generation
Automatic Prompt Learning
Primary area: deep_learning
TL;DR: We introduce NeuComBack, a novel benchmark for Neural Compilation, and propose an automatic prompt learning method that significantly improves both correctness and optimization of LLM-generated assembly.

Abstract: Compilers, while essential, are notoriously complex systems that demand prohibitively expensive human expertise to develop and maintain. The recent advancements in Large Language Models (LLMs) offer a compelling new paradigm: Neural Compilation, which could potentially simplify compiler development for new architectures and facilitate the discovery of innovative optimization techniques. However, several critical obstacles impede its practical adoption. Firstly, a significant lack of dedicated benchmarks and robust evaluation methodologies hinders objective assessment and tracking of progress in the field. Secondly, systematically enhancing the reliability and performance of LLM-generated assembly remains a critical challenge. Addressing these challenges, this paper introduces NeuComBack, a novel benchmark dataset specifically designed for IR-to-assembly compilation. Leveraging this dataset, we first define a foundational Neural Compilation workflow and conduct a comprehensive evaluation of the capabilities of recent frontier LLMs on Neural Compilation, establishing new performance baselines. We further propose a self-evolving prompt optimization method that enables LLMs to iteratively evolve their internal prompt strategies by extracting insights from prior self-debugging traces, thereby enhancing their neural compilation capabilities.
Experiments demonstrate that our method significantly improves both the functional correctness and the performance of LLM-generated assembly code. Compared to baseline prompts, the functional correctness rates improved from 44% to 64% on x86_64 and from 36% to 58% on aarch64, respectively. More significantly, among the 16 correctly generated x86_64 programs using our method, 14 (87.5%) surpassed clang-O3 performance. These consistent improvements across diverse architectures (x86_64 and aarch64) and program distributions (NeuComBack L1 and L2) validate our method's superiority over conventional approaches and its potential for broader adoption in low-level neural compilation.

QiMeng-NeuComBatee Self-Evolving Translation from IR to Assembly Code
Hainan Fang!.?, 1.2, Yuanbo Wen', Jun Bi', Yihan Wang Tonghui Hel Yanlin Tang* Di Huang', Jiaming Guo", Rui Zhang', Qi Guo', Yunji Chen
Stttt Key Lab of Processors, Institute of Computing Technology, Chinese Academy of Sciences, Beijing, China University Chinese Academy of Sciences, Beijing, China
(Fanghainan23s, wenyuanbo, bijun, wangyihan25s, hetonghui23s, tangyanlin21s, huangdi, guojiaming, zhangrui, guoqi, cyj}@ict.ac. cn
Abstract
Compilers, while essential, notoriously complex systems that demand pro- hibitively expensive human expertise to develop and maintain. The recent ad- vancements in Large Language Models (LLMss) offer compelling Neural Compilation, which could potentially simplify compiler development for new architectures and facilitate the discovery of innovative optimization tech- niques. However, several critical obstacles impede its practical adoption. Firstly, a significant lack of dedicated benchmarks and robust evaluation methodologies hinders objective assessment and tracking of progress in the field. Secondly, sys- tematically enhancing the reliability and performance of LLM LLM- generated assembly remains a critical challenge. Addressing these challenges, this paper introduces NeuComBack, a novel benchmark dataset specifically designed for IR-to-assembly compilation. Leveraging this dataset, we first define foundational Neural Compila- tion workflow and conduct comprehensive evaluation of the capabilities of recent frontier LLMs Neural Compilation, establishing further propose a self-evolving prompt optimization method that enables LLMs to iteratively evolve their internal prompt strategies by extracting insights from prior self-debugging traces, thereby enhancing their neural compilation capabilities. Ex- periments demonstrate that our method significantly improves both the functional correctness and the performance of LLM -.M--eeeerteed assembly code. Compared to baseline prompts, the functional correctness rates improved from 44% to 64% on x86_66 64 and from 36% to 58% on aarch64, respectively. More significantly, among the 16 correctly generated x86 64 programs using method, 14 (87.5%) surpassed clang-03 performance. These consistent improvements across diverse architectures (x86 64 and aarch64) and program distributions (NeuComBack LI and L2) validate our method's superiority over conventional approaches and its potential for broader adoption low-level neural compilation.
paradigm:
performance baselines. We
Introduction
Compilers are indispensable and highly complex software systems, meticulously engineered over decades to translate high-level programming languages into low-level machine code executable by specific hardware architectures (Lattner & Adve 2004). This process involves multiple stages of analysis, transformation, and optimization, often requiring deep expertise and significant development effort. In recent years, Large Language Models (LLMs) have made significant progress in natural language processing (OpenAl 2024b Liu et 2024
Yang et 2002aa Grattafiori et al. 2024)
"Corresponding Author.
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

and increasingly excel at code-related tasks such as auto- completom,, program synthesis, and bug detection. sparking interest in their broader software engineering applications (Hou et al. 2024 Wang
et al. 2024 Rozière et al. 2024 Zhong & Wang 2024).
Given their proficiency in handling complex code-related tasks, a natural and compelling research direction is the exploration of LLMs as core components in compilation (i.e., Neural Compilation). By directly translating high-level source code or intermediate representations (IR) into low-level assembly code using LLMs, Neural Compilation has the potential to augment or even replace traditional compiler stages. Compared to rule-based compilation, Neural Compilation has two appealing benefits. Firstly, it can drastically reduce the time and effort needed to build compilers for emerging Instruction Set Architectures (ISAs) (Armengol Esstae & O O'Boyle| (202]) Secondly, can uncover new optimization strategies based on imput program semantics due to LLM's ability to process code as lossless text (Cummins et al. 2023). Thus, Neural Compilation has the potential to transform how programming language researchers and hardware architects explore new designs.
Despite the escalating interest and conceptual feasibility demonstrated by early works (Armengol- Estapé & O' O'Boyle| 2021), Neural Compilation still faces major practical challenges, particularly in ensuring lunctiionl correctness and optimizing performance. First, achieving semantic equivalence comparable to traditional compilers remains a substantial challenge, even for LLMs trained on vast code corpora (Cummins et al., 2024). Second, while LLMs might uncover novel optimization strategies (Cummins et al., 2023). consistently surpassing the optimization levels of mature compilers (e.g., clang- -3) is an even greater hurdle. Moreover, the absence of systematic benchmarks for evaluating L.MS' Neural Compilation capabilities hinders objective progress assessment. Addressing this gap requires (1) a well-defined dedicated benchmark for consistent and measurable criteria, and (2) methods to improve the reliability and performance of LLM-aided neural compilation.
To this end, in this paper, we first introduce NeuComBack, a novel benchmark dataset specifically designed for evaluating IR-to-assembly compilation. Derived from ExeBench Bernengol-Estappe al. 2022) and TSVC (Maleki et al., 201T). NeuComBack provides a diverse set of programs to systematically assess fundamental compilation and optimization capabilities. Note that the benchmark focuses on the critical backend compilation task of translating LLVM IR hardware -pre-s assembly code across different ISAs. Using this benchmark, we conduct systematic evaluation of state-of-the-art I.LMs, including DeepSeek-R1 (Guo al., 2025), establishing erucial performance baselines that were previously unavailable in this emerging research area. Second, we define foundational Neural Compilation workflow, encompassing generation and optimization steps, which serves as a basis for our experiments and can inform future research in this domain. Building upon this workflow, we propose an automatic prompt learning method for assembly generation that learns from the LLM's self-debugging traces, systematically extracting insights from generation and correction attempts and iteratively evolving internal prompt strategies.
The main contributions of this work are summarized as follows:
We introduce NeuComBack, novel benchmark dataset tailored for the LLVM IR-to- assembly Neural Compilation task. Based on it, we conduct a comprehensive evaluation of state-of-the-art LLMs, establishing critical and previously unavailable baselines.
We propose self-evolving prompt optimization technique that enables LLMs to auto- matically improve their compilation capability through iterative refinement. By analyzing compilation errors and performance optimization trails, our method dynamically adapts prompt strategies, enhancing both the correctness and the performance.
We conduct extensive experiments in terms of the functional correctness and the performance of LLM LeM--gerrrtted assembly code. Compared to the baseline prompt, correctness increased from 44% to 64% (x86 64) and 36% to 58% (aarch64). More importantly, 14 of 16 correct x86_64 64 programs (87.5%) surpassed the performance of clang-03. These results demonstrate consistent gains across diverse architectures and benchmarks, validating our approach's 's superiority in neural compilation.
2

---

2 Related Work
2.1 Neural Compilation
Machine learning has long history in compiler optimization, traditionally focusing heuristics for tasks like pass selection and phase ordering (Wang & O' O'Boyle 2018; Agakov et al.) 2006 Trofin ct al. 2021). A more ambitious direction, Neural Compilation, aims for end-to-end assembly code generation. Early efforts in this vein employed neural machine translation techniques. For instance, Armengol--sttaeé & O' O'Boyle (202T) pioneered direct C-to-x86 assembly translation using Trans- formers, and (Hosseini & Davan-Gavitt 2022: Eimengol-Estappe et al. 2024) developed specialized
models for decompilation.
The advent of LLMs has significantly advanced neural compilation. Cummins et al. (2033) demon- strated that LLMs, trained on millions of LLVM assembly and corresponding optimal compiler optimizations, could predict and perform the beneficial optimizations. Based on that, Cummins et al. (2024) further introduced LLM Compiler, a family of foundation models pretrained on compiler IRs and assembly. These models facilitate downstream tasks like compiler flag tuning and disassembly. LLMs are also increasingly applied other compiler-level tasks such as sophisticated decompilation
(Tan et al., 22024] Wong et al., 2023) and compiler fuzzing (Deng et al. 2023 Yang et al. 2024b).
Fang & Mukhanov (2224) explored whether advanced reasoning mechanisms, like chain-of chain-of-thought
(Wei et al. (022), could enhance LLM performance in applying single peephole optimization on
AArch64 assembly. Their findings suggest that conventional LLMs, even when fine-tuned, struggle
due to lack of reasoning, and that models augmented with explicit reasoning capabilities (e.g..
GPT-ol-poe -previewrr can significantly outperform them et al. 2023a).
Our work builds upon these diverse efforts in leveraging machine learning for compilation. While prior studies have demonstrated the initial feasibility of neural translation for compilation and showcased the growing power of LLMs in both optimizing and generating low-level code, achieving robust functional correctness and competitive performance simultancously in Neural Compilation remains a primary research objective. As defined in Section 3 our research aims at translating LLVM IR to assembly code, targeting both functional correctness and performance optimization.
2.2 Automatic Prompt Learning
The performance of Large Language Models (LLMs) heavily depends input prompts, making prompt engineering crucial. While manual techniques like zero-shot, few-shot prompting (Radford Brown et al., 2020, Wei et al. 2024 Liu ct al. 2022), and Chain-of-Thoughh (CoT) (Wei
ct al.
2019
et al.
2022 Zhang et al. 2022 Zhao et al. 2024 Zhou et al. 2023) Wang et al.) 20236) exist, their
laborious creation and often suboptimal inconsistent results have spurred research into Automatic
Prompt Optimization (APO).
APO methods algorithmically discover effective prompts through iterative refinement, sometimes using evolutionary approaches or L.MSs as optimizers (Zhou et al. 2022 Pryzant et al. [023 Yang
et al. 2023) Guol2 et al.) (Shum et al. 2024: Sun et al. (003). Gao et al. (2025) introduced MAPS for test case generation, focusing on LLM -ailored prompts through diversity, domain knowledge, and failure-driven rule induction. Ye et al. (0255) proposed Prochemy for code generattion/taanlatiio using exccution- -druvie iterative refinement to produce a fixed, reusable, optimized prompt. These highlight trend towards automated, task-aware, and model-specific prompt engineering for code.
Our Automatic Prompt Learning (APL) method (Section 4 optimizes prompts for neural IR-to- assembly compilation, differing from general APO techniques by uniquely learning from the complete self-debugging trails. To the best of our knowledge, there are no research studies that specifically investigate the impact of automatic prompt learning on driving LLMs to generate assembly code. Experiments (Section 5) show that the proposed method effectively enables L.LMs to learn from past practices, enhancing both the correctness and optimization quality of the generated assembly code.
3

---

3 Neural Compilation Task
3.1 Problem Formulation
Let Peource denote program expressed in a source representation, which can be either a high-level programming language LHL (c.g.. C, Python) or an Intermediate Representation (IR) CıR (c.g.. LLVM IR). Let Atarget be the corresponding program represented in a low-level assembly language LaS specific to target hardware architecture M (e.g., x86_64, aarch64).
Formally, the task of Neural Compilation is defined as learning a mapping function fNC. parameter- ized by a Large Language Model (LLM) with parameters 8, which directly translates input program Psoarcn into an output assembly program Atarget: Atarget
.NC( (P Mourree,;; 0), Peouree E
(CH,,Cin), Atarget LASM.
This translation explicitly conditions on the target architecture M, as assembly syntax and semantics inherently architecture-dependee A successful Neural Compilation system must satisfy two critical objectives simultancously:
Functional Correctness:
The generated assembly program Aurget must maintain semantic equivalence to the source program Psource- Formally, this means that for any valid input or execution context, the observable behavior and final computational state must match pre- cisely: [Arrgel] = [ where [I-] denotes the semantic interpretation of program, and [II. specifically refers to execution semantics on the target hardware architecture M.
Performance Optimization: Beyond correctness, the generated assembly code should ex- hibit high performance. Performance metrics include execution speed (runtime), instruction count, code size, memory footprint, and energy consumption. Ideally. the assembly output from the LLM (Auagget should be competitive with or outperform code produced by highly op- timized traditional compilers, such as those using the -03 flag: c( (AAugee)) S c( (Asoomileettss) where c(.) denotes performance cost function applicable to the target architecture.
Building on this formulation, we introduce a foundational workflow for Neural Compitation that systematically addresses both correctness and performance, drawing from common patterns in LLM-based code generation and optimization. The workflow follows the general structure below:
Initial Generation: Given an input program P. Prource. the LLM generates an initial assembly candidate A conditioned on the target architecture M. If the initial generation fails to produce functionally correct program, the process terminates early.
Iterative Optimization: Starting from A the model performs up to T rounds of optimization. In each round t = 1,2,..., T, a new candidate A is generated based on (1-1) aiming to improve performance. target
Note that after each generation or optimization step, self-dehugging procedure may be applied to verify and enforce functional correctness.
3.2
NeuComBack Dataset
3.2.1 Data Collecting
Our primary objective construct a comprehensive dataset for the LLVM IR to Assembly (ASM) compilation task. To the best of our knowledge, no publicly available datasets specifically focus on this IR-to-ASM translation. Consequently, we adapted existing C-to-ASM benchmarks. We selected ExeBench (Armengol Estapé et al. 2022), a widely used collection of C programs, and the Test Suite for Vectorizing Compilers (TSVC) (Maleki et al. (01T), a compiler performance benchmark. The distinct characteristics of these sources naturally lead to two-tiered dataset:
Level (Fundamental Compilation, 200 tasks): test suite covers broad variety of real-world C programs and features diverse control- flow patterns. significant portion of these programs, derived from embedded systems applications, exhibits intensive I/O operations. While extremely complex control flow (c.g..
This selection from the ExeBench

---

deep recursion, concurrency) is underrepresented in available sources, L1 is designed with breadth-first focus assess ceneral-purposs functional correctness.
Level 2 (Optimization Potential, 151 tasks): This level, drawn from the TSVC benchmark suite, features programs characterized by computationally simple execution paths but no- table loop intricacy. Such programs provide a strong basis for assessing the optimization capabilities reflected in the generated assembly.
We curated two benchmark levels: NeuComBack-LLA comprising 200 ExeBench test cases selected for the longest IR lengths from a cleaned set of 1,618; and NeuComBack-L22 the entire TSVC benchmark (151 cases). While both levels have similar lines of code, Clang AST analysis shows TSVC programs average significantly more variables (26.06 vs. 13.73 for ExeBench). Crucially for LLM-based compilation, TSVC's nested loop structures (prevalent in L2) present far greater difficulty than ifs or gotos due to complex register allocation and scheduling, making L2 a more demanding benchmark.
3.2.2 Data Cleaning
The original ExeBench dataset presented challenges in terms of code quality and complexity. Many programs were notably short (as illustrated in Appendix A Figure 2 and the original authors employed g++ -fpermissive for compilation due to numerous deviations from C/C++ standards. To address this, we focused on the ExeBench test set, performing substantial rewriting ensure C/C++ standard compliance. A critical step in our data cleaning pipeline was to rigorously verify that clang could successfully compile each modified program into LLVM IR.
Furthermore, to mitigate the issue of programs being overly simplistic or short, we filtered the cleaned ExeBench data. We specifically selected programs that, when compiled by clang, produced the longest LLVM IR sequences. This selection process enriches our Level 1 dataset with more substantial and representative examples for the IR-to-ASM task, resulting in a set of 200 programs.
3.2.3 Evaluation Metries
The dataset adopts two primary metrics, designed to reflect the dual goals of Neural Compilation: achieving functional correctness and generating high-perfone regh-performance assembly code.
This metric measures the percentage test cases where the LLM-ggnerated assembly code produces the exact same output as the reference assembly code compiled by clang -00 (or another appropriate baseline for correctness) for a given set of inputs. For the TSVC dataset, we enhanced its original correctness verification methodology. While the existing checks might suffice for traditional compilers, they are less adequate for LLMs, where errors manifest at any instruction. Our improved validation is more rigorous in detecting such fine-grate mre-grained discrepancies.
Functional Correctness (ACC):
This metric quantifies the percent- age of programs that not only functionally correct but also exhibit execution performance superior to that achieved by clang -03. This directly assesses the LLM's capability to generate optimized assembly.
Correctness with Superior Performance (ACC+Perf):
For ExeBench-derived cases, we reuse the dataset's official inputs and checkers, with only lightweight wrappers adapted to accommodate our IR AR-M setting. For TSVC, to avoid false positives from degenerate constant initializations, we replace fixed-value array initializations with deterministic pscudorandom values drawn under a fixed seed, providing diverse inputs while preserving repro- ducibility.
Performance evaluations are conducted exclusively on the Level (TSVC) dataset, as the Level I (ExeBench) dataset, though containing diverse real-world programs, was not primarily designed for performance evaluation, making its programs unsuitable for such analysis. To ensure reliable performance measurements, we execute each program (both LLM -LM-generated and the clang -03 baseline) 11 times. We then discard the first three (warm-up) and last three (cool-down) runs, retain runs 4-8 (middle five by execution order), and report the median of these five execution times (median computed by their execution time cost) to mitigate system noise and provide a stable performance figure.
5

---

Training
Debog Traile
Estract Imights
Esa ciiiiing gld spole
Dann befie wall
Showe global any levaniie fire
f
Compilation
Exolution
laitial
New
Finial Chippt
Ntagy Offfiie Learaing
Finial Osiput
(/>
Source Code
Compilation
Optimization
Salsceng
Amembbl Code
Iterate Gimes
Failed
Easly Step
Stage Online Inference
Figure Pipeline our automatic prompt learning method on Neural Compilation.
4
Method
4.1
Overview
Our Neural Compilation framework leverages novel automatic prompt-learing mechanism specifi- cally tailored to improve L.M-generated assembly code. Distinct from previous prompt evolution methods, our approach's
insight lies in learning from the L.M''s complete iterative self-debugging process, which effectively enables the LLM to learn from its past practices in diagnosing and resolving complex errors in assembly code generation. Our method comprises two distinct stages, illustrated in Figure [ offline prompt learning and online inference. In the offline stage, the model iteratively evolves prompts based on comprehensive analysis and insights derived from past generation trials. Subsequently, in the online stage, the model utilizes these refined prompts to generate and iteratively optimize assembly code, progressively improving quality and performance.
4.2
Self-Evolving Prompt Optimization for Neural Compilation
4.2.1
Offline: Prompt Learning
The offline learning phase focuses automatically discovering and evolving prompts to effectively guide the LLM toward generating correct and performant assembly code. This process begins with initializing empty prompt template. Following this, self-debug trials are collected by using the LLM to perform a "compilation" process and then test the generated assembly translations; errors trigger iterative self-debugging. refining code until correctness iteration limits reached. Subsequently, critical insights are extracted by analyzing the complete self-debug trajectories to identify error patterns and effective strategies via LLM-assisted analysis. Finally, prompt evolution occurs by refining prompts through the integration of experience and insights (extracted from a batch of self-debug trials), which are then reviewed for clarity and effectiveness.
Concretely, we maintain single evolving prompt π. During offline learning, after each mini-batch of compilation tasks perform update of म: collect self-debugging trajectories from those programs that were initially incorrect but became correct via self-debug, and we ask the same model used for compilation to act "prompt optimizer." A meta-prompe instructs it to analyze the trajectories, extract error patterns and effective fixes, and propose edits to the existing prompt æ to reflect these insights. After the LLM suggests revisions, comparison and confirmation step follows: we prompt the LLM to review the change list and confirm only the necessary and beneficial edits, adding stability to the evolution process. The updated is then used for the next mini-batch.
4.2.2 Online: Inference
The online inference stage deploys evolved prompts from offline learning to iteratively generate and optimize assembly code. First, the initial assembly generation is performed, using evolved prompts to generate initial assembly code from IR. This initial assembly is then tested, and iterative self-

---

debugging is triggered until functional correctness is achieved; if self self-debug fails after all attempts, the generation is terminated, and a failure is reported. Next, iterative optimization is applied further refine the initial assembly code, with evolved prompts still provided to minimize error introduction. Each optimization iteration includes testing and self-debugging as necessary to ensure continued conrectness Finally, the process outputs the optimized assembly code, demonstrating competitive or superior performance compared to compiler-generated code.
Experiments
5.1 Empirical Evaluation of Existing Large Language Models
To establish performance baseline for current L.LMs on the Neural Compilation task, we conducted an empirical study evaluating several state-of-the-arr LLMs. The pri- mary objective was to assess their capability in translating programs from an IR to correct and performant x86_6e 64 assembly code. We utilized NeuComBack-L2 as the main test data for our experiment. For each of the 151 test cases, each LLM was tasked to generate x86 64 assembly code, involving an initial attempt followed by a maximum of two rounds of self-debugging. We evaluated five models: GPT-4o OOpenAl 2024b), O3-Mini (OpenAl 2025), O1 (OpenAl 2024a), DeepSeek- V3 (Liu et al. 2024). and DeepSeek-RI (Guo et al. (025))
Table 1: Baseline performance of ad- vanced LLMs, NeuComBack-L2 (over-
all 151 cases), x86_66 64
Model
ACC
ACC+Parf (S)
GPT-4o O3-Mini
1.99 (3/151)
0.66 (1/151)
21.19 (32/151)
5,30 (8/151)
01
19.87 (30/151)
5.30 (8/151)
DeepSeck-V3 DeepSeck-R1
14.57 (22/151)
3.31 (5/151)
45.70 (69/151)
21.85 (33/151)
The prompt for baseline assessment is detailed in Appendix The results are summarized in Table DeepSeek-R1 demonstrated the strongest baseline, achieving 45.70% functional correctness and outperforming O3 in 21.85% of cases. Other reasoning models like O3-Mini and 01 also showed notable capabilities. Among models without specialized, extensive reasoning training, DeepSeek-V? performed best.
The experimental results lead to several key observations. Firstly, advanced LLMs, particularly those enhanced with reasoning capabilities like DeepScek-RI, exhibit substantially improved performance in the Neural Compilation task of (R-to-assembly translation compared to previous models (e.g.. GPT-4o). DeepSeek-RI's ability to correctly generate assembly for nearly half of the benchmark cases, and for a significant portion of those to be more performant than -O3, a promising devel- opment. Secondly, even for the ress-eeforning model, achieving functional correctness remains a significant hurdle, and surpassing traditional compiler optimizations is an even greater challenge. The performance of DeepSeck-V3 suggests that foundational model capabilities are improving. These baseline results highlight both the potential of state-of-the-art LLMs in compiler tasks and the substantial room for improvement necessary for practical, widespread adoption.
5.2 Evaluation of Self-Evolving Prompt Optimization
Table 2: Performance of automatic prompt learning vs. baseline on NeuComBack-LL (test set, 40 sam- ples), x86_64, DeepSeek-RI.
Building on this empirical study, we evaluate the impact of our proposed automatic prompt learning method, using the DeepSeek-R. -R model due to its superior performance. We assess its effectiveness on both NeuComBack-LJ and NeuComBacc--L2, targeting the x86_ 64 architecture. The NeuComBack-LI dataset was divided into 120 training sam- ples, 40 for validation, and 40 for testing, while NeuComBack-LL2 comprised 101 training samples, 25 for validation, and 25 for testing. For both datasets, we conducted prompt learning over three epochs with a batch size of 5, additionally introducing 1 self-debugging round per generation for NeuComBack-LI and 2 rounds for NeuComBack-L2. From this process, we selected the highest- performing prompt based on validation metrics (referred to as the "Learned Prompt"), which we compare against the baseline prompt detailed in Appendix B
Method
ACC (%)
Baseline
50.00 (20/40)
80.00 (32/40)
Consequently, for NeuComBack-LI, we evaluate functional correctness (ACC) as its programs, derived from ExeBench, serve as fundamental test cases for basic compilation correctness. For NeuComBack-132 sourced from TSVC and featuring programs with higher loop complexity and optimization potential, we assess both ACC and ACC+Perf (correctness with superior performance over clang -03).
7

---

Results on Neu As shown in Table 2 the application of the learned prompt led to substantial improvements: the number of functionally correct solutions increased from 20 (50.00% ACC) with the baseline prompt to 32 (80.00% ACC), with a relative 60% improvement.
Results NeuComBack-I.2. We con- ducted two sets of experimental setups: (a) using a baseline prompt for both initial generation and subsequent optimization, and (b) using the learned prompt (see Ap- (n, performing up to two self-debug rounds after each generation or optimiza- tion step." Table 3 details the results on the NeuComBack-L22 benchmark ("-" indi- cates no change compared to the previous stage). In the initial generation phase, our method increased correctly solved programs on the NeuComBack-L2 test set from 11 (44.00% ACC) with the baseline prompt to 16 (64.00% ACC), achieving a 45% relative improvement, and solutions outperforming -03 increased from 6 to 10 (a relative improvement of approximately 67%) When evaluating the full Neural Compilation workflow, we observe that the advantages of the learned prompt are not merely preserved but substantially amplified through iterative optimization. The baseline prompt produced 7 solutions (28% ACC+Perf) that outperformed the 03 optimization level, whereas the learned prompt consistently generated 14 such solutions (56% ACC+Perf), yielding 100% relative improvement in segh-performance assembly code generation. This significant improvement arises from the learned prompt's 's ability to guide the model effectively across the entire compilation process. By steering the model away from common suboptimal decisions and error patterns, the learned prompt enhances the optimizer's 's overall effectiveness, leading to consistently superior results.
Table 3: Performance of automatic prompt learning vs. baseline prompt on NeuComBack-L.2 (test set, 25 samples), x86_6 64, DeepSeck-RI.
Method & Stage
ACC (%)
ACC+Perf (%)
Baseline Prompt
After Initial Generation
44.00 (11/25)
24.00 (6/25)
After Rounds lter. Opt.
28.00 (7/25)
Our Method (Learned Prompt)
After Initial Generation
(16/25)
40,00 (1025)
After 2 Rownds Iter. Opt.
56.00 (14/25)
noteworthy fact is that, among the 16 programs correctly generated using learned prompts, 14 (87.5%) were further optimized to surpass clang-03 03 performance (Detailed speedups are reported in Appendix D). This remarkably high conversion rate demonstrates that when properly guided through iterative refinement, LLMs achieve not just functional correctness but genuine optimization mastery in complex assembly-level tasks, a capability that appears substantially underutilized in current approaches. Furthermore, the consistent performance improvements across both datasets validate the robustness and general applicability of our prompt learning methodology, suggesting its potential for broader adoption in low-level code optimization scenarios.
Meanwhile, the consistent positive impact across both datasets also underscores the robustness and general applicability of our learning approach.
5.3 Ablation
Method Effectiveness on Different Instruction Sets. To evalu- ate adaptability to different hardware, we experimented with the aarch64 instruction set using the DeepSeck-R I model and our NeuComBack- L.2 dataset (101 train, 25 validation, 25 test). The learning process involved 3 epochs with 4 self-debugging rounds per generation. The prompt was chosen the same. As detailed in Table this approach significantly enhanced performance on the test set: functional correctness doubled from 9 to solutions (a 100% relative improvement), and solutions surpassing -03 performance increased from 2 to 7 (a 250% relative improvement), demonstrating effectiveness for aarch64.
Table 4: Effectiveness of auto-
matic prompt learning. DeepSeek- R1, NeuComBack-LL (test set, 25 samples), aarch64
Method
ACC (%)
ACC+Perf (%)
Baseline
(9/25)
8.00 (2/25)
Our 72.00 (18/25)
28 (7/25)
Transferability of Learned Prompts across Different Data Distributions. We investigated prompt transferabil- ity by applying an x86 64 prompt learned on NeuComBack-L2 directly to NeuComBack-LI (120 training, 40 validation, 40 test samples) without fur-
Table 5: Performance on NeuComBack-Ll (x86_6n 64, DeepSeek-RI) using different prompt strategies, showing test set and overall ACC
Prompt Strategy
Test Set ACC (%)
Overall ACC (%)
Default Prompt
50.00 (20/40)
54.50 (109/200)
Learned NeuComBacckLLi
80.00 (32/40)
Learned on NeuComBack-L2
67.50 (27/40)
74.50 (149/200)
ther learning. As shown in Table the
NeuComBack-L22
prompt achieved 67.50% functional correctness the NeuComBack-
8

---

LI test set, a 35% relative improvement over the default prompt's s 50.00%. While still lower than the 80.00% from prompt learned specifically on NeuComBack-LI. it demonstrates positive transferability. suggesting the learned prompts can capture generalizable patterns across program distributions.
Effectiveness in Reducing Self-Debug Rounds. We analyzed the average number of self-debug rounds for tested programs that were eventually solved correctly. This evaluation was per- formed across different architectures and datasets, comparing our method against the baseline. As shown in Ta- b our method consistently reduces the average number of self-debug rounds needed. These results indicate that, in addition to achieving higher final functional correctness (as demonstrated previous sections), our automatic prompt learning approach enables the LLM to converge on correct assembly code more efficiently, requiring fewer self-correction attempts.
Table 6: Average self-debug rounds for successfully com- piled programs by DeepSeek-RI on test sets, comparing our method with the baseline. Lower better.
Architecture
Dataset
Max Dehug Rounds
Avg. Self-Deehog Rounds
Buseline
Our Method
NeuComBaac-LI NeuComBack-1122
0.90
0.28
x86 .4
2
1.09
0.25
sarch64
NeaComBaac-1122
6
Case Studies
Analysis of LLM-generated Code on Performance As shown in experiment (Section 5 LLMs demonstrate a promising ability to optimize assembly code, sometimes achieving performance superior to traditional compilers. Below, we present examples that illustrate the specific optimization techniques employed by LLM. For instance, LLMs can enhance performance by reducing the instruction count. Consider the NeuComBack-LL2 function s452 in Figure 3 While both the original LLV --geerrrtee version and the LLM-oppimized version exhibit effective vectorization, the former contained redundant addition operations in its detailed implementation. These redundancies could be eliminated by pre-calculating constants. The LLM successfully implemented this optimization, notably reducing two paddd (packed doubleword add) instructions to a single one within the inner loop. This demonstrates the LLM's capability to iteratively refine and further optimize its own generated code.
example, shown
Another illustrative example is s332, shown in Figure In this case, the LLM achieves optimizations superior to those produced by LLVM's 03 optimization level. When tasked with finding the first index of an array element satisfying an inequality condition, LLVM 03 employs sequential comparisons within the inner loop, without leveraging vectorization. In contrast, the LLM leverages vector instructions such as cmpps (compare packed peelee-preciion floating- point values). This enables the simultaneous comparison of four floating-point numbers within the inner loop. Furthermore, employs the bsfl (bit scan forward, least significant) instruction to efficiently extract the index from the resulting comparison mask, thereby achieving functional equivalence with the original code but with significantly improved performance.
Analysis of the Learned Prompts The effectiveness of our automatic prompt learning method hinges on the quality and specificity of the evolved prompts. To shed light on what the LLM learns and incorporates into its guiding instructions, this subsection provides content analysis of the learned prompts. A complete example of a learned prompt, illustrating these aspects in full, can be found in Appendix
In our iterative prompt engineering process, the LLM was progressively enhanced by integrating a comprehensive set of summarized rules. These rules were meticulously designed to address various dimensions of code generation, aiming to systematically elevate the quality, consistency, and accuracy of the generated outputs. By codifying best practices into structured guidelines, the process not only refined the LLM's syntactic and semantic understanding but also established a robust framework for maintaining alignment with specific coding standards and conventions.
Table 7: Cases of rules learned
These rules spanned mul- tiple aspects of code gen- eration, as shown in Ta- ble 7 Firstly. formatting rules were established, man- dating critical principles
Placing function code exclusively in text section
Formatting
Pusitioning siie directives immediately after function hodies
Using L prefix with exact spelling for local labels
Syntatic
Append PLT suffix to external function calls (c.g. 'call (ummy@PPTT
Semantic
Clear return registers (XORL &eax, &eax) for void functions
Pass stack based parameters reverse order with aligament padding

---

like the distinct separation of data and code segments, and the precise IEEE 754 binary repre- sentation for dloating-piim numbers. Secondly, syntactic rules were introduced govem structural correctness, exemplified by the specifications for XMM register indexing and conventions. Thirdly, semantic rules provided guidance and hest-practice recommendations regarding the nuanced usage of particular instructions, thereby improving the LLM's s understanding of instruction semantics.
7
Conclusion
This paper advances the field of Neural Compilation by introducing a dedicated IR-to- -t-AAsseblly benchmark and a novel self-evolving prompt optimization method, enabling more systematic and rigorous evaluation of LLM-based compilation techniques. We first introduce NeuComBack, benchmark specifically designed for IR-to-assembly compilation, and conduct a comprehensive evaluation of state-of-tie sare--fftth--arr LLMs to establish performance baselines. We further propose self- evolving prompt optimization method that enables LLMs to iteratively improve their compilation strategies by learning from self-dcbugging traces. Our experimental results demonstrate significant improvements in both functional correctness (increasing from 44% to 64% on x86 6_64 and from 36% to 58% aarch64) and optimization performance (with 87.5% of correct x86 _6444446 programs surpassing cl ang. These consistent gains across multiple architectures and benchmark distributions validate the effectiveness of our approach and its potential advance lowelevel neural compilation.
While our method achieves promising results, there remains room for improvement in both functional correctness and optimization performance. Future work will focus on further refining these LLM- driven compilation techniques and expanding benchmarks to encompass more complex real-world compilation scenarios.
8 Acknowledgement
This work is partially supported by the Strategic Priority Research Program of the Chinese Academy of Sciences (Grants No.XDD0660300, XDB0660301, XDB0660302), NSF of China (Grants No.62525203, 62302483, U22A2028, 6240073476), the Science and Technology Major Special Program of Jiangsu (Grant No. BG2024028), CAS Project for Young Scientists in Basic Research (YSBR-029) and Youth Innovation Promotion Association CAS.
References
Agakov, F., Bonilla, E.. Cavazos, J., Franke, B., Fursin, G., O'Boyle, M. F., Thomson, J., Toussaint, M., and Williams, C. K. Using machine learning focus iterative optimization. In International Symposiun on Code Generation and Optimization (6G)066) pp. pp. IEEE, 2006.
Himengol-Estapee J. and O' Boyle, M. F. Learning x86 translation: An experiment in neural compilation. arXiv preprint arXiv:2108.076.33 07639,,,, 2021.
Armengol-Fstape J., Wondruff, J., Brauckmann, A., Magalhães, J. W. d. S., and O'Boyle, M. F. Exehench: an ml-scale dataset of exccutable functions. In Proceedings of the 6th ACM SIGPLAN International Symposium on Machine Programming, pp. 50-59, 2022.
Armengol- -stappé, J., Woodruff, J., Cummins, C., and O'Boyle, M. F. Slade: A portable small language model decompiler for optimized assembly. 2024 IEEE/ACM International Symposium on Code Generation and Optinization (CGO), pp. 67-80. 80. IEEE, 2024.
Brown, T. Mann, B., Ryder, N., Subbiah, M., Kaplan, D., Dhariwal, Neelakantan, A., Shyam, P., Sastry, G., Askell, A., et al. Language models are fow-shot leamers. Advances in neural information processing systems, 33:1877-1901, 2020.
Cummins, C., Sceker, V., Grubisic, D., Elhoushi, M., Liang, Y., Roziere, B., Gehring, J., Glocckle, F., Hazelwood, K., Synnaeve, G., et al. Large language models for compiler optimization. arXiv preprint arXiv:2309.0MM 2023.
Cummins, C., Seeken, V., Grubisic, D., Roziere, B., Gehring, J.. Synnaeve, G., and Leather, H. Meta large language model compiler: Foundation models of compiler optimization. arXiv preprint arXiv:2407.02524, 2024.
10

---

Deng, Y., Xia, S. Peng, H. Yang, C., and Zhang, L. Large language models are zero-shot fuzzers: Fuzzing deep-leaming libraries via large language models. In Proceedings of the 32nd ACM SIGSOFT international sympusium on seftware testing and analysix, pp. 423-435, 2023.
Fang, X. and Mukhanov, L.. Towards Ilm-based optimization compilers. can 1lms learn how to apply a single Ilms need! arXiv preprint arXiv:2412.12163. 2024.
peephole optimization? reasoning
Gao, S., Wang, C., Gao, C., Jiao, X., Chong, C. Y., Gao, S.,, and Lyu, M. The prompt alchemist: Automated llm-tailored prompt optimization for test case generation. arXiv preprint arXiv:2501.00 01332,, 2025.
Grattafiori, A., Dubey, A., Jauhri, A., Pandey, A., Kadian, A., Al-Dahle, A., Letman, A., Mathur, A., Schelten, A., Vaughan, A., et al. The Ilama herd of models. arXiv preprint arXiv:2407.21783, 2024.
Guo, D., Yang. D., Zhang, H., Song, J., Zhang, R., Xu, R., Zhu, Q.. Ma, S., Wang. P., Bi, X., et Deepseek-rl: Incentivizing reasoning capability in ulms via reinforcement learning. arXiv preprint arXiv:2501. 2025.
Guo12, Q.. Wang, R., Guo, J., Li23, B., Song, K., Tan, X., Liu, G., Bian, J., and Yang. Connecting large language models with evo-lutionary algorithms yields powerful prompt optimizers.
Hosseini, I. and Dolan- Beyond the c: Retargetable decompilation using neural machine translation. arXiv preprint arXiv:2212.0000:: 029500 2022.
Hou, X., Zhao, Y., Liu, Y., Yang, Z., Wang, K., Li, L., Luo, X., Lo, D., Grundy, J., and Wang. H. Large language models for software engineering: A systematic literature review, ACM Transactions on Software Engineering and Methodology, 1338)81179 2024.
Lattner, C. and Adve, V. Llvm: A compilation framework for lifelong program analysis & transformation. In International symposium code generation and optimization, 2004. CGO 2004., pp. 75-86. IEEE, 2004.
Liu. A., Feng, B., Xue, B., Wang. B., Wu, B., Lu, C., Zhao, C., Deng, C., Zhang, C., Ruan, C., et al. Deepseek-v3 technical report. arXiv preprint arXiv:2412.1 24...1442777 2024.
Liu, H. Tam, D., Mugeeth, M.. Mohta, J. Huang. T., Bansal, M.. and Raffel, C. Few-shot parameter-efficient fine-tuning is better and cheaper than in-context learning. In Proceedings of the 36th International Conference on Neural Information Processing Systems, NIPS '22, Red Hook, NY, USA, 2022. Curran Associates Inc. ISBN 9781713871088.
Maleki, S., Gao, Y.. Garzar, M. J., Wong, T., Padua, D. A., et al. An evaluation vectorizing compilers. In 2017 International Conference on Parallel Architectures and Compilation Techniques, pp. 372-382. IEEE, 2011.
OpenAI, Openai ol system card. arXiv preprint arXiv:2412.16720 2024a.
OpenAT, Gpt-4o system card, arXiv preprint arXiv:2410.. 140.22276, 2024b.
OpenAL. Openai o3-mini system card, Technical report, OpenAI, January 2025. httphhttp://ceee openai
com/ o3- mini- system- card- feb10. pdf Published on January 31, 2025. Accessed: 2025-05-16.
Pryzant, R., Iter, D... Li, J., Lee. Y. T., Zhu, C., and Zeng. M. Automatic prompt optimization with" gradient descent" and beam search. arXiv preprint arXiv:2305.034955 2023.
Radford, A., Wu, J., Child, R., Luan, D., Amodei, D., Sutskever, I., et al. Language models unsupervised multitask learners. OpenAl blog, 188)9 2019.
Rozière, B., Gehring, Gloeckle, F., Sootla, S., Gat, I., Tan, X. E., Adi, Y., Liu, J., Sauvestre, R., Remez, T., Rapin, J.. Kozhevnikov, A., Evtimov, I., Bitton, J., Bhatt, M., Ferrer, C. C., Grattafiori, A., Xiong, W., Défossez, A., Copet, J., Azhar, F,, Touvron, H., Martin, L., Usunier, N., Scialom, T., and Synnaeve, G. Code llama: Open foundation models for code, 2024. URL htllhhttss///rrtte
org/ abs / 2308. 12950
Shum, K., Diao, S., and Zhang, T. Automatic prompt augmentation and selection with chain-of- ofouthtttttttttthh from labeled data, 2024. URL https:tt/// /// http:/xxivvv..r wwtprrg//bba////// abs/ /to.b/22222222 12822
Sun, H., Li, X., Xu, Y., Homma, Y., Cao, Q. Wu, M., Jiao, J., and Charles, D. Autohint: Automatic prompt optimization with hint generation, 2023. htl.sttts:/// //aavvv..//iv..aaaa orwoorgggbbo........o...e tata//307..00 .04155
Tan, H., Luo, Q., Li, J., and Zhang, Y. Lim4decompile: Decompiling binary code with large language models. arXiv preprint ar XxXx:24033002880 2024.
Trofin, M. Qian, Y. Brevdo, E., Lin, Z., Choromanski, K., and Li, D. Mlgo: machine learning guided compiler optimizations framework. arXiv preprint arXiv:2101.04888 2021.

---

Wang. B., Yue, X., and Sun. H. Can chaigpt defend its belief in truth? evaluating Ilm reasoning via debate. 2023a. URL buttps://arii.. www.orgggbb..... abs/ 2305 13160
Wang, J., Huang, Y., Chen, C., Liu, Z., Wang, S., and Wang, Q. Software testing with large language models: Survey, landscape, and vision. IEEE Transactions Software Enggineering, 2024.
Wang. X., Wei, J., Schuurmans, D., Lc, Q.. Chi, E., Narang, S., Chowdhery, A., and Zhou, D. Solssscooyy improves chain of thought reasoning in language models, 2023b. URL https://arxiv org/ ab//abbs/2222 2203
1111
Wang, Z. and O'Boyle, M. Machine learning in compiler optimisation. arXiv preprint arXi:: 2018.
Wei, J., Wang. X., Schuurmans, Bosma, M.. Xia. F., Chi, E., Le, Q. Zhou, D., et al. Chain-of- thouhht prompting elicits reasoning in large language models. Advances in neural information processing systems, 35: 24824-24837, 2022.
Wei, X., Cui, X., Cheng, N., Wang, X., Zhang. X., Huang, S., Xie, P., Xu, J., Chen, Y., Zhang, M.. Jiang. Y. and Han, W. Chatie: Zero-shot information extraction via chatting with chatgpt, 2024. URL https // //arxiv..v org/abs 2302. 10205
Wong, W. K., Wang, H., Li, Z., Liu, Z., Wang. S., Tang. Q., Nie, S., and Wu, S. Refining decompiled code with large language models. arXiv preprint arXiv:2310.0655: 2023.
Yang, A., Yang, Zhang, B., Hui, Zheng, Yu, B., Li, C., Liu, D., Huang, Wei, H., et al. Qwen2. technical report. arXiv preprint 1511:2412.15115, 2024a.
Yang. C.. Wang. X., Lu. Y. Liu, H., Le. Zhou, D., and Chen, X. Large language models optimizers. arXiv preprint arXiv:2309.034444 2023.
Yang, C., Deng, Y., La, R., Yao, J., Liu, J., Jabharvand, R., and Zhang, L. Whitefox: White-box compiler fuzzing empowered by large language models. Proceedings the ACM on Programming Languages, 8(OPPSLA2): 709-735, 2024b.
Ye, S., Sun, Z. Wang, G., Guo, L., Liang, Q. Li, Z., and Liu, Y. Prompt alchemy: Automatic prompt refinement for enhancing code generation. arXiv preprint arXiv:2503.11085, 2025.
Zhang, Z., Zhang, A., Li, M., and Smola. A. Automatic chain of thought prompting in large language models. 2022. htLphttts///aaat org/ abs /bw./2/......ee.aap..ppaaaaaaa.a 2210. 03493
7hao, X., Li, M., Lu, W., Weher, C., Iee, J. H., Chu, K., and Wermter, S. Enhancing zero-shot chain-of tfothhtt reasoning in large language models through logic, 2024. URI. /ttt:s:ss:/: org/ httgg//b//309..... 2309. 13339
Can Ilm replace stack overflow? a study on robustness and reliability of large language model code generation. In Proceedings of the Hhirty-Eightt AAAI Conference on Artificial Intelligence and Thirty-Sixth Conference on Innovative Applications of Artificial Intelligence and Fourteenth Symposium on Educational Advances in Artificial Intelligence, AAAI244AAAI24EAAPP22 AAAI Press, 2024. ISBN 978- 1- 57735-887-9. doi: 10. 16166699aaii,. ht00///a...38.9...... v38119,30185.
Zhong, L. and Wang, Z.
URL htt.hhtts:///oo org/ 10. 1609/ aaai. v38119..... 30188
Zhou, Y., Muresanu, A. I., Han, Z., Paster, K.,, Pitis, S., Chan, H., and Ba, J. Large language models human-leve prompt engineers. In The Eleventh International Conference on Learning Representations, 2022.
Zhou, Y., Geng, X., Shen, T., Tao, C., Long, G., Lou, J.-G., and Shen, J. Thread of thought unraveling chaotic contexts, 2023. httpstt::: arwaa.ivvvv.. wrgg
www.//ggg///./eeo.oe//..o...ee/a abs/ 2t1//221110000 08734
12

---

A Original ExeBench C Code (Test Set) Statistics
The original ExeBench dataset, prior to our cleaning and filtering process, contained large number of C programs. As discussed in Section 3.2 (specifically the Data Cleaning subsubsection), many of these programs were notably short. Figure provides visual representation the code line count distribution within the original ExeBench dataset (test set),
meaningtul kemnls
600
500
Numbor of cands
402
100+
40
60
100
Number of lines
Figure 2: Original ExeBench C code statistics. This figure illustrates the distribution of C code lines in the programs from the original ExeBench dataset (Test Set).
B
Baseline Prompt
This appendix shows the baseline prompt utilized for the x86 64 architecture. This prompt serves as the initial, unoptimized of instructions given the Large Language Model for translating LLVM IR AT&T format assembly code.
c<cllvm ir
.ce
"assembly
<cc
You are an expert in IR code and assembly code. Please help me translate the given IR code into x86_64 CNU assembly code (using AT&T format).
Note that:
1. You MUST use the following template to give out the complete, final target assembly code, and MUST NOT apply this template to any other part of your
response:
'assembly
(Provide the assembly code here)
2. Do not return any information other than the assembly code.
C Example of Learned Prompt
This appendix presents an example of prompt that was evolved using our automatic prompt learning method. Specifically, this prompt is the result of the learning process conducted the x86 64 architecture using the NeuComBack-L.2 dataset, following the experimental practices described in Section 5.2
c<cllvm ir rcc
''assembly
13

---

are an expert IR code and assembly code. Please help me translate the given IR code into x86_64 GNU assembly code (using AT&T format).
Note that:
1. You MUST the following template give the complete, final target assembly code, and MUST apply this template to any other part your
response:
"assembly
(Provide the assembly code here)
"<"
2. not return any information other than the assembly code.
3. Additionally, to guarantee the correctness of the generated assembly code, please ensure that:
Correctly structure sections and assembler directives by:
Placing function code exclusively in text section and constant data in rodata (or other appropriate sections)
Maintaining strict section boundaries no intermixing of code and data decionations between sections
Positioning size directives immediately after function bodies within text section, before any data/comm declarations or section transitions Placing comm directives exclusively in BSS context (typically bas section), never within text .rodata
Using prefix with exact spelling for local labels and ensuring sensitive consistency in all references
Use
addressing for global symbols:
Access all global data exclusively via RIP-relative addressing using 'leaq symbol (Krip), 'reg''
addresses in registers accesses, maintaining separate registers for distinct arrays
Store global array
LEAQ before indexed
Use register+index addressing (e.g., "XXbase_reg, Kindex, scale) with scaling factors matching element size (4 for 32-bit floats/ints)
Never combine absolute symbol names with index registers; base registers initialized vía LEAQ for
indemed accesses
Maintain proper register usage and stack management:
Preserve callee-saved registers (rbx/ri2-r15) via push/pop sequences when modified, with dedicated registers for persistent data like array bases and loop counters
Use 64-bit registers (rsi/rdi/etc) for loop counters indices handling ranges exceeding 32-bit capacity, initializing with XORQ for zeroing Store loop counters in callee-saved registers separate from array pointers using distinct registers for different control variables
Maintain 16-byte stack alignment by calculating adjustments
regitt_reg parameters_size + 15) & "15 before calls. Ensure alignment persists after push/sub operations, especially when storing XMM registers with movaps
registera (XMMO for floats, RAX/EAX for integers) BEFORE restoring callee-saved registers function epilogue
Transfer final reaulta
Clear return registers (XORL Xeax) for void functions before RET and validate all exit paths set EAX/RAX
Explicitly extend 32-bit values to 64-bit via MOVSLQ/CDQE when using 32- bit operations with 64-bit registers
Handle function calls and external references correctly:
Append GPLT suffix to external function calls (e.g., 'call dummy@PLT') Preserve XMM registers XMM6-XXX15 across calls via save/restore sequences if reused post-call. When storing computed values these registers, originals stack with movaps (aligned) or movups (unaligned), then restore before returning
Load parameters into registers via 'leaq symbol (Krip) immediately before call setup to minimize register pressure
order with aligument padding, recalculating offaets after stack adjustments to ensure correct argument positioning
Pass stack-based parameters in
14

---

Never assume XMMO-5 retain values across function calls; explicitly preserve results callee-saved registers if needed
Maintain strict data type consistency:
Match IR operations with correct instruction types (e.g., use addss for float addition vs. addl for integers)
Represent immediate float constants hexadecimal literals (e.g., x3f800000 for 1.0f) instead of separate memory constanta when possible
Handle residual elements using same data type as vectorized operations XMM registers for floats)
Use MOVSS for 32-bit float transfers between memory/XMN registers instead of integer NOV variants
Implement loop semantics accurately:
Place loop initializations inside headers when IR indicates per-iteration requirements (PHI-node dependencies)
Reinitialize array bases/values in outer loops if inner loops modify
global state
Maintain separate vectorized (16-element) scalar residual processing
paths
Track register/nemory modifications through nested loops to prevent cross- iteration dependencies
Handle vector operations correctly:
Use SIMD registers matching operation width (XMM for 128-bit ops) Verify memory alignment against IR attributes before using aligned accesses (movaps)
Default unaligned accesses (movups) when alignment isn't exxpliicci guaranteed
using htw.://cc.sssnacccaaaa.o..oppppp scalaar,oorrationn
aatching.originnll.aaaaa....
D Detailed Runtime Performance Results
Table 8 reportts per-program speedups of method (learne--rrompt, DeepScek-RI relative to clang -03 for the 16 functionally-ccoree programs referenced Section (.2.oon552220 (after initial generation and 2 rounds of itcrative
optimization). Speedup is defined as Speedup RumimeLLM so values >> 1 indicate our code runs faster than -03. median
Runineg:
Runtimes follow the same protocol as in Section each program is executed times and we of the middle runs.
Table 8: Per-program speedups (LLMM over clang -03) on NeuComBack-L12 (x86_64). Average speedup across these 16 programs is 1.28 x; maximum is 3.25x (s331).
Program
Speedup
s331
3.2485
s1232
2.0621
s332
1.9954
s112
1.1048
s152
1.0852
$2251
1.0543
s451
1.0288
s319
1.0278
vdotr
1.0277
s3112
1.0275
s4121
1.0274
s231
1.0253
s $113
1.0247
s422
1.0224
$452
0.8621
s341
0.7924
15

---

E Illustrative Examples of LLM-Driven Code Optimization
This appendix provides the visual representations (Figures 3 and for the code optimization studies discussed in Section Figure showcases the optimization function s452. where the Large Language Model (LLM) reduced the instruction count. Figure Fillustrate the optimization function a332, where L.LM achieved performance superior to LLVM O3 by employing vector instructions. These figures compare the original LLVV--generaie assembly code with the LLM-oppimized version.
movdqa .LCO (Xrip),
%xmm8
movdqa LC1 (/rip),
%xmm9
movdqa LC5 (/rip),
%xmm10
paddd Vxmm8, %xmm4 paddd Xxmm9, 7xmm4
movdqa LCC (/rip), %xmm 10
movdqa .C11 (/rip), %xmm 1
paddd /xmm8, /xmm5
paddd 4xmm10, 7xmm5
paddd Zxmm10, %xmm4
sectiin .rodtta
paddd %xmm11, 7xmm5
aalign 16
.LC:
section .rodata
.long 0, 1, 2, 3
allign 16
.C1:
.CC:
.long
1, 1, 1, 1
long 1, 2, 3, 4
LC4:
LC1:
long
4, 4, 4,
.long 5, 6, 7, 8
(a) original version
(b) optimized version
Figure A case of performance self-optimization (s452)
moves
-16(Krbx Xrax, Xxmm1 Xxmm0, Xxmm1
uconiss
ja
LBBO_ _10
moves
-12 (Krbx, 4), Xxmm1 Yxmm0, Xxmm1
uconiss
LBBO_9
movups (%r14, /r13, 14), %xmm0
-8 (Xrbx, Krax, Xxmm1 Xxmm0, Xxmm1
cmpps $6, /xmm1, %xmm0
ja
movmskps /xmm0,, %eax testl %eax, %eax jnz boound_in_vectill
moves
-4 (XrrxxXrax, 4), Xxmm1 \xımo, Xxmm1
uconiss
ja
LBBO_12
(a) original version
(b) optimized version
Figure 4: A case of performance better than 03 (s332)
F
Cost-Controlled Model Comparison
We first estimated the relative inference cost and found that a single experimental run with DeepSeek-RI is approximately 14 times expensive than with DeepSeek-V3. Based on this finding, we conducted a cost-controlled evaluation. We allocated DeepSeek-V3 14-fold computational budget, allowing it perform match the of single run of DeepSeek-RI. All other experimental settings remain the same those used Section on NeuComBack L2. In our original experiments, DeepSeekRI solved 11 out of 25 problems (achieving functional correctness). However, under this new cost-cont cost-controlee setting, DeepSeckV3 V3 successfully solved 17 out of 25 problems.
sampling,
16

---

Table Cost-controlled comparison on NeuComBack-L2 (test set, 25 cases).
Setting
Problems Solved (ACC)
DeepSeek-RI (single run)
11/25
DeepSeck-V3. V3 (14x sampling)
17/25
G
Limitations
While this work advances Neural Compilation new benchmark and self-evolving prompt optimization method, we also highlight several limitations.
IR scope. Our study focuses on LLVM IR, for is the most general Intermediate Representations, used by various programming languages, so the techniques are likely to generalize to other IRs, particularly those based on Static Single Assignment (SSA) form; however, generalization is not guaranteed. The design choices and assumptions in method may not hold equally across all IR designs, especially those with significantly different semantics control-flow structures.
Benchmark scale. NeuComBack is moderate size and not intended for pre-training or fine-tuning, designed primarily for evaluation: cach task includes a verifiable test hamess for automatic correctness (and, for L2, performance) measurement, and the dataset size keeps the cost/time of running frontier models practical and acceptable.
Benchmark diversity. NeuComBack-L1 focuses on basic correctness; MeuComBack-L2 (TSVC-derived) emphasizes loop-centric, vectorization-friene kemels that expose clear optimization headroom. This leaves less coverage other compiler challenges (e.g.. pointer-heavy code, irregular control flow, recursion, dynamic mem- ory, interprocduraal/ccdllayyooo optimizations, register pressure, and nemory-hherandee effects). Consequently, ACC+Perf gains on L2 should not be over-interpreted universal across all optimization classes.
We will work to address these limitations in future work.
17

---

NeurIPS Paper Checklist
Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: The abstract and introduction accurately describe the paper's is contributions, including the NeuComBack dataset, the baseline LLM evaluations, the proposed automatic prompt learning method, and the experimental results demonstrating its effectiveness in improving IR-to-assembly compilation. These are detailed in Sections (Introduction), 3.2 (NeuComBack Dataset), (Method).
Guidelines:
The answer NA means that abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contri- butions made in the paper and important assumptions and limitations. No NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected generalize to other settings.
is fine to include aspirational goals as motivation as long as it clear that these goals are not attained by the paper.
Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer: [No]
Justification: The paper does not have a dedicated "Limitations" section. However, potential limitations that could be discussed include:
Scope of NeuComBack Benchmark: The current NeuComBack benchmark primarily lever- aging ExeBench and TSVC as data sources and currently focuses on the Re-to-assembly tramslation task. Future work could expand it with more data and additional compilation- related tasks (c.g., different IRs, other optimization stages).
The proposed automatic prompt learning method has shown strong results. Nevertheless, it could be further extended by exploring more sophisticated prompt cvolution strategies, potentially incorporating techniques like genetic algorithms or other search heuristics to navigate the prompt space more broadly.
Extensibiliity of the Prompt Learning Method:
Computational Cost: The automatic prompt learning process, involving multiple LLM inference calls for generation, self-debugging, and insight extraction, can be computationally intensive. The associated costs might pose practical challenge for very large-scale prompt searches when using highly expensive LLM APIs.
Generalizability to Weaker Models: The current method demonstrates significant effec- tiveness with recent, advanced LLMs (e.g.. DeepSeek-RII RR). R.). Its efficacy and the transferability of learned insights might be more limited when applied to smaller or less capable models, which would require further investigation.
Guidelines:
The answer NA means that paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors encouraged create separate "I "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust results are to violations of these assumptions (c.g., independence assumptions, noiseless settings, model well-specifien cell--pecfficatonn. asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would he.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance the approach. example, facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or speech-to-tex, system might not be used reliably to provide closed captions for online lectures because it fails handle technical jargon.
18

---

The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address prohlems of privacy and faireess.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection. worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community, Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory assumptions and proofs
For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Question:
Answer: [NA]
The paper presents an empirical study with a new benchmark dataset (NeuComBack) and a novel methodology (automatic prompt learning) for assembly generation. It does not propose new theoretical results that would require formal assumptions and proofs. The problem formulation in Section 3.1 defines the task but does not introduce theorems.
Justification:
Guidelines:
The answer NA means that the paper does not include thcoretical results.
All the theorems, formulas, and proof's in the paper should be numbered and cross- referenefddd
All assumptions should be clearly stated referenced in the statement of any thcorems.
The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should he complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Question:
Answer: [Yes]
Justification: The paper descrihes the dataset creation process (Section 3.2), evaluation metrics (Section 3.2.3), LLMs used (Section 5.1). experimental setup including data splits (Section 5.1, 5.2), prompt learning parameters (epochs, batch size, self-debug rounds, Section 5.1). and architectures tested (x86 64, aarch64). Examples baseline and learned prompts are also stated in Appendices and B, which are crucial for understanding the inputs to the LLM.
Guidelines:
The answer NA mcans that the paper does not include experiments.
If the paper includes experiments, No answer this question will not perceived well by reviewers: Making the paper reproducible important, regardless of whether the code and data are provided or not.
If the contribution is datasct and/or model, the authors should describe the steps taken make their results reprodueible verifiable.
Depending on the contribution, reproducibility can he accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, if the contribution is specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, or provide access to the model. In general. relcasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to hosted model (c.g., in the case of large language model), releasing of model checkpoint, or other means that appropriate to the research performed.
While NeurlPS does not require releasing code, the conference does require all submissions to provide some reasonable avenue for reproducibility. which may depend on the nature of the contribution. For example
19

---

(a) If the contribution is primarily new algorithm. the paper should make it clear how to reproduce that algorithm.
(h) If the contribution is primarily new model architecture, the paper should descrihe the architecture clearly and fully.
(c) If the contribution either way access this model for reproducing the results a way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
new model (e.g., large language model), then there should
(d) We recognize that reproduciiility may be tricky in some cases, in which case authors are welcome to descrihe the particular way they provide for reproducibility. In the case of closed-source models, may he that access to the model limited in way (e.g.. to registered users), but it should possible other researchers have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data code, with sufficient instructions to Faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [No]
Justification: We plan to release our dataset and code upon acceptance; currently no public repository is linked. We will include link in the final version.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines htttpe:/niips.cc/peeee
guides/Code CodeSubmi as onsoolicy for more details.
While we encourage the relcase of code and data, we understand that this might not be possible, so "No" is an acceptable answer. Papers cannot rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).
The instructions should contain exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https /ttps..ccppbllcc//// http:/ppbbcggguiddleee wtw.:dddee.oodeSuiiiiiiiiisss onP www.sssionooloiooooo olicy) for more details.
The authors should provide instructions on data access and preparation. including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts reproduce all experimental results for the new proposed method and baselines. only a subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, preserve anonymity, the authors should release anonymized versions (if applicable).
Providing much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental sefting/details
Question: Does the paper specify all the training and test details (c.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary understand the results?
Answer: [Yes]
Justification: Section 5 (Experiment) details the data splits for NeuComBack-L.I and NeuComBack-I.. the L.MMs used (specifically DeepSeck-R1 for the core method evaluation), the number of epochs and batch sizes for prompt learning, the number of self self-debugging rounds allowed generation. and how the "learned prompt" selected (best performing the validation set). This provides good understanding the experimental procedure.
Guidelines:t
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, or as supplemental material.
7. Experiment statistical significance
20

---

Question: Does the paper report error bars suitably and correctly defined or other appropriate informa- lion about the statistical significance of the experiments?
Answer: [No]
The paper reports performance metrics as percentages and raw counts (e.g. "28.00% (7/25)") in tables such as Table 1, 2, 3, 4, and 5. However, does not include error bars, confidence intervals, or formal statistical significance tests for these results. Conducting multiple rums for each experimental condition to gather data for robust statistical significance analysis was deemed prohibitively expensive. This is primarily due to the high cost of LLM API calls, particularly when employing advanced reasoning models for complex, token- -iteenintensive tasks such as IR-to-assembly translation, where each sample can consume a significant number of tokens.
Justification:
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results accompanied by error bars, confidence intervals, or statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the error bars capturing should be clearly stated (for example, train/test split, initialization, random drawing of parameter, overall with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to library function, bootstrap, ctc.)
The assumptions made should given (e.g.. Normally distributed errors).
It should be clear whether the error bar is the standard deviation or the standard error of the mean.
is OK to report 1 I-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).
If error bars reported in tables plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables the text.
8. Experiments compute resources
Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Answer: [No]
Justification: The paper specifies the Large Language Models used (e.g.. DeepSeck-RI, GPT-4o). It is clarified that model inference for these LLMs was performed via API calls. Consequently, the specific underlying compute hardware (e.g., GPU type, memory the provider's side) for the inference step is managed by the API providers.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU GPU, internal cluster, cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g.. preliminary or failed experiments that didn't 't make it into the paper).
9. Code of ethics
Question: Does the research conducted the paper conform. every respect, NeurlPS Code Fthics
Answer:
[Yes]
The research focuses on LLM-based code generation for compilers, using publicly available benchmarks (ExeBench, TSVC) to create a new dataset. It does not involve human subjects, sensitive personal data, or applications with immediate high-risk ethical concerns outlined in the NeurlPS Code of Ethics. The goal to advance compiler technology.
Justification:
21

---

Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics.
The authors should make sure preserve anonymity (e.g., if there is a special consideration due to laws or regulations their jurisdiction).
10. Broader impacts
Question: Does the paper discuss hoth potential positive socictal impacts and negative societal impacts of the work performed?
Answer: [No]
The Introduction (Section 1) of the paper extensively discusses several potential positive societal impacts of Neural Compilation. These include its potential to streamline compiler develop- ment for novel architectures, reduce the time and effort needed for creating compilers for emerging Instruction Set Architectures (ISAs), facilitate the discovery of innovative optimization techniques. and transform how programming language researchers and hardware architects explore new designs. However, the paper does not explicitly discuss potential negative societal impacts. Examples of such impacts that could be relevant include the potential for deskilling in the compiler engineering workforee if traditional roles are significantly altered, the risk of misuse if the technology were adapted generate obfuscated malicious code (though this is not the paper's 's focus), the environmental impact related to the energy consumption training and deploying large LLMs for these compilation tasks.
Justification:
Guidelines:
The answer NA means that there is no societal impact of work performed.
If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address socictal impact.
Examples of negative socictal impacts include potential malicious or unintended uses (c.g., disinformation, generating fake profiles, surveillance), fairness considerations (c.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate point out that an improvement the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there are negative socictal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mecha- nisms for monitoring misuse, mechanisms monitor how system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g.. pretrained language models, image generators, scraped datasets)?
Question:
Answer: [NA]
Justification: The paper utilizes existing Large Language Models (c.g., DeepSeek- R1, GPT-4o) and introduces a benchmark dataset (NeuComBack) derived from public coding benchmarks. It does not release new foundational pretrained language models or image generators itself, nor does it use scraped datasets that would pose a high risk for misuse terms of sensitive content or PII. The primary artifacts are the methodology and the benchmark.
Guidelines:
The answer NA means that the paper poses such risks.
22

---

Released models that have high risk for misuse or dual-use should be released with necessary safeguards allow for controlled of the model, for example by requiring that users adhere to usage guidelines or restrictions access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging. and many papers do not require this, but encourage authors take this into account and make best faith effort.
12. Licenses for existing assets
Question: Are the creators or original owners of assets (c.g., code, data, models), used in the paper, properly eredited and are the license and terms of usc explicitly mentioned and properly respected?
Answer: [Yes]
Justification: The paper properly credits the creators of the datasets used: ExeBench (Armengol -ertengol-Estapé ct al. 2022) and TSVC (Malcki et al. [ cited in Section 3.2.1. According to public sources, both datasets are released under open-source licenses, and their use in this paper complics with the respective licensing terms.
Guidelines:
The answer NA means that paper does not use existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of asset is used and, possible, include a URL..
The name of the license (c.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (c.g., website), the copyright and terms of service of that source should be provided.
assets are released. the license. copyright information, and terms of use the package should be provided. For popular datasets, pppprswi saperswithcode com/ datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset.
For existing datasets that re-packaged, both the original license and the license of the derived asset (if has changed) should be provided.
If this information is not available online, the authors encouraged to reach out the asset's creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [Yes]
The paper introduces the NeuComBack dataset, and its construction, sources, levels, and metrics are documented Section 3.2. The proposed automatic prompt learning method is described in Section 4. The paper itself serves as the primary documentation for these new assets. If these assets were to he released, further practical documentation (e.g., README files) would typically accompany them.
Justification:
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/codeemmodel as part of their sub- missions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset used.
At submission time, remember anonymize your assets (if applicable). You either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and sereenshots, applicable, as well as details about compensation (if any)?
Answer: [NA]
23

---

Justification:
The research described in the paper does not involve crowdsourcing experiments or research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information supplemental material is fine, if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According the NeurIPS Code Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals or equivalent for research with human subjects
Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Question:
Answer: [NA]
Justification: The rescarch does not involve human subjects and therefore did not require IRB approval equivalent review.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Depending on the country in which research conducted. IRB approval (or equivalent) may be required for any human subjects research. you obtained IRB approval, you should clcarly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do include any information that would break anonymity (if applicable), such as the institution conducting the review.
16. Declaration of LLM usage
Question: Does the paper describe the usage of LLMs if it important, original, non-standard component
core methods this research? Note that if the LLM used only for writing. oditing, or formatting purposes and does not impact the core methodology, scientific rigorousness, or originality of the rescarch, declaration is not required.
Answer: [Yes]
Justification:
The use of Large Language Models (LLMs) such as DeepSeek-RI fundamental and component of this rescarch. The entire paper is about "Neural Compilation," defined as using LLMs to generate assembly code, and proposes an "automatic prompt learning method" specifically enhance LLM performance this task. This is detailed throughout paper. particularly in Sections (Introduction). (Neural Compilation Task), 4 (Method), and 5 (Experiment).
Guidelines:
The answer NA means that core method development in this research does not involve LLMs as any important, original, or non-standard components.
for Please refer to LLM policy hhatpp://bearrpss.cc////ffeeeeee what should should not be described.
24


Reviews:
[{"review_id": "NwRX4pi2Gq", "reviewer": "Reviewer_7UhH", "summary": "The authors highlight a gap in the literature regarding the generation of assembly by large language models, specifically a dedicated benchmark for the neural compilation paradigm. While there exists previous work that has targeted high-level to assembly (C-to-x86_64), in this work, the authors focus on LLVM Intermediate Representation to x86_64 and aarch64. To produce a benchmark, the authors consider two sources of problems: ExeBench and TSVC, with TSVC being more challenging and targeting performance evaluation rather than purely correctness by creating problems where vectorisation is expected. To assess how well LLMs perform, they focus on two metrics: functional correctness (modulo a test suite) and correctness with performance gain. The authors further argue that the ExeBench problems are insufficiently complex to measure performance gain; therefore, the second metric only applies to problems derived from TSVC. To improve/control the impact of the prompt, the authors also devise an Automatic Prompt Optimisation flow that uses debug information from previous runs to refine the LLM prompt for the task. The authors start from an initial prompt template, which is then gradually evolved by asking the LLM to abstract information from debug traces and incorporate it into the prompt. The authors evaluate five models: GPT-4o, O3-Mini, O1, DeepSeek-V3, and DeepSeek-R1. No model achieves a performance higher than 50%, suggesting the benchmark is difficult. Further, performance improvement is only observed for ~22% of the cases (33 out of 151). To demonstrate the value of the prompt evolution, the authors conisder the strongest performing model (DS-R1) and show that, on 40 samples that were kept as a test set, the model improves significantly from 50% functional correctness, to 80%; noteworthy here is that the model demonstrates the ability to perform novel vectorisation optimisations when using the evolved prompt in conjucture with two rounds of self-debug. This example, and an additional one where precomputing a constant allows merging multiple add instructions into a packed double-word add, were used as case studies to demonstrate the discovered optimisations.", "questions": "Q1. In the paper, you discuss the various cost functions that can be used for performance evaluation; however, it remains unclear which exact cost function was used for this work. Details in §3.2.3, line 220, suggest run-time recordings with appropriate burn-in (ramp-up/ramp-down). Is this the only component of the cost function, or were others, such as instruction numbers, some weighted instruction sum (for example, accounting for micro-code), etc., also used?\n\nQ2. For functional correctness, my understanding from the paper is that it is done via differential testing against a known good compiler. How are the driving inputs chosen? Do you perform random testing? Fuzzying? Some other technique (for example, manually pre-selected edge case inputs)?", "limitations": "The authors properly address limitations from the machine learning side: does the learnt prompt learn transferable/general patterns? Is the self-debug mechanism useful for the task? And some software engineering limitations: does the result generalise to other architectures (or at least one other architecture)?, are the learnt optimisations interesting (the case study)? \n\nOne limitation, which exists by design, is not addressed. This work targets LLVM IR to assembly, and there may be a risk that the results do not generalise to other IRs. I believe this risk is minimal, and the results should generalise to other Static Single Assignment form IRs; however, this limitation should be mentioned in the work.", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 4, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "+ S1:  Novel benchmark that is not saturated (best model is <50%)\n+ S2: Targetting the LLVM IR allows the higher-level programs to come from a variety of languages as long as they provide an LLVM compiler (flang, clang, etc.)\n+ S3: Example of a Domain Specific APO (though the claim in S2.2 is extremely specific (Line 128), making the novelty claim trivially true; I think there is merit in the approach, however, that paragraph made it _seem_ overly defensive.)\n- W1: Some design decisions are unclear: why is the initial generation + iterative optimisation chosen? Is the output stable under iterations, i.e. are the changes by the LLM minimal in some sense (such as instructions changed)?\n- W2: Missing details concerning input generation for functional correctness. I would guess equivalence testing under a fuzzing framework; however, I have to guess.\n\n+/- (Mixed/S4/W3): They detail the experimental design for the runtime evaluation (performance) -- which is good methodology (+) -- but they do not provide the raw runtime results of the LLM snippets vs `clang -O3` in the Appendix.", "was_revised": true, "final_justification": "I believe the authors have reasonably addressed concerns raised and have reasonably defined and positioned their work. With the revisions discussed during the rebuttal phase, I believe this is a strong paper and maintain my positive score.", "rebuttal": "Thanks for your valuable feedback. We've addressed your comments on our workflow and evaluation. We discussed and supplemented the experiment to address the\nconcern. Please refer to the specific responses below and our rebuttal for more information.\n\n---\n\n**Weakness 1.** Global Workflow Design\n\nOur choice of a two-stage workflow involving \"initial generation\" followed by \"iterative optimization\" was driven by two primary considerations:\n\n- **Inspired by Compiler Design:** This approach draws an analogy to the design of traditional compilers, which separate code generation (lowering) from subsequent optimization passes. By decomposing the complex task of compilation, we can better guide the Large Language Model (LLM) to focus on distinct objectives.\n- **Reducing Task Complexity:** This separation is designed to reduce the cognitive load on the LLM of achieving high-quality compilation in a single step.\n    - In the **Initial Generation** phase, the model's primary goal is to ensure **functional correctness** by faithfully translating the IR into executable assembly code. This serves as the fundamental requirement.\n    - Building upon this foundation of correct code, the **Iterative Optimization** phase then focuses on enhancing **performance**. The model can make targeted improvements based on performance feedback.\n\nThis separation of concerns makes the entire neural compilation task more tractable and effective for the LLM. It also allows for more precise analysis and improvement of the model's specific capabilities (i.e., whether it struggles with generation or optimization).\n\n---\n\n**Weakness 1.** LLM Iterative Process\n\nIn our current design, we do not explicitly constrain the scope of modifications the LLM can make (e.g., by limiting the number of instructions changed). However, our analysis of the optimization traces reveals that the LLM tends to perform incremental improvements and local adjustments on top of the existing correct code, rather than conducting radical rewrites.\n\nTo quantify this observation, we calculated the average percentage of code lines modified by each model. The results are as follows:\n\n| Model | Average Change Ratio of Code Lines |\n| --- | --- |\n| GPT-4o | 23.09% |\n| O3-Mini | 26.39% |\n| O1 | 17.99% |\n| DeepSeek-V3 | 24.52% |\n| DeepSeek-R1 | 21.79% |\n\nThis data indicates that the optimization process is **relatively stable**, even without explicit constraints. The average modification rate, ranging from approximately 17% to 27%, confirms that the process is an evolution of the existing code rather than a complete reconstruction.\n\n---\n\n**Weakness 2 & Question 2.** Functional Correctness Evaluation\n\nWe sincerely thank the reviewer for pointing out this missing detail regarding our evaluation methodology. \n\nFor the portion of the dataset derived from ExeBench, we directly used the correctness test cases provided by ExeBench itself; we only modified the program format to make it suitable for our task. For the portion derived from TSVC, based on the original test cases in the dataset, we added loop iteration counting, and import random arrays to get more rigorous verification mechanism to ensure that the correctness reports are as accurate as possible, as is described below:\n\nMost of the test cases in the TSVC dataset involve loop-based computations on arrays. In the original validation setup, many examples initialize arrays with fixed values (e.g., all elements set to 1). This can lead to situations where, for certain operations such as averaging, the input and output arrays remain identical, making it impossible to detect specific types of model errors. To address this, we modified the initialization phase to assign array values using pseudorandom numbers generated with a fixed random seed. This ensures diverse initial values for testing computational correctness while maintaining reproducibility.\n\nOn another front, TSVC improves timing accuracy by running the same function many times in a loop, but it does not record the actual number of iterations. As a result, for functions where the output is the same whether computed once or multiple times, the model might incorrectly generate a loop with insufficient iterations, and such errors would go undetected. To improve this, we utilize the external function calls used in TSVC as a hook to count iterations. We maintain an external counter that records how many times the loop is executed and use this information for validation.\n\nWe will incorporate these specific details into the methodology section of our revised manuscript to ensure our verification process is clearly described.\n\n---\n\n**Weakness 3.** Detailed Runtime Performance\n\nWe thank the reviewer for suggesting that we include the detailed performance results. We agree that this data improves the paper's transparency and will add a corresponding table to the Appendix in the revised manuscript.\n\nThe table below presents the speedup of our method's generated code relative to `clang -O3` for the 16 functionally correct programs in Table 3.\n\nThe data shows that our method outperformed `clang -O3` in 14 of the 16 cases. The average speedup across these 16 programs is **1.28x**, with a maximum observed speedup of **3.25x**.\n\n| Program Name | Speedup (LLM over O3) |\n| --- | --- |\n| **s331** | **3.2485** |\n| **s1232** | **2.0621** |\n| **s332** | **1.9954** |\n| **s112** | **1.1048** |\n| **s152** | **1.0852** |\n| **s2251** | **1.0543** |\n| s451 | 1.0288 |\n| s319 | 1.0278 |\n| vdotr | 1.0277 |\n| s3112 | 1.0275 |\n| s4121 | 1.0274 |\n| s231 | 1.0253 |\n| s113 | 1.0247 |\n| s422 | 1.0224 |\n| s452 | 0.8621 |\n| s341 | 0.7924 |\n\nThis detailed data will be included in the Appendix of our revised manuscript to provide a more complete performance overview.\n\n---\n\n**Question 1.** Cost Function\n\nWe thank the reviewer for this question seeking further clarification. \n\nTo be precise, the **sole metric** used for the performance cost function, `c(·)`, in our work was the **program's execution time**. Our methodology for measuring this was to run each program 11 times and record the median **execution time of the middle 5 runs**. We adopted this approach to mitigate system noise and ensure that our performance results were stable and reliable.\n\nWe did **not** incorporate other factors, such as instruction count, into our primary performance metric for this study. While these are all valid and interesting metrics, we chose execution runtime because it is the most direct and practical measure of the final performance experienced by a user, which was the primary focus of our evaluation.\n\nWe will revise the manuscript to make this definition more explicit. Thank you for pointing out this need for clarification.\n\n---\n\n**To Paper Formatting Concerns**\n\nThank you for the concerns. We will make our best effort to improve the formatting in the final version to enhance the overall clarity and visual presentation of the paper.\n\n---\n\n**To Limitations**\n\nWe appreciate the thoughtful summary and highlighting an important limitation that we had not explicitly discussed.\n\nAs noted, our study focuses on LLVM IR, for it is the most general Intermediate Representations, used by various programming languages, so the techniques are likely to generalize to other IRs, particularly those based on Static Single Assignment (SSA) form. We agree that this is a limitation worth stating clearly. The design choices and assumptions in our method may not hold equally across all IR designs, especially those with significantly different semantics or control-flow structures.\n\nWe will revise the paper to explicitly acknowledge this limitation and clarify the scope of our current evaluation. We also appreciate the reviewer’s insight that the risk of non-generalization may be minimal in the SSA context, which helps contextualize the limitation."}, {"review_id": "X9H6El5BZW", "reviewer": "Reviewer_Yr1A", "summary": "This paper tackles the challenge of using large language models (LLMs) to automate compiler development, a process known as Neural Compilation. They introduce NeuComBack a dedicated IR-to-assembly benchmark. Using that dataset they propose a self-evolving prompt optimization technique that lets the model iteratively refine its own prompts through self-debugging traces. In experiments on x86_64 and aarch64, this method raises functional correctness from 44% to 64% and from 36% to 58%, respectively, and yields assembly code that often outperforms clang-O3, demonstrating its promise for low-level neural compilation.", "questions": "1) Why aren't you using a bigger dataset from the community? AFAIK this should be \"compile\" and just lower that to assembly code? \n2) How much time do you spend in optimizing a code using your approach how much time does it take to compile using O3?\n3) Why didn't you use software that is being used to understand performance of systems on your L2 dataset, and you focused on the vectorization dataset?", "limitations": "There is a limitation discussion in the appendix.", "rating": 2, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 4, "significance": 2, "originality": 2, "strengths_and_weaknesses": "The paper is well written and easy to follow.\n\nWeaknesses:\n\nThe L1 dataset they propose is fairly limited. There exist way larger LLVM IR corpuses in literature that can be used to generate machine assembly by just invoking the LLVM pipelines (opt, llc). For example \"Grossman, Aiden, et al. \"Compile: A large ir dataset from production sources.\"\".\n\nThe L2 dataset is interesting yet it is based on a pretty old publication (2011). The dataset is focused on vectorization, whereas compilers need to provide generic behavior. So using this as a datasets creates Neural Compilation that is effective for vectorization, an important optimization, but disregards other complex optimization opportunities. Nevertheless collecting performance data for the L2 dataset is a significant contribution.\n\nI find the comparison with O3 misleading and some of the discussion around the purpose of compilation. Overall compilers need to be fast at compiling,  increase performance (execution time, binary size etc), be correct. A large part of the evaluation focuses on the last 2 parts and completely disregard the notion of \"fast compilation\". I think an interesting question would be provided a fixed compile time of \"X\" seconds which technology produces the fasters and correct binary. This is not addressed by this work and in my opinion it should be addressed by the \"Neural Compilation\" community overall.", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for your valuable feedback. We've addressed your comments on our dataset and compilation. We discussed and supplemented the experiment to address the concern. Please refer to the specific responses below and our rebuttal for more information.\n\n---\n\n**Weakness 1 & Question 1.** Why not use existing dataset\n\nWe thank the reviewer for this comment regarding our L1 dataset and for highlighting important large-scale IR datasets like \"Compile\". We agree that our benchmark's scale is a topic worth discussing, and we appreciate the opportunity to clarify our design philosophy.\n\nThe primary goal of our benchmark is to provide a practical and effective tool for the **evaluation** of LLM-based compilation capabilities, rather than for large-scale pre-training or fine-tuning. This focus on evaluation has two main implications for the dataset's design:\n\n1. **Practicality and Cost-Effectiveness:** For evaluation tasks, especially those involving powerful but costly models accessed via APIs (e.g., advanced reasoning models), a moderate, high-quality, and carefully curated benchmark is often more practical than a massive one. This approach keeps the financial and time costs of each evaluation cycle within a manageable range, which we believe encourages broader adoption and more rapid, iterative research in the community.\n2. **Requirement for Verifiable Test Cases:** Another constraint is the requirement for verifiable test cases (i.e., defined inputs and corresponding expected outputs) to automatically assess the functional correctness of the generated assembly. While datasets like \"Compile\" offer a vast corpus of IR, **they do not provide the accompanying test cases needed for correctness verification**. Similarly, a significant portion of the original `ExeBench` dataset was unsuitable for automated testing. This necessity for complete, verifiable programming tasks, not just code snippets, greatly limited the pool of readily available data.\n\nAt the same time, we agree with the reviewer that the current scale of our benchmark is a starting point, and your point is very constructive. In our future work, we plan to continuously improve the quality and expand the scale of `NeuComBack`. We will explore methods to incorporate more diverse programs and potentially semi-automatically generate test harnesses for larger code corpora. This will not only enhance its utility for evaluation but also pave the way for creating valuable resources for LLM training in the compilation domain.\n\nWe will add a note in the revised manuscript to clarify this evaluation-centric design philosophy. Thank you for the valuable feedback.\n\n---\n\n**Weakness 2 & Question 3.** Dataset Diversity\n\nWe thank the reviewer for their valuable and constructive feedback on the L2 dataset, and for acknowledging the contribution of our performance data collection.\n\nThe TSVC dataset we use mostly consists of multi-level loop-based array floating-point computations, which resemble the structure of complex hot functions in real systems, while remaining relatively simple. This dataset is used by many recent works, such as **LLM-Vectorizer**[1] and [2].  Moreover, it provides optimization opportunities in vectorization, revealing the potential for neural compilation. Therefore, despite appearing limited in diversity, our L2 dataset is a suitable starting point for neural compilation optimization research.\n\nAt the same time, **we fully agree with your critical point**: a benchmark focused primarily on vectorization provides a limited view of a compiler's full capabilities. A comprehensive evaluation must extend to more general program structures and optimization types. We take this concern seriously and have, in fact, already begun work to address this. We plan to release an enhanced version of the dataset in the near future.\n\nThis concern also raises a question about the generalizability of our findings: are the compilation strategies learned on the vectorization-heavy L2 dataset specific to that task, or do they have broader applicability in improving correctness and performance across diverse program structures?\n\nTo address this, we have curated a supplementary test set from our ongoing **L2-extension efforts.** This new set contains 24 programs (a scale close to original L2 test set) and was specifically chosen to cover a diverse range of categories beyond vectorization: control flow, floating-point operations, memory access patterns, and pointer arithmetic, derived from [`LLVM Test Suite`](https://github.com/llvm/llvm-test-suite/tree/main/MultiSource). We then conducted a new experiment to test the generalization of our method. We took the exact same prompt that was learned exclusively on our original L2 training data and applied it to this new, diverse test set without any further prompt learning.\n\nThe results, using the DeepSeek-R1 model, still demonstrate a strong positive transfer effect, over the baseline vanilla prompt:\n\n| Category | Metric | Improvement |\n| --- | --- | --- |\n| Functional Correctness | Success Rate | 32.0% → 41.7% |\n| Runtime Performance | Outperformed -O3 Cases | 16% → 25% |\n| Debugging Efficiency | Avg. Self-Debug Rounds Required | 0.75 → 0.2 |\n- Functional Correctness (ACC): On this new diverse set, the prompt learned from L2 data improved the success rate from a baseline of 32.0% to 41.7%.\n- Runtime Performance (ACC + Perf):  After two rounds of iterative optimization, the learned prompt guided the model to produce code that outperformed the -O3 baseline in 25% of cases, up from 16% without the prompt, demonstrating its effectiveness in optimizing diverse code.\n- Debugging Efficiency: For the successfully compiled programs, the average number of self-debug rounds required was significantly reduced from 0.75 to 0.2 (with a maximum of 2 rounds allowed).\n\nNote that if we directly use this dataset to self-improve the prompt, the improvement may be more effective. \n\nWe hope the above explanation, supported by new experimental data, can alleviate your concerns regarding the L2 dataset's focus. We are grateful for this insightful question, as it has motivated us to more rigorously validate our findings and ultimately strengthen this work.\n\n[1]  LLM-Vectorizer: LLM-Based Verified Loop Vectorizer. CGO 2025.\n\n[2] All you need is superword-level parallelism: systematic control-flow vectorization with SLP. PLDI 2022.\n\n---\n\n**Weakness 3 & Question 2.** Compilation Time\n\nWe thank the reviewer for raising this crucial point about compilation time. This is a fundamental aspect of any compilation technology, and your questions rightly push us to clarify the positioning and purpose of our Neural Compilation approach.\n\nFirst, to directly answer Q2: a standard `clang -O3` compilation for programs in our benchmark is very fast, typically completing in **less than a second**. In contrast, current advanced LLM requires significantly more time in a single neural compilation process: approximately 0.5 **minute** for non-reasoning models and 5 **minutes** for reasoning models.\n\nWhile it's true that for most compilation tasks, compilation time is more important than small gains in runtime performance, there are scenarios where optimizing heavily-used core or hot functions, despite only marginal improvements, can be worthwhile. A typical example is CUDA kernels in LLMs, where even minor performance gains justify high optimization costs. Many compute libraries invest significant manual effort in writing these kernels, and super-optimization techniques like STOKE[1] or LENS[2] also demonstrate the willingness to spend substantial time searching for marginal improvements. These tools often require **hours** to find optimal instruction sequences for small code kernels. Similarly, searching for optimal compiler **phase ordering** using techniques like genetic algorithms (as seen in early research) can involve **hours or even days** of searching to yield performance gains, such as [3] and [4]. \n\nViewed in this context, our method is an automated approach that invests a moderate amount of time—far less than exhaustive search, but more than a standard compile—to discover novel, high-performance assembly code that general-purpose heuristics of `-O3` might miss. This investment is highly valuable in domains where runtime performance is paramount, such as in scientific computing, database query engines, or deep learning kernels. In these areas, a one-time optimization cost is easily amortized by the cumulative savings from repeated, faster executions.\n\nRegarding your suggestion of a fixed-time-budget comparison: this is an excellent and insightful framing. Given the current state of the technology, the outcome is quite clear:\n\n- For a **short budget** (e.g., < 10 seconds), `clang -O3` is the undisputed winner.\n- For a **longer budget** (e.g., > 10 minutes), Neural Compilation offers the *potential* to produce a superior binary in terms of runtime performance.\n\nThis frames the choice not as a direct competition on all axes, but as selecting the right tool for the job: rapid iteration vs. achieving peak performance.\n\nIn conclusion, we fully agree that our work does not address \"fast compilation.\" We will revise our manuscript to more clearly position Neural Compilation's goal: not to be fast *at* compiling, but to surpass the optimization limit of current compilers (i.e.,  `clang -O3`). Thank you for pushing us to clarify this important distinction.\n\n[1] Stochastic superoptimization. ASPLOS 2013.\n\n[2] Scaling up Superoptimization. ASPLOS 2016.\n\n[3] Exhaustive optimization phase order space exploration. CGO 2006.\n\n[4] Exponentially Expanding the Phase-Ordering Search Space via Dormant Information. CC 2024.\n\n---\n\n**To Paper Formatting Concerns**\n\nThank you for pointing this out. We will revise the formatting in the final version to ensure that table layouts are clearer and more visually distinct from the main text, improving overall readability."}, {"review_id": "3vTC7URE6s", "reviewer": "Reviewer_qGZW", "summary": "The paper studies the task of neural compilation. Unlike past works in machine-learning for compilation, the compiler is not supported by a machine learning system that, for instance, predicts a good pass ordering for the compiler, but the compilation itself is carried out by a pre-trained language model. The specific task tackled is the stage of compiling LLVM IR to machine language.\n\nTwo benchmarks are sourced, one targeting compilation accuracy of IO-intensive real-world applications, the other for measuring both compilation accuracy and compiler optimization on compute-heavy functions from a compiler vectorization benchmark.\n\nThe paper establishes that:\n- Reasoning models significantly outperform non-reasoning models on both benchmarks.\n- A simple prompt evolution technique massively improves performance without any parameter updates.", "questions": "See above.", "limitations": "Limitations have not been addressed in the paper. (There is a section in the paper checklist on \"what could have been discussed in the paper but wasn't\".) It would improve the paper to state the obvious limitations clearly, e.g.:\n- due to the small size of the dataset, it is difficult to use it for training,\n- no studies on small models and SFT/RL,\n- scope of the automatic prompt learning method is exactly for cases like this (reasoning-intensive, low-resource task).", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 4, "significance": 2, "originality": 2, "strengths_and_weaknesses": "**Strengths:**\n\n1. The paper is clearly written and a pleasure to read. Congratulations!\n2. The task of neural compilation is almost novel and considered at the right time when language models begin to be a viable option.\n3. The proposed prompt learning technique is surprisingly effective, and calls for more intensive study of such techniques in reasoning-intensive, low-resource domains and potential integration into existing SFT and RL pipelines even.\n\n**Weaknesses:**\n1. The prompt learning technique lacks details. Please provide the exact update rule for the learned prompt (When/how often do you ask which LLM with which prompt to do what? Is there at any time exactly one prompt, or is it a simple genetic algorithm?) [The checklist says \"Yes, reproducible\", which isn't the case without such details in the appendix.]\n2. The evaluation method lacks details. How is functional correctness evaluated? L206 mentions comparison of outputs, but on which inputs? Are the test cases model generated? Do they come with the originally sourced datasets? For TSVC, what is the \"enhanced\" correctness verification methodology (L208)? [Again, without, the paper is not reproducible as claimed in the checklist.]\n3. The paper could benefit greatly from some additional analyses that shed more light onto what currently works and doesn't work with LLMs in this domain, for example:\n- An inference scaling analysis that measures pass@k for different k for different model sizes, and similarly for the maximum number of self-debugging turns. Could there be a small cheap non-reasoning model that achieves similar scores as large reasoning models when comparing in a cost-controlled (compute-controlled) setup?\n- Experiments on fine-tuning smaller open-weights models on sequences from large reasoning models.\n- Generally, a comparison of SFT and RL to the automatic prompt learning technique.\n- A comparison to other prompt learning techniques or genetic algorithms.\n- Experiments on prompt transfer from a large reasoning model that figures out frequent mistakes and how to avoid them to smaller models / non-reasoning models that may be able to apply these rules.\n\n[I do not request all these experiments, but stress that the paper could have benefitted from more results and analyses than the model comparison and the effectiveness of prompt learning.]", "was_revised": true, "final_justification": "With a more detailed method section and additional experiments from the discussion period, the paper will improve further, so increasing my score by 1.", "rebuttal": "Thank you for the detailed review and your constructive suggestions. To address your main concerns, we have provided precise details on our prompt learning and evaluation methods, conducted new experiments on cost-controlled comparison and prompt transfer as you suggested, and will add a dedicated Limitations section to the revised paper. We hope these additions strengthen the paper and address your questions. Please see our detailed responses below.\n\n---\n\n# Weakness1: Details for Prompt Learning\n\nWe thank the reviewer for highlighting the need for more precise details regarding our prompt learning process. Below we clarify when and how the prompt is updated, how updates are generated, why we use a single evolving prompt (not a genetic algorithm), and how we ensure stability in the update process.\n\n- Update rule and frequency: The prompt is updated after each mini-batch of compilation tasks during training. Specifically, after processing a batch, we collect self-debugging traces from those programs that were initially incorrect but corrected via self-debugging.\n- How the update is generated: We use the *same LLM* under evaluation to serve as a \"prompt optimizer.\" The LLM is prompted (via a meta-prompt) to analyze these successful correction traces, extract error patterns and effective fixes, and propose edits to the existing prompt to reflect these new insights.\n- Single prompt, not genetic: At any time, there is a single evolving prompt; we do not maintain a population nor use a genetic algorithm.\n- Stability and confirmation: After the LLM suggests revisions, a comparison and confirmation step follows: we prompt the LLM to review the change list and confirm only the necessary and beneficial edits, adding stability to the evolution process.\n\nWe will add these concrete implementation details to the revised Methods section for clarity. To further support reproducibility, we are preparing to publicly release both our benchmark and the full prompt learning framework code.\n\n---\n\n# Weakness2: Details of Evaluation and Data Source\n\nWe sincerely thank the reviewer for highlighting these important details regarding our evaluation methodology. We agree that precise descriptions are critical for reproducibility and appreciate the opportunity to clarify. \n\n## 1. Inputs and Test Cases\n\n- All test cases are **sourced directly from the original ExeBench and TSVC benchmarks**; we do not use any model-generated test cases.\n- For each program, the assembly code produced by the LLM is assembled and executed using the standard input sets provided by the source benchmarks.\n- A program is considered functionally correct only if its output **exactly matches** the expected ground-truth output from the benchmark.\n\n## 2. Enhanced TSVC Correctness Verification\n\nThe TSVC benchmark mainly consists of array-based computation kernels evaluated through repeated loop operations. The original TSVC verification had two major limitations:\n\n- **Input Initialization:**\n    - Arrays were typically initialized with fixed values (e.g., all ones). This could cause certain computations (such as averaging) to yield the same result before and after the operation, potentially masking errors (e.g., the model fails to write back results but still passes correctness checks).\n    - **Our Enhancement:** We now initialize arrays with pseudo-random values generated using a fixed random seed. This provides a diverse set of initial conditions as test standards, while ensuring that results are fully reproducible.\n- **Loop Iteration Verification:**\n    - For timing accuracy, TSVC often runs functions for many iterations but does not always verify that the correct number of loop iterations are performed. In cases where single and multiple iterations yield the same result, an incorrect model implementation (with too few iterations) could go undetected.\n    - **Our Enhancement:** We use the original TSVC dummy external function as a hook and maintain an **external loop counter**. This allows us to check whether the generated code executes the intended number of iterations by comparing the observed count to the expected count.\n\nAll of these improvements and the corresponding scripts will be included in our open-source evaluation framework for full transparency and reproducibility.\n\n---\n\n# Weakness3: Cost-Controlled Model Comparison and Prompt Transfer\n\nYou have provided several excellent suggestions for additional analyses that can shed more light on the capabilities and limitations of LLMs in this domain. We agree that these analyses significantly strengthen the paper. Following your suggestions, we have conducted two new experiments focusing on cost-controlled model comparison and prompt transfer, which correspond to your first and fifth points. \n\n## 1. Cost-Controlled Model Comparison\n\nTo address your question, we performed a comparison between DeepSeekR1 and DeepSeekV3.\n\n- Cost Analysis: We first estimated the relative inference cost and found that a single experimental run with DeepSeekR1 is approximately 14 times expensive than with DeepSeekV3.\n- Experiment Setup: Based on this finding, we conducted a cost-controlled evaluation. We allocated DeepSeekV3 a 14-fold computational budget, allowing it to perform more sampling, to match the cost of a single run of DeepSeekR1. All other experimental settings remain the same as those used in Section 5.2 on NeuComBack-L2.\n- Results: In our original experiments (Table 3), DeepSeekR1 solved 11 out of 25 problems (achieving functional correctness). However, under this new cost-controlled setting, DeepSeekV3 successfully solved 17 out of 25 problems.\n    \n    | Setting | Problems Solved (Functional Correctness) |\n    | --- | --- |\n    | DeepSeekR1 | 11/25 |\n    | DeepSeekV3 (14x sampling) | 17/25 (+6↑) |\n\nThis result directly validates the reviewer's suggestion. It demonstrates that under a fixed computational budget, a **non-reasoning model** can leverage its efficiency to afford more sampling attempts, ultimately outperforming a **reasoning-focused counterpart of the exact same parameter size**. Despite DeepSeekR1 and DeepSeekV3 sharing the same parameter count, this finding strongly suggests that the potential of a \"small cheap non-reasoning model\" to match \"large reasoning models\" in a cost-controlled setting is very achievable.\n\n## 2. Prompt Transfer Across Models\n\nInspired by your suggestion, we investigated whether the insights learned by a reasoning model could benefit a non-reasoning one.\n\n- Experiment Setup: We took the optimized prompt that was automatically learned by the large DeepSeekR1 model on the L2 training dataset, and applied this prompt to the DeepSeekV3 model without any other changes to the configuration as in Section 5.1.\n- Results: DeepSeekV3 with a basic prompt solved 4 out of 25 problems on the L2 test set. Remarkably, by simply providing it with the prompt learned by DeepSeekR1, DeepSeekV3's performance doubled, solving 8 out of 25 problems.\n    \n    | Setting | Problems Solved (Functional Correctness) |\n    | --- | --- |\n    | DeepSeekV3 (basic prompt) | 4/25 |\n    | DeepSeekV3 (prompt learned by DeepSeekR1) | 8/25 (+4↑) |\n\nThis finding provides strong evidence for the generalization performance of our proposed prompt learning method. The successful knowledge transfer demonstrates that the learned prompts are effective not only across different data distributions, but are also robustly transferable between different models. This capability significantly enhances the practical utility of our approach, as an effective prompt can be optimized once and then deployed across various models.\n\n---\n\n# To Limitations\n\nThanks for highlighting this important point. We fully agree that explicitly stating the key limitations of our work would provide a more balanced and honest perspective.\n\nIn our revision, we will add a dedicated “Limitations” section that addresses the following points:\n\n- **Dataset Size:** Our benchmark is relatively small compared to large-scale code datasets, which makes it less suitable for fully supervised training of large models. Its primary role is for evaluation and targeted method development rather than model pretraining.\n- **Model and Method Scope:** Our experiments focus on large reasoning-capable models; we have not systematically studied small models or compared SFT/RL approaches, which would certainly provide further insights.\n- **Applicability:** The proposed automatic prompt learning method is designed for challenging, reasoning-intensive, low-resource tasks like IR-to-assembly, and its generalizability to other scenarios remains to be validated."}, {"review_id": "VHvuyzmUdE", "reviewer": "Reviewer_NLqo", "summary": "The paper introduces a new benchmark dataset called NeuComBack for IR to Assembly neural compilation and evaluates it with state-of-the-art LLMs to establish baselines. It also proposes a self-evolving prompt optimization that lets LLMs improve by learning from self-debugging traces.\n\nThe method is validated in experiments with state-of-the-art models, where it is able to improve on the best performing baseline, DeepSeek-R1.", "questions": "- What is novel in the prompt learning?\n- Is prompt learning also transferable on NeuComBack-L2, with respect to performance improvements?\n- In Table 3, why was the accuracy not improved after two rounds?\n- How was the correctness verification methodology of the TSVC dataset changed?\n- How was the number of self-debug rounds determined? Why is it different in the different evaluations?\n- Isn't the median of the middle 5 datapoints the same as just the median with an odd number of measurements?", "limitations": "The paper doesn't discuss limitations. I think the chief limitations are the size of the dataset and the simple execution paths of the programs in it.", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "## Strengths\n\nThe benchmarks use different state-of-the-art models, and the best performing model, DeepSeek-R1, is selected as the baseline. The presented method considerably improves upon it in both the correctness and performance of the compiled programs. Most of the correct programs (86.5%) surpassed clang-O3 performance.\n\nThe paper has ablation studies and interesting case studies of concrete optimizations where neural compilation surpasses the optimization of clang.\n\n## Weaknesses\n\nThe Method section is very short and doesn't provide sufficient detail. It gives an idea about the prompt optimization, and the examples in the Appendix help fill some of the gaps, but one would be hard pressed to reproduce the method or evaluate its novelty based on this description. The whole workflow is mainly exposited in a Figure without detailed explanations.\n\nThe introduced dataset has two levels, but both levels have simple execution paths (the difference is in the loop intricacy because of data conditions). So as I understand there are no examples in the dataset that have more complex execution paths. The dataset seems to be specialized and not for general programming.\n\nThe introduced dataset is very small compared to ExeBench. It has hundreds of examples, ExeBench has millions. As a consequence, the test set is also very small, just 25 examples for NeuComBack-L2.\n\nI don't understand why the Conclusion states that \"This paper presents a novel compiler paradigm called Neural Compilation\", because, as established in the Introduction and Related Work, Neural Compilation is an already existing and active field.\n\nThe word \"performance\" could be sometimes misunderstood, because performance evaluation could also refer to evaluating models. For example, on line 216-218: \"Level 1 ... was not primarily designed for performance evaluation, making its programs unsuitable\".\n\nTypos:\n- Figure 1 should not be positioned just after the section heading\n- Typo in Figure 1: finial output\n- line 269 is confusing, as O3 can be both an optimization level (meant here), but also an LLM.", "was_revised": true, "final_justification": "I believe that the revisions will significantly strengthen the paper and address my concerns.", "rebuttal": "Thank you for your feedback. We have clarified our methodology and reproducibility, including the workflow, key hyperparameters, and open-source plans. We have given clearer descriptions of our dataset design, scale, and coverage. We corrected terminology and formatting for better clarity. We added new experiments on prompt transfer and cost-controlled evaluation. We also openly discussed limitations and plans to expand the dataset. These focused revisions directly address your main concerns and make our work clearer and stronger.\n\n---\n\n## W1: Methodological Clarity and Reproducibility \n\nThanks for emphasizing the need for methodological detail and reproducibility. Our workflow is as follows:\n\n- For each mini-batch, an advanced LLM (e.g., DeepSeek-R1) generates assembly from IR using the current prompt, with up to N self-debug rounds.\n- Only successful self-correction traces are collected.\n- After each batch, a meta-prompt instructs the same LLM to extract error patterns and suggest prompt edits.\n- We always maintain a single evolving prompt—no genetic algorithm or population is used.\n- This runs for three epochs, and the prompt with the best validation accuracy is used for final testing.\n- All implementation details and meta-prompts are documented in the manuscript and appendix.\n\nWe will provide comprehensive details and open-source our implementation to ensure transparency and reproducibility\n\n---\n\n## W2: Dataset Coverage and Generality \n\nThank you for the opportunity to clarify the design intent of the NeuComBack benchmark. NeuComBack is structured with two complementary levels to jointly evaluate: (1) general-purpose functional correctness (L1, from ExeBench, covering a broad variety of real-world programs), and (2) advanced optimization performance (L2, from TSVC, focused on vectorization and loop optimization).\n\nL1 assesses breadth—an LLM’s ability to handle standard program logic across diverse constructs—while L2 targets depth, testing optimization strength on industry-standard kernels. We acknowledge that highly complex control flow (e.g., deep recursion, concurrency) is currently underrepresented, reflecting broader gaps in available datasets.\n\nDespite this, NeuComBack provides a strong foundational benchmark. As a next step, we plan to further expand the dataset to include more complex general-purpose programs, improving its coverage and value to the community.\n\n---\n\n## W3: Dataset Scale and Test Coverage\n\nThank you for raising this important point about dataset scale. NeuComBack was designed for practical and rigorous evaluation—not pre-training—so it is moderate in size but high in quality and fully verifiable. We only include tasks from ExeBench with well-defined inputs/outputs and reliable test harnesses, ensuring correctness and performance can be automatically measured.\n\nWe agree that scaling up is important. We are actively expanding NeuComBack with more sources, automated test harnesses, and greater diversity, which will further improve coverage and community value in future releases.\n\nWe will clarify these points and roadmap in the revised manuscript. Thank you for your helpful feedback.\n\n---\n\n## W4: On the Use of “Novel Paradigm” in Conclusion\n\nThank you for identifying this issue in our Conclusion. The reviewer is correct: Neural Compilation is an established field, and our work contributes to it rather than introducing it as a novel paradigm. The sentence \"This paper presents a novel compiler paradigm called Neural Compilation\" was an error and will be removed.\n\nTo accurately reflect our contributions, we will revise the Conclusion to:\n\n*“This paper advances the field of Neural Compilation by introducing a dedicated IR-to-Assembly benchmark (NeuComBack) and a novel self-evolving prompt optimization method, enabling more systematic and rigorous evaluation of LLM-based compilation techniques.”*\n\nOur work builds on prior studies (see Section 2), providing a reproducible, extensible platform and new methods to support the Neural Compilation research community.\n\nWe appreciate the reviewer’s careful reading and for helping us clarify and correctly position our contributions.\n\n---\n\n## W5: Clarification of “Performance” Terminology\n\nThank you for highlighting the ambiguity in our use of \"performance.\" We agree it is crucial to distinguish between a model’s task-solving accuracy and the efficiency of the generated code.\n\nTo improve clarity, we will revise the manuscript to use:\n\n- “runtime performance,” “execution efficiency,” or “optimization quality” for code efficiency;\n- “functional correctness,” “success rate,” or “task accuracy” for model capability.\n\nThese distinctions will be applied consistently, including in Section 3.2 and the Discussion. For example, lines 216–218 will be revised to:\n\n> “Level 1 … was not primarily designed to assess runtime performance or optimization quality, making its programs unsuitable for such evaluation.”\n> \n\nWe appreciate the reviewer’s feedback and believe these changes will improve the precision and readability of our paper.\n\n---\n\n## Formatting, Typographical, and Clarity Issues (Typos questions)\n\nThank you for carefully noting these formatting and typographical issues.\n\n- **Figure 1** will be moved to avoid following the section heading.\n- **Typo** “finial output” will be fixed to “final output.”\n- **\"O3\"** will be clarified throughout to indicate the compiler’s optimization level.\n\nWe appreciate your attention to detail and will correct these points to improve clarity and professionalism.\n\n---\n\n## Q1: Novelty in Prompt Learning Approach\n\nThank you for the question—here are the novel aspects of our prompt learning method:\n\n1. **Selective Learning:** We use only self-debug traces from programs that were initially wrong but corrected, so the LLM learns effective correction strategies. Ablations confirm this is more data-efficient and achieves top results.\n2. **Controlled Evolution:** Every prompt update is meta-verified by the LLM, ensuring changes are necessary and stable.\n3. **Automatic Prompt Learning for Compilation:** We are among the first to show that LLMs can use their own self-debug history to improve correctness and optimization.\n4. **Different from Prior Work:** Our method is fully automated and tailored for neural compilation, unlike earlier prompt optimization approaches.\n\nWe will emphasize these contributions further in the revision.\n\n---\n\n## Q2: Transferability of Prompt Learning on NeuComBack-L2\n\nThank you for asking about prompt transferability on NeuComBack-L2.\n\nwe have curated a supplementary test set from our ongoing L2-extension efforts. This new set contains 24 programs (a scale close to original L2 test set) and was specifically chosen to cover a diverse range of categories beyond vectorization: control flow, floating-point operations, memory access patterns, and pointer arithmetic, derived from LLVM Test Suite.\n\n**Results (DeepSeek-R1):**\n\n- **Accuracy** improved from 32.0% to 41.7%.\n- **Performance**: Outperforming -O3 cases increased from 16% to 25% (after two optimization rounds).\n- **Debug rounds** dropped from 0.75 to 0.2.\n\nThis shows our prompt learning generalizes well to new L2 tasks, improving both correctness and optimization. We will include this supplementary result in the revision.\n\n---\n\n## Q3: Interpretation of Stable Accuracy (ACC) Across Two-Phase Optimization in Table 3\n\nThank you for raising this point about Table 3’s ACC metric.\n\nOur two-phase workflow is designed such that:\n\n- **Phase 1 (Generation + Self-Debug):** Produces functionally correct code. The “ACC” metric reflects the final success rate here.\n- **Phase 2 (Iterative Optimization):** Operates only on correct outputs from Phase 1, aiming to further improve runtime performance while maintaining correctness. ACC is not expected to change in this stage; instead, “ACC+Perf” indicates how many correct outputs also outperform the baseline.\n\nFor context, with the **learned prompt**, 16 correct programs yielded 14 faster-than-O3 outputs (87.5%), while the baseline yielded 7/11 (63.6%).\n\nIn summary, the stable ACC is expected, and the learned prompt boosts the chance of further optimizing correct programs. We will clarify these metrics and workflow in the revision.\n\n---\n\n## Q4: Changes to the Correctness Verification Methodology for TSVC\n\nThank you for asking about our TSVC evaluation improvements.\n\n- **Input Initialization:** We now use random values (with a fixed seed) for array initialization, instead of fixed values. This helps reveal subtle errors and ensures reproducibility.\n- **Loop Iteration Verification:** We added an external loop counter (via the dummy function) to explicitly check that the required number of loop iterations is executed.\n\nThese changes are included in our open-source evaluation framework for transparency and reproducibility.\n\n---\n\n## Q5: Determination of Self-Debug Rounds and Rationale for Differences\n\nThanks for your question about self-debug rounds.\n\n- For **x86_64**, we fixed the max rounds at two for fairness and reproducibility.\n- For **aarch64**, we allowed up to four rounds, as it is a more challenging target for LLMs.\n\nThis balances comparability (x86_64) and task difficulty (aarch64). We will clarify these settings in the revision.\n\n---\n\n## Q6: Clarification of “Middle 5” Median Performance Measurement\n\nThank you for highlighting this subtle point about our performance measurement.\n\nOur process is based on execution order, not value:\n\n1. **Chronological Filtering:** Each program is run 11 times; we discard the first 3 (warm-up) and last 3 (cool-down), keeping only the 4th–8th runs (“middle 5” by order).\n2. **Stable Median:** We take the median of these 5 central runs to capture a robust, stable performance measurement.\n\nThis approach helps minimize transient system effects and is standard in benchmarking. We will clarify this process in the revised manuscript. Thank you for prompting this clarification."}]

Author rebuttal:

Meta-review: This work studies the extent to which a LLM can "compile" LLMV IR to machine code. It both evaluates a range of models on this task, showing that reasoning models are substantially better at it, and introduces a prompt evolution workflow that substantially increases compilation rates.

The reviewers generally agreed that this is a timely and novel contribution, the work is well-written, and the prompt evolution approach is remarkably effective. They noted some concerns that stemmed from unclear aspects in the writing. The authors addressed these thoroughly in the rebuttal.

Overall, the significant novelty and interesting results motivated the acceptance of this paper. Please ensure the final version includes all the suggested and discussed revisions.
Decision: Accept (poster)