REASONING COMPILER: LLM-Guided Optimizations for Efficient Model Serving
Authors: Annabelle Sujun Tang
Christopher Priebe
Rohan Mahapatra
Lianhui Qin
Hadi Esmaeilzadeh
Venue: NeurIPS 2025 poster ()
Keywords: Compiler
Optimization
LLM
LLMs for Code Reasoning and Optimization
Primary area: infrastructure

Abstract: While model serving has unlocked unprecedented capabilities, the high cost of serving large-scale models continues to be a significant barrier to widespread accessibility and rapid innovation. Compiler optimizations have long driven substantial performance improvements, but existing compilers struggle with neural workloads due to the exponentially large and highly interdependent space of possible transformations. Although existing stochastic search techniques can be effective, they are often sample-inefficient and fail to leverage the structural context underlying compilation decisions. We set out to investigate the research question of whether reasoning with large language models (LLMs), without any retraining, can leverage the context-aware decision space of compiler optimizations to significantly improve sample efficiency. To that end, we introduce a novel compilation framework (dubbed REASONING COMPILER) that formulates optimization as a sequential, context-aware decision process guided by a large language model and structured Monte Carlo tree search (MCTS). The LLM acts as a proposal mechanism, suggesting hardware-informed transformations that reflect the current program state and accumulated performance feedback. MCTS incorporates the LLM-generated proposals to balance exploration and exploitation, facilitating
a structured, context-sensitive traversal of the expansive compiler optimization space. By achieving substantial speedups with markedly fewer samples than leading neural compilers, our approach demonstrates the potential of LLM-guided reasoning to transform the landscape of compiler optimization.

REASONING COMPILER: LLM-Guided Optimizations for Efficient Model Serving
Annabelle Sujun Tang University of Californa San Diego sujun@ucsd.co edu
Christopher Priebe' University of California San Diego
Rohan Mahapatra" University of California
San Diego rohan@ucsao edu
cpriebe@ucsd.edd
Lianhui Qin
Hadi Esmaeilzadeh
University of California
University of California
San Diego 1 Llanhui@ucsd. edu
San Diego hadi@ucsd.eed
Abstract
While model serving has unlocked unprecedented capabilities, the high cost of serving large-scale models continues to significant barrier to widespread ac- cessibility and rapid innovation. Compiler optimizations have long driven sub- stantial performance improvements, but existing compilers struggle with neural workloads due to the exponentially large and highly interdependent space of pos- sible transformations. Although existing stochastic search techniques can ef- fective, they often cemplee-nefffcient and fail to leverage the structural context underlying compilation decisions. We set out to investigate the research question of whether reasoning with large language models (LLMs), without any retrain- ing, can leverage the context-aware decision space of compiler optimizations to significantly improve sample efficiency. To that end, introduce novel compi- lation framework (dubbed REASONING COMPILER) that formulates optimization sequential, context-aware decision process guided by large language model and structured Monte Carlo tree search (MCTS). The LLM acts as a proposal mechanism, suggesting hardware-informed transformations that reflect the cur- rent program state and accumulated performance feedback. MCTS incorporates the -.Meeeeeeteed proposals balance exploration and exploitation, facilitating a structured, context-sensitive traversal of the expansive compiler optimization space. By achieving substantial speedups with markedly fewer samples than lead- ing neural compilers, our approach demonstrates the potential of LLM-guided reasoning to transform the landscape compiler optimization."
Introduction
The rise of model serving for LLMs, diffusion models, and other neural models has enabled new class of intelligent systems, driving transformative applications in healthcare, education, and sci- entific discovery. These models incur significant computational demands during inference, which proportionally translate into substantial monetary costs. Driving down the cost of model serving is critical, not merely to broaden access and democratize inference, but to catalyze faster cycles of novation in model design and deployment. Achieving this goal demands reducing inference runtime on computational infrastructure, resources that are not only expensive but also increasingly limited in availability. Compiler optimizations are critical enabler, not only for cost-efficient inferencing across diverse applications but also for empowering rapid research iteration.
CCode is available at http://gshhb..cmmmt-aaaalaaAAff CAA. COMPILER
"quaal contribution
39th Conference Neural Information Processing Systems (NeurIPS 2025).

---

Existing compilers struggle with neural models due to the exponentially large space of valid pro- gram transformations (e.g.. tiling, fusion, and layout changes). Each decision, such as selecting a tiling factor or a parallelization strategy, introduces dependencies and constraints that influence the feasibility and performance benefits of subsequent transformations. Rule-based optimizations also often rely on hand-tuned heuristics that can overfit to specific workload or hardware target. The seminal work in superoptimization [[3 aimed to tackle these shortcomings through enumerative or symbolic search, but the search space proved combinatorial and rugged. STOKE 4 showed that high-quality programs often lie in regions separated by low-probability paths, and therefore adopted Markov chain Monte Carlo (MCMC)-based) randomized search. Neural compilers followed suit, using evolutionary search or simulated annealing to navigate similarly irregular landscapes [5) While these methods have shown promise in discovering performant configurations, they are fun- damentally sample-ineffiiient They overlook synergistic transformations that emerge only when decisions are made with contextual awareness. These techniques also often explore redundant sub- spaces or invalid configurations.
In contrast, we set out to investigate the research question of whether reasoning with large lan- guage models (LLMs), without any retraining, can leverage the context-aware decision space of compiler optimizations to significantly improve sample efficiency. To that end, we introduce a novel compilation framework that couples LLM reasoning with Monte Carlo tree search (MCTS) [9] to guide compiler optimization. Hence, in our approach, compiler optimization is cast sequential decision-making process, in which each transformation, such as tiling. fusion, or vectorization, is selected with awareness of the current program state, while also assimilating downstream informa- tion and propagating its implications upstream to guide future decisions. Our approach avoids the prohibitive cost of fine-tuning L.MMs as compilation policies, does it require additional training or task-specific adaptation. In this formulation, the LLM evaluates partial transformation sequences and proposes contextually appropriate next steps, drawing upon hardware-informed cost models and the historical trajectory of optimization decisions to inform its proposal. The LLM serves as a context-aware proposal engine: given the current schedule and its observed performance, it gener- ates candidate transformations that are likely to be effective in the context of the traversed trajec- tory. These LLM-guided reasoning choices are integrated into an MCTS framework that provides a structured mechanism for balancing exploration and exploitation by evaluating LLM- LLM-suggested transformations, expanding promising branches, and leveraging rollout feedback to adaptively steer the search toward high-performing regions of the exponentially large optimization space.
This integration of LLM-bbsed chain-of-thought (CoT) guidance with tree search combines con- textual reasoning and adaptability with principled, structured decision-making enabling the com- piler navigate the complexity of the search space with significantly improved sample efficiency. We evaluate the REASONING COMPILER and compare its improvements and sample efficiency with TVM, which employs evolutionary search. Results show that the REASONING COMPILER consis- tently achieves significantly higher speedups than what TVM achieves using significantly fewer samples. On five representative benchmarks (Llama-3-88 Attention Layer, DeepSeek-Ri R1 MoE Layer, FLUX Attention Layer, FLUX Convolution Layer, and Llama-4-Scout MLP Layer) and across five hard- ware platforms (Amazon Graviton2, AMD EPYC 7R13, Apple M2 Pro, Intel Core 19, and Intel Xeon E3), the REASONING COMPILER achieves 5.0x average speedup using 5.8x fewer samples, resulting in an average of 10.8x improvement over TVM in sample efficiency. For the end-to-end Llama-3-8P 3-BB benchmark across five hardware platforms, the REASONING COMPILER uses 3.9x fewer samples to achieve 4.0x speedup, yielding a 5.6x sample efficiency improvement. These results underscore the promise of LLM-guided reasoning in neural compilation for efficient and scalable model serving.
2 Problem Formalization
Sap. argmax Sloo o o... o (p)) s'cor, S'SST
We consider the problem of optimizing an input program Po E P representing a layer from a neu- ral network for some objective function f P R Z 0. This objective function represents an evaluation of the program on the target platform for some figure of merit (e.g.. latency, power, utilization). Any program p E P can be transformed through the application of some transforma- tion/optimization (used interchangeably from here out) E O, where each optimization is
2

---

function o P + P that performs targeted transformation to the program, thus introducing a new variant of the program that is semantically equivalent to the original program but may perform better or worse on target hardware platform. In this way, successive application of transforma- tions to program can yield significant performance differences from the original. Therefore, given some maximum transformation sequence length T, the goal is to find sequence of transformations Sopt. = (01, 02..0.,... On) such that n < T and f (Popt) = maxs.co |8') f(oo) o ... o of (po)) where Pope. = (o, o On-1 o ................................ 01) (po) and O is the Kleene star of O. These constraints collec- tively deñne the optimization objective given in Equation (T).
To facilitate an efficient search over the space of valid program transformation sequences, we cast the optimization problem finite-horizon Markov decision process (MDP) defined by the tu- ple M AS P,R). This formulation provides a structured approach for sequential decision- making in the transformation space, allowing the search process to account for how individual transformations compound over time to affect final program performance. Compared to unstruc- tured methods such as exhaustive purely stochastic search, which often require large number of expensive program evaluations, casting the problem as an MDP enables more deliberate explo- ration, offering the potential for improved sample efficiency. Each state s, 8 corresponds to program Pt E P obtained by applying a sequence of transformations to the original program po- 0n (pn). An action a, E A corresponds to selecting transformation
i.., St Pt (or o E O to apply at step t, transitioning the current program to a new variant. Since the application of a transformation is deterministic, the transition function P( (;,aa) az) is 1 if s141 = ae (st) and 0 otherwise. The reward function is defined as the objective value caused by the optimization sequence, i.e., R(s,, de) = s (pt)) where always better.
(+1,-1) is chosen so that larger rewards are
By formulating the problem Carlo tree search (MCTS) [9] to explore program transformation sequences. Under standard as- sumptions, such finite branching, bounded rewards, and tree policy (e.g., UCT) that guarantees persistent exploration, MCTS is consistent on finite-horizon problems: as the number of simula- tions tends to infinity, it converges (with probability 1) the optimal root action/sequence Sopt that maximizes the objective. With any finite simulation budget, it returns a high-quality but approx- imate solution. Consequently, our framework (see $3) yields sequence Sype. that approximately maximizes the objective in practice while enjoying asymptotic optimality in theory.
MDP, we enable the use of planning algorithms such Monte
3 REASONING COMPILER: Integrating LLM-Guided Contextual Reasoning with Monte Carlo Tree Search
We present the REASONING COMPILER, a novel compilation framework that unifies the structured exploration capabilities of Monte Carlo tree search (MCTS) [9] with the contextual, history-awand reasoning of large language models (LLMs). While MCTS provides principled approach to explor- ing sequences of program transformations, compiler optimization introduces unique challenge: the successive application of transformations can exhibit complex, non-local interactions that are diffi- cult to capture through purely stochastic or myopic policies. To address this, we employ an LLM to model program transformation context, tracking which transformations have been applied, how they impact performance, and what directions remain promising. This contextualization is essential to enabling effective and sample-efficient search in compiler optimization.
Optimization interactions are complex, making efficient search challenging. Unlike tasks where actions are relatively independent, program transformations compose in subtle and complex ways. For example, the profitability of applying loop tiling may depend on the prior application of loop fusion or unrolling. Additionally, transformations can introduce new, unforeseen opportunities/con- straints for future transformations. These dependencies make the space of valid and useful trans- formation sequences both combinatorial and deeply contextual. While black-box methods such as evolutionary search and some implementations of reinforcement learning have achieved notable suc- cess in compiler autotuning (6 (8) [0] [1], they often do not explicitly model the nuanced structural and temporal dependencies between transformations. This can limit their ability to generalize across contexts, as optimization efficacy depends transformation histories. Even when guided by local
2The Kleene star operator, denoted with asterisk (*), represents the set of all finite-length sequences, includ- ing the empty sequenee, formed from elements of given set.
3

---

MCTS Tree Update
MCTS Tree
Chain-of- Thought Prompt
LLM
Hardware cost model
PLa
Ph-1
Yow and semiin pini voke Aor
N4-2
The sote
01
MCCT) liw. Turrea de the crar
Select leaf node W
groncaamen refly the sectrtos
YoU
IN
Blus
UCT value
based
enturnn dene poop ento ser aboutting regin
procee anrif ouf voctter lire phagee
Star
pretccd merter Lurl 1y
Response
Apply 9u+1 to
Figure 1: Overview of the optimization workflow. The algorithm explores the tree to select candidate node. At this node, the LLM is prompted with contextual information generate sequence of transformations, which are then applied to produce optimized code variants.
reward signals, they may struggle to capture the interplay between past decisions and future op- portunities, limiting their effectiveness in deeply contextual optimization landscapes. Our insight that efficient search in this space benefits from an agent that reasons over transformation history, structural code changes, and observed performance dynamics to choose the next step.
3.1 LLM-Guided Contextual Reasoning for Program Transformation Proposal
Contextual reasoning via LLMs. To address these challenges, the REASONING COMPILER lever- ages a large language model (LLM) as a contextual reasoning engine. The LLM is tasked with synthesizing program transformations that are not only syntactically valid but also informed by the program's 's full history and structure. By prompting the LLM with a rich, structured representation of the current optimization state, we enable reason over the cumulative effects of prior transfor- mations, analyze performance trends, and identify differential improvements over prior programs.
Figure iilustraats the optimization workflow. From the root program, the REASONING COMPILER traverses the tree by computing the UCT score [12]. selecting promising leaf node (i.e., program) P; for expansion by balancing exploitation of high-reward paths and exploration of under-sampled branches based on visit statistics and empirical mean rewards (see (3.2).
Prompt construction. At each expansion step in the search, the LLM receives a prompt that in- cludes the source code and predicted performance for the current program pt. its parent P;-1, and its grandparent p--2- It also includes the ordered sequences of transformations that were applied to reach each of these program variants, denoted S;, S;-1, and S;-2. Finally, the full set of available transformation operations O is included. Given this context, the LLM is explicitly instructed to: (1) analyze the differences between program variants and their associated performance scores, identify- ing which transformations contributed to observed performance changes; (2) reason about potential interactions between previously applied and candidate future transformations, including both syn- ergistic and antagonistic effects; (3) synthesize new sequence of transformations that is justified in the context of the current program structure and transformation history; and (4) provide a ratio- nale for the proposed sequence, referencing specific code features and transformation interactions. This structured prompt is designed to elicit chain-of-thouge
(CoT) reasoning [13], encouraging the LLM to perform deep, multi-step analysis and move beyond surface-level edits, instead generating proposals that are both semantically meaningful and tailored to the evolving optimization trajectory.
Transformation proposal and validation. The LLM proposes candidate transformation 0++11 O in the form of a string. Given the generative nature of the LLM, the output may include an invalid or unrecognized transformation even though it is guided by a predefined set of valid transformations. To ensure correctness, the output string is first parsed and filtered to retain only transformation that matches known valid names and transformation parameters. If no valid transformation is found, the REASONING COMPILER samples a random transformation from the valid set. The successfully validated and applied transformation yields a new program variant Pi+1, with its transformation history updated as S = S, (0+41), where @ denotes sequence concatenation. This new program variant is scored using a hardware cost model and used to update the MCTS tree (see (3.2).
It is important to emphasize that the LLM is not the centerpiece of our contribution, but a necessary enabler of effective search in this domain. Compiler optimization poses a uniquely challenging setting due to the non-local, compositional nature of transformatio interactions. Traditional black- box search or heuristic- neuristic-guided guided methods struggle to navigate such spaces efficiently. The REASONING

---

Wied
Mi Mpaa)+
Maa
Fi
Hardware cost model
Pb
Ph
(a) MCTS Expansion via LLM Suggested Transformations
(b) MCTS Rollout
(c) MCTS Backpropagation
Figure 2: Structured tree search where nodes are (a) selected and expanded with the LLM suggested transformations, (b) scored by a learned hardware cost model, and (c) updated with performance estimates to guide future search.
COMPILER uses structured search (via MCTS) with learned contextual reasoning (via LLM + CoT) to overcome these challenges. The result is a
optimization algorithm capable of discovering performant transformation sequences in high-dimensional, high-interaction spaces.
3.2 Structured Optimization via Monte Carlo Tree Search
MCTS as a sample-efficient planner. As described in 2 we cast program optimization finite- horizon decision process over the space of transformation sequences. Framing the problem as an MDP allows the REASONING COMPILER to consider long-tew -erg-termm optimization effects and leverage planning algorithms such as Monte Carlo tree search (MCTS) to explore this space efficiently.
a tree J (V, E¹) where V P and E O such that each node P is program from the state space 8 and each edge o E O corresponds to a transformation from the action space A. This tree structure naturally supports the reuse of common transformation prefixes and allows the planner to backpropagate value estimates from downstream program variants to upstream decisions. Such reuse is critical in compiler optimization, where transformation sequences exhibit both compounding effects and long-range interactions.
MCTS operates
Selection via UCT. During the selection phase, MCTS traverses T from the root, recursively select- ing child programs p; to maximize the UCT (Upper Confidence bounds applied to Trees) criterion:
((:)) N(p)))
In N(i-1)
UCT(p;)
+ +VV N(p:)
where W is the cumulative reward of pt. (p:) is the visit count of node (i.e., program) pt, and c governs the apploration-exploitatoio tradeoff.
LLM-guided expansion. As shown in Figure aa), a), once a promising leaf node p; is selected, an LLM is queried to propose transformation conditioned on p; and its ancestors (see $3.1). The model generates a candidate transformation 0 E O, which is applied to p; to produce a new the updated program and extended transformation path. To ensure T remains acyclic, if pi+1 already exists in the tree, it is not added. By leveraging the LLM's contextual reasoning, the system proposes globally informed transformations that extend beyond myopic heuristics.
program 1++1 0.411 (p:). This results in a new node Pi+1 added to T corresponding
Rollout for local reward estimation. As shown in Figure b). once new node P++1 is added to the tree, the REASONING COMPILER performs a lightweight MCTS rollout to estimate the long- term impact of the transformation sequence that produced it. This is done by sampling random- ized sequence of legal transformations o1,..., Og and applying them to obtain a terminal program Paim = (o, 0...0.. 000 1 Directly measuring hardware-leve performance requires compiling and running on real hardware, which is too expensive for the inner loop of a planning algorithm. Follow- ing standard practice in compiler autotuning, the REASONING COMPILER
learned, hardware- informed surrogate ff for f that is cheap to evaluate and accelerates search while preserving final quality [6. 10 16) We convert this prediction into a rollout reward (pp++) = 8 = (psmm) where 8 E (+1,-11 -1) is chosen so that larger values indicate better performance (e.g.. & = -1 for latency). This noisy but informative proxy lets MCTS trade off immediate and downstream effects
without real-hardware runs.
5

---

Backpropagation. As shown in 2 c), the estimated reward W (p+++) is then backpropagated all ancestors along the path to the root according to the update step W (pA)= +- W (PA) + W (p++1) where PA is some ancestor program. The visit counts are also updated according to the update step (pA)) N(pA) + 1. These updates refine the empirical estimates that guide future selections.
4
Evaluation
We implement the REASONING COMPILER as an extension to MetaSchedule [8] The framework introduces three modular components: (1) prompt generator that serializes the current scheduling state, including the IRModule, transformation trace (i.e., the applied schedule history), and hard- ware cost model outputs, into structured prompts that capture the textual difference from the base IRModule and reflect the current schedule's performance; (2) an LLM interface that queries an ex- ternal API (e.g., OpenAT) and parses the LLM's output into candidate transformation sequences: and (3) tree manager that performs MCTS with selection based on UCT score, expansion using LLM suggested transformations, simulation with a hardware-informed cost model, and backpropagation for tree statistics updates.
4.1 Methodology
We evaluate the REASONING COMPILER five representative computational kernels drawn from production-scale models: (1) a self-attention layer from Llama-3-88 [17.. (2) a mixture-of-expprrss
(MoE) layer from DeepSeek-RY [8] (3) self-attention layer from FLUX (stable diffusion) (19), (4) convolution layer from FLUX 19 and (5) an MLP layer from Llama-4-Scout [20]. In addition, perform an end-to-end evaluation of Llama-3-8B. Compiler optimization is framed as sequen- tial decision process and guided by MCTS 9 using the Upper Confidence bounds applied to Trees (UCT) criterion 12 with exploration parameter e = /2 and branching factor B = 2 following prior work 21 22]. During search, the LLM (OpenAI GPT-4o mini (23)) queried using hicrarchical context- Speciitlpecifiically. the parent and grandparent schedules and their transformations-to enable in- formed proposal generation. We compare three optimization strategies: (1) TVM MetaSchedule (8), which uses Evolutionary Search; (2) MCTS without LLM guidance (MCTS); and (3) the REASON- ING COMPILER that uses prompt-base based proposal generation (LLM-Guided MCTS). All experiments are conducted using Apache TVM v0.20.0 [IO 24 Our experimental environment is dedicated Intel Core 19 workstation under fixed software and hardware stack to isolate scheduling effects. This environment covers all five kernels above and is the ablation environment. To show portability and scalability across consumer and datacenter processors, we evaluate each of the five kernels on five hardware platforms: Amazon Graviton2, AMD EPYC 7R13, Apple M2 Pro, Intel Core 19, and Intel Xeon E3. Each experiment is repeated 20 times, and we report the mean performance to en- sure statistical stability. Additionally, we leverage OpenAl and HuggingFace model serving APIs to access the respective models. The implementation is open-sourced.
(Evolufoonary Soarch) R1 MJE
Reasoning Compler Guided MCTS)
Liama-SS 3-I8 Allantion
FLUX
Convouuar
Liarn- Booul
Code
10-
Reative
100
1.000
100
1000
1000
100
1000
Number Samples
Number Simples
Number Samples
Number of Samples
Number Samples
Figure 3: Relative speedup over pre-optimized code function of evaluated transformation proposals. The REASONING COMPILER achieves superior sample efficiency, discovering high- quality code with fewer samples across all operators in low-budget regimes.
4.2 Experimental Results
We assess the sample efficiency the REASONING COMPILER by analyzing how code quality evolves with increasing search budget, quantified in terms of evaluated transformation proposals. F results across five representative workloads, encompassing both transformer-style attention layers and convolution- architectures. Across all benchmarks, our method achieves

---

competitive or superior code performance with significantly fewer samples than state-of-the- black-box autotuners such TVM with Evolutionary Search. These results directly support the central hypothesis of our work: leveraging LI context-aware reasoning enables more effi- cient and effective exploration of the compiler optimization space.
Rapid convergence in low-sample regimes. A consistent trend across all benchmarks is the rapid ascent of code quality the initial stages search. This early-stage performance is critical in prac- tice, as real-world compiler pipelines often operate under strict tuning time budgets. Figure shows Relative Speedup over Pre- Tre-Optinized Code on the y-axis, with the number of evaluated transforma- tion proposals on the x-axis. Speedup is defined as the ratio of the execution time of the unoptimized code that of the optimized code after tuning. Higher values indicate more efficient and optimized code. For instance, on the Llama-3-8B self-attention layer, the REASONING COMPILER achieves 7.08x speedup over the untuned baseline with just samples, whereas TVM with Evolutionary Search requires 72 samples, which is twice the budget to achieve comparable gains. On the Llama- 4-Scout MLP layer, the gap even more pronounced: the REASONING COMPILER achieves 12.7 speedup at 20 samples, while TVM falls short of this mark even after 3000 samples.
Quantitative sample efficiency. To formally quantify sample efficiency, we compare the number of samples required by each method to reach target speedups. On the FLUX self-attention layer, the REASONING COMPILER attains 2x speedup using only samples, while TVM with Evolutionary Search requires more than 600 samples, 16x reduction in tuning cost. On the FLUX convolution layer, the REASONING COMPILER consistently outperforms TVM across nearly all budget levels and reaches T'M's s final performance after evaluating just 400 samples.
Speedup relative to baselines. The REASONING COMPILER not only produces better code, but does so more aggressively and earlier in the search process. For example, on the DeepSeek- R1 MoE Layer, the REASONING COMPILER achieves 3.3x speedup over TVM with Evolutionary Search at 36 samples; on the Llama-4-Scour MLP layer, the REASONING COMPILER achieves 9.3x speedup over TVM 20 samples. This trend, which shows strong initial gains followed by conver-
Table 1: Sample efficiency comparison between the REASONING COMPILER and TVM with Evo- lutionary Search on layer-wise benchmarks across various hardware platforms. Improvement
TVM
REASONING COMPILER
Hardware Platform
Benchmark
Sample Reduction
Sample Efficiency
Samples Speedup # Samples Speedup
Gain
Llama-3-8B Layer
510
3,9x
60
5.1x
8.5x
11.1x
DeepSeek-R1 MoE Layer
980
2.7x
150
5.9x
6.5x
14.4x
Amazon
FLUX Attention Layer
320
1.6x
130
3.0x
2,5x
4.6x
Graviton2
FLUX Convolution Layer
160
1.8x
20
4.1x
8.0x
18.2x
Llama-4-Scour MI.P Layer
1,630
1.7x
500
4.0x
3.3x
7.7x
Llama-3-8B Layer
1,400
2.1x
200
12.1
7.0x
40.3x
DeepSeek-R1 MoE Layer
2,290
1.7x
330
2.3x
6.9x
9.4x
AMD
FLUX Attention Layer
2,460
1.5x
230
3.1x
10.7x
22.1x
EPYC 7R13
FLUX Convolution Layer
2,520
1.3x
470
4.8%
5.4x
19.6x
Llama-4-Scout MLP Layer
510
6.4x
100
10.2x
5.1x
8.1x
Llama-3-8B Attention Layer
1,010
3.3x
190
9.7x
5.3x
15.6x
DeepSeek-R1 MoE Layer
1,040
2,8x
230
4.8%
4.5%
7,8x
Apple
FLUX Attention Layer
270
2.1x
50
3.7x
5.4x
9.5x
M2 Pro
FLUX Convolution Layer
2,260
1.5x
510
5.5X
4.4X
16.2x
Llama-4-Scout MLP Layer
2,460
2.2x
440
3.4X
5.6x
8.6x
Llama-3-8B Attention Layer
920
10.5
11.0X
7.1X
7.4x
DeepSeek-R1 MoF Layer
1,632
9.1x
192
9.1x
8.5x
8.5x
Intel
FLUX Attention Layer
1,000
5.1x
150
S.4X
6.7%
7.0x
Core i9
FLUX Convolution Layer
400
2.3x
72
2.3x
5.6x
5,6X
-lama-4-Scour MI.P Layer
230
5.6x
20
12.7x
11.5x
26.1x
Llama-3-8B Attention Layer
2,760
3.9x
320
5.8%
8.6x
12.8x
DeepSeek-R1 MoE Layer
1,000
3.7x
4.4x
5.6x
6.6x
Intel
FLUX Attention Layer
1,340
1.4x
3.4x
3.0x
7.1x
Xeon E3
FLUX Convolution Layer
220
1.9%
40
2.2x
5.5x
6.4X
Llama-4 4-Scuut MLP Layer
1.200
2.0x
300
6.1x
4.0x
12.2x
Geomean
2.7x
5.0
5.8%
10.8

---

genee, demonstrates that the REASONING COMPILER quickly identifies high performing regions of the search space, while TVM's uninformed search requires substantial exploration reach similar quality.
Operator-specific trends. We observe that certain operator types, such matrix multiplication operations extracted from attention layers and MLP layers, exhibit sharper performance improve- ments. This is likely due recurring structural patterns such as loop fusion, tilling, and vector- ization, which pretrained LLMs can more readily recognize and exploit. Convolutional operators, by contrast, expose a broader and less regular transformation space. Nonetheless, the REASONING COMPILER consistently matches or exceeds baseline performance with fewer samples, underscoring its effectiveness across diverse operator characteristics.
Sample efficiency across hardware platforms. As shown Table the REASONING COMPILER demonstrates superior sample efficiency compared TVM with Evolutionary Search across five hardware platforms on five benchmarks. We define sample efficiency as the speedup achieved per Speedup sample #f Sumples.. On average, across all 25 platform-operator pairs, the REASONING CoM- PILER achieves a 5.0x speedup using 5.8 fewer samples, resulting in 10.8 improvement in sample efficiency. The performance gains are particularly significant for compute-intensive work- loads. For instance, for the Llama-3-8B self-attention layer on AMD EPYC 7R13, the REASONING COMPILER achieved 12.1 x speedup in just 200 samples, while TVM required 1,400 samples to reach a 2.1 x speedup. This represents a 7.0x sample reduction and 4 sample efficiency gain. On Intel Core 19, the REASONING COMPILER often matches or exceeds TVM's peak with fewer tri- als: on the Llama-4-Scout MLP layer, the REASONING COMPILER used 11.5x x fewer samples for 26. efficiency gain.
Table 2: Sample efficiency comparison between the REASONING COMPILER and TVM with Evo- lutionary Search end-to-end Llama-3-8B across various hardware platforms.
TVM
REASONING COMPILER
Improvement
Hardware Platform
Sample Efficiency Gain
Sample Reduction
Samples
Speedup
Samples
Speedup
Amazon Graviton2
4,560
3.7x
1,440
5.1x
3.2
x
AMD EPYC 7R13
410
2.0x
140
2.2x
2.9x
x.2x
Apple M2 Pro
4,820
2.2x
1,770
3.9x
2.7x
4.8
Intel Core 19
3,800
2.2x
720
4.9%
5.3x
11.8
Intel Xeon E3
4,640
5.0x
670
5.0x
6.9x
6.9x
Geomean
2.8x
4.0x
3.9x
5.6x
End-to-end sample efficiency. For end-to-end Llama-3-8B across the five hardware platforms in Table 2 the REASONING COMPILER's sample efficiency improvement over TVM ranges from 3.2x on AMD EPYC 11.8x on Intel Core 19. End-to-end speedups range from 2.2x AMD EPYC to 5.1x x on Amazon Graviton2. The REASONING COMPILER consistently achieves significantly higher speedups: using 3.9x fewer samples, it achieves 4.0x speedup and yields 5.6x geometric-mean sample efficiency improvement over TVM with Evolutionary Search.
These findings reinforce our core thesis: compiler optimization should be cast structured decision process, enriched by prior knowledge and contextual reasoning. Our integration of LLMS into Monte Carlo tree search results in strategically guided and sample-efficient search, particularly valuable in scenarios with constrained tuning budgets. By generating performant code with orders-of-mant mf-ggniuue fewer samples, framework offers both practical deployment advan- tages and a compelling alternative to conventional, sample-inefficient compilation pipelines.
Implications.
4.3
Ablation Studies
4.3.1
Impact of LLM Choice and Reasoning Strategy
To better understand the contributions of different components in our approach, conduct an ab- lation study focused on the effects of LLM selection and reasoning modality. Figure shows the relative speedup over unoptimized code as a function of the number of schedule samples evaluated by the REASONING COMPILER on Llama-3-85 self-attention layer using range of LLM models

---

UP14 Mis
Usmad Sirotred (195)
Unmad L. retruct (850
Pro-Optinzon Optiiizee Code
Opends Shmmel
DeepSeen Deri 1376
DeepSreb Dist
Rele Speedup Pre-Optinized Code
Grandparent
Relative Speeduu
20
1.0
Number of Samples
Number of Samples
(a)
(b)
Figure 4: Ablation studies for the Llama-3-8B self-attention layer. (a) Comparing different
LLMs as proposal engines shows stronger LLMs lead to faster convergence. (b) Increasing the prompt's historical trace depth improves sample efficiency.
for API calls. The x-axis indicates the cumulative number of schedules explored, while the y-axis shows the best speedup achieved so far. This setup enables us to directly compare how effectively various LLMs leverage contextual information to guide the search. The general trend of the re- sults supports our central claim: compiler optimization benefits from goal-directed, context-awan reasoning in terms of sample efficiency. Below, we discuss the specific behaviors that exemplify different reasoning strategies.
Large instruction-tuned Llama3.3 (70B) achieves exceptional sample efficiency. The instruction-tunee Llama3.3-70B model rapidly attains near-optimal performance, reaching a 9.69x speedup after only 36 samples, roughly 86% of the GPT-4o mini's maximum speedup but with less than 6% of its sampling budget. This corresponds to an approximately 15x improvement in sample efficiency. Instruction tuning also significantly improves the ability of LLMs to generate domain-specific, context-aware transformation proposals. The consistent performance advantage of instruction-tur tundd models over untuned counterparts of comparable size confirms that semantic task alignment, combined with sufficient model capacity, synergistically enhances the effectiveness of sequential context reasoning in guiding compiler optimizations.
(32B) excels in long- horizon optimization. The DeepSeek-RI Distill-QWenn32BB model, employing mixture-of-expert. (MoE) architecture, exhibits more grad- ual improvement, starting with a 7.07 x speedup at 18 samples and reaching 9.98x after 579 sam- ples. The sparse expert routing inherent MoE architectures likely facilitates exploration of com- plex transformation sequences over extended horizons, complementing context-aware reasoning by enabling specialized and conditional poccision-making
DeepSeek-RR--Distll--oe
Lower-parameter models also achieve high sample efficiency. Despite their reduced scale, smaller models still produce notable speedups relative to the untuned baseline. For example, LJama3.1-Insteee (8B) reaches a 5.87 speedup, and DeepSeek-R1 -easSeek-RR-Disiil----aaa (7B) achieves a 4.86x speedup at just 36 samples. When compared to the widely used Evolutionary Search strategy, which requires around 72 samples to achieve a 7.0x speedup and fails to reach comparable per- formance for tuning the DeepSeek-R1 MoE layer. Even after 3000 samples, these smaller models consistently outperform. The REASONING COMPILER with lower-parameter models achieves at least twice the sample efficiency of TVM with Evolutionary Search, making them well-suited for efficient compiler optimization in local or edge deployments.
Open-source models match proprietary models in performance. Our results demonstrate that open-source LLMs, when adequately scaled and instruction-tumme match or exceed the performance of proprietary baselines such GPT-4o mini. This underscores the broad applicability of our ap- proach and its independence from proprietary data or architectures, enabling widespread adoption of context-aware, -.L--uddded compiler optimization.
4.3.2 Impact of Historical Trace Depth on Optimization Efficiency
Figure 4bb presents the relative speedup over unoptimized code as a function of the number of schedule samples evaluated by the REASONING COMPILER the Llama-3-8B self-attention layer. Using deeper historical trace (see Figure I in the prompt (parent + grandparent + great- grandparent) leads to faster convergence compared the shallower trace (parent + grandparent). For example, at 36 samples, the deeper trace achieves speedup of approximately 7.13x, slightly surpassing the 7.08x of the shallower trace. However, by 72 samples, the deeper trace saturates at 1 11.36x speedup, while the shallower trace reaches only 8.38x, requiring many more samples

---

(around 579) to approach 11.3x performance. This demonstrates that including longer histori- cal context enables the LLM to better capture dependencies and synergies in transformation se- quences, resulting in more sample-efficient and goal-directed exploration, validating the advantage of context-aware reasoning.
Related Work
Superoptimization. While high-level goal of discovering highly efficient program variants shares motivation with the superoptinization literature, our formulation and tractability differ sub- stantially. Superoptimization aims to find the globally optimal instruction-level program, typically via enumerative [ 3. symbolic (2). or stochastic [4] search over low-level assembly variants; hybrid [25] and neural [26] approaches have also been explored. STOKE 4 showed that high- quality programs often reside in low-probability regions and made the leap to use randomized search (MCMC). Neural compilers followed suit and relied evolutionary search simulated annealing algorithms [518] In contrast, the REASONING COMPILER treats optimization as a planning prob- lem that leverages MCTS reason contextually about dependencies among transformations over structured intermediate representations.
ML-Based Autotuning. Autotuning frameworks optimize performance-criticcl parameters (c.g., loop tile sizes, phase orderings, memory layouts) using a variety of ML-based techniques, includ-
ing linear models 27
28], tree-based methods (29 30 Bayesian networks [31 32 evolutionary
algorithms 22
35]. The REASON- ING COMPILER shares the same goal of performance-driven parameter selection, but distinguishes
33 34 clustering [2 34), and reinforcement learning [
itself by combining LLM-based contextual reasoning with structured search (via MCTS) to explore transformation sequences in a history- and structure-aware manner.
Techniques for Neural Compilation. A large body of work targets the optimization of neural network inference pipelines, spanning graph-level transformations and scheduling (36143) and low- level code generation [5] 49). Many modern systems incorporate learned components (c.g., cost models) and search strategies navigate large configuration spaces; for example, TVM/An- sor 66 7 [0] and FlexTensor [50] use learned performance models and evolutionary strategies for tuning. While highly effective for tensor-program tuning, these approaches often emphasize local parameter optimization or rely on domain-specific heuristics. The REASONING COMPILER moves beyond these works by introducing LLM-based contextual reasoning over transformation history, structural changes, and performance trends, enabling history- and structure-aware exploration not addressed in prior neural compilation work.
LLMs for Code Reasoning and Optimization. LLMs have demonstrated capabilitics in code gen- eration (5))) fuzzing [57]. bug repair (58), and even high-level optimization [59]. Recent work
has explored the use of LLMs to generate phase orderings or perform disassembly [60] 61 The
REASONING COMPILER advances these approaches by embedding an LLM in structured decision loop, leveraging it for context-aware reasoning within a grounded search process.
Conclusion
Compiling neural workloads remains bottleneck for scalable model serving: traditional compilers struggle with combinatorial transformation spaces, and the state-of-the- neural compilers rely on stochastic search, lacking sample efficiency and contextual awareness. This paper introduced the REASONING COMPILER, novel framework that formulates compiler optimization as a sequential, context-aware decision process, pairing L.MM-generate proposals with MCTS and performance feedback to reason and navigate through the optimization space efficiently. By enabling LLM rea- soning in the compiler optimization process, we achieve a leap from randomized search informed and guided compilation. Our results show that the REASONING COMPILER consistently yields faster runtimes with markedly fewer evaluations without any retraining. These gains directly translate to reduced operational cost of LLM services, lower energy usage per query, improved system respon- siveness, more agile model deployment, faster model training, and accelerated innovation cycles, among other benefits. Looking ahead, the same LLM that guides compilation accelerate its inferencing, creating a virtuous, self- optimizii cycle in which sped-up LLMs enable more efficient transformations and progressively better models and services.
10

---

Acknowledgments
We thank the anonymous reviewers for their insightful feedback. This work was in part supported by the National Science Foundation (NSF) award CCF #2107598. The U.S. Government is autho- rized to reproduce and distribute reprints for governmental purposes not withstanding any copyright notation thereon. The views contained herein are those of the authors and should not be interpreted as necessarily representing the official policies or endorsements, either expressed or implied by the U.S. Government.
References
[1] Henry Massalin. Superoptimizer: a look at the smallest program. In ASPLOS, 1987.
[2] Rajeev Joshi, Greg Nelson, and Keith Randall. Denali: goal-directed superoptimizer. In
PLDI, 2002.
[3] Sorav Bansal and Alex Aiken. Automatic generation of peephole superoptimizers. In ASPLOS,
2006.
[4] Eric Schkufza, Rahul Sharma, and Alex Aiken. Stochastic superoptimization. In ASPLOS,
2013.
[5] Nicolas Vasilache, Oleksandr Zinenko, Theodoros Theodoridis, Priya Goyal, Zachary VeVito, William S. Moses, Sven Verdoolaege, Andrew Adams, and Albert Cohen. Tensor Comprehen- sions: Framework-agnostic high-performance machine learning abstractions. arXiv, 2018. [6] Tianqi Chen, Lianmin Zheng, Eddie Yan, Ziheng Jiang. Thierry Moreau, Luis Ceze, Carlos Guestrin, and Arvind Krishnamurthy. Learning to optimize tensor programs. In NeurIPS, 2018.
[7] Lianmin Zheng, Chengfan Jia, Minmin Sun, Zhao Wu, Cody Hao Yu, Ameer Haj-Ali, Yida Wang, Jun Yang, Danyang Zhuo Koushik Sen, Joseph E. Gonzalez, and Ion Stoica. Ansor: Generating high-performance tensor programs for deep learning. In OSDI, 2020.
[8] Junru Shao, Xiyou Zhou, Siyuan Feng, Bohan Hou, Ruihang Lai, Hongyi Jin, Wuwei Lin, Masahiro Masuda, Cody Hao Yu, and Tianqi Chen. Tensor program optimization with proba- bilistic programs. In NeurlPS, 2022.
[9] Cameron B Browne, Edward Powley, Daniel Whitehouse, Simon M Lucas, Peter I Cowl- ing. Philipp Rohlfshagen, Stephen Tavener, Diego Perez, Spyridon Samothrakis, and Simon Colton. A survey of Monte Carlo tree search methods. IEEE Transactions on Computational Intelligence and Al in games, 4(1):1-43, 2012.
[10] Tianqi Chen, Thierry Moreau, Ziheng Jiang, Lianmin Zheng, Eddie Yan, Meghan Cowan, Haichen Shen, Leyuan Wang, Yuwei Hu, Luis Ceze, Carlos Guestrin, and Arvind Krishna- murthy. TVM: An automated end-to-end optimizing compiler for deep learning. In OSDI, 2018.
[11] Byung Hoon Ahn, Prannoy Pilligundla, and Hadi Esmaeilzadeh. Chameleon: Adaptive code optimization for expedited deep neural network compilation. In ICLR, 2020.
[12]
Levente Kocsis and Csaba Szepesvári. Bandit based Monte-Cart planning. In ECML, 2006.
[13] Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Brian Ichter, Fei Xia, Ed H. Chi, Quoc V. Le, and Denny Zhou. chain-of-thoughl prompting clicits reasoning in large language models. In NeurlPS, 2022.
[14] Minjia Zhang, Menghao Li, Chi Wang, and Mingqin Li. DynaTTnne: Dynamic tensor program optimization in deep neural network compilation. In /CLR, 2021.
[15] Byung Hoon Ahn, Sean Kinzer, and Hadi Esmaeilzadeh. Glimpse: Mathematical embedding of hardware specification for neural compilation. In DAC, 2022.
[16] Perry Gibson and José Cano. Transfer-Tuning: Reusing auto-schedules for efficient tensor program code generation. In PACT, 2023.
[17] Aaron Grattafiori, Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Alex Vaughan, Amy Yang, Angela Fan, et al. The Llama 3 herd of models. arXiv. 2024.
[18] Daya Guo. Dejian Yang, Haowei Zhang, Junxiao Song, Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang, Xiao Bi, al. DeepSeek-R1: Incentivizing reasoning capability in
LLMs via reinforcement learning. arXiv, 2025.
[19]
Black Forest Labs. Flux, https//github..om/bbe http:blllcc---os---eeee
2024.
[20]
Meta. The Llama 4 herd: The beginning of new era of natively multimodal AI innovation. 2025.
11

---

[21]
Rémi Coulom. Efficient selectivity and backup operators in Monte-Carlo tree search. In CG,
2007.
[22] Peter Auer, Nicold Cesa- Biaa-Biacchi, and Paul Fischer. Finite-time analysis of the multiarmed bandit problem. Machine Learning, (7(2-3):235 256, 2002.
[23]]
OpenAl. GPT-4o mini API. htttps:/plattormmooooo wwwpttttooooooooo.oooommmmam wttpm//.w.wddoc/ooo////tt-oooooo
[24]
Apache TVM Community. Apache TVM v0.20.0. hutp://gghhbbbouo/pppppe///vveas
v0.20.0, 2025.
[25] Phitchaya Mangpo Phothilimthana, Aditya Thakur, Rastislav Bodik, and Dinakar Dhurjati.
Scaling up superoptimization. In ASPLOS, 2016.
[26] Rudy Bunel, Alban Desmaison, M. Pawan Kumar, Philip H. S. Torr, and Pushmeet Kohli.
Learning to superoptimize programs. In ICLR, 2017.
[27] Mark Stephenson and Saman Amarasinghe. Predicting unroll factors using supervised classi-
fication. In CGO, 2005.
[28] Amir H. Ashouri, Andrea Bignoli, Gianluca Palermo, Cristina Silvano, Sameer Kulkarni, and Mitigating the compiler phase-ordering problem using optimization sub-sequences and machine learning. ACM Trans. Archit. Code Optim., 14(3), 2017.
John Cavazos. MiCOMP:
[29] Douglas Simon, John Cavazos, Christian Wimmer, and Sameer Kulkarni. Automatic construc- tion of inlining heuristics using machine learning. In CGO, 2013.
[30] Ameer Haj-Ali, Hasan Gene, Qijing Huang. William Moses, John Wawrzynek, Krste Asanović, and Ion Stoica. ProTuner: Tuning programs with Monte Carlo tree search. arXiv,
2020.
[31] Amir Hossein Ashouri, Giovanni Mariani, Gianluca Palermo, Eunjung Park, John Cavazos, and Cristina Silvano. COBAYN: Compiler autotuning framework using Bayesian networks.
ACM Trans. Archit. Code Optim., 13(2), 2016.
[32] Erik Orm Hellsten, Artur Souza, Johannes Lenfers, Rubens Lacouture, Olivia Hsu, Adel Ej- jeh, Fredrik Kjolstad, Michel Steuwer, Kunle Olukotun, and Luigi Nardi. BaCO: A fast and portable Bayesian compiler optimization framework. In ASPLOS, 2023.
[33] Mircea Trofin, Yundi Qian, Eugene Brevdo, Zinan Lin, Krzysztof Choromanski, and David Li. MLGO: a machine learning guided compiler optimizations framework. arXiv, 2021.
[34]
Haolin Pan, Yuanyu Wei, Mingjie Xing, Yanjun Wu, and Chen Zhao. Towards efficient com- piler auto-tuning: Leveraging synergistic search spaces. In CGO, 2025.
[35] Ameer Haj-Ali, Qijing Jenny Huang, John Xiang, William Moses, Krste Asanovic, John Wawrzynek, and Ion Stoica. AutoPhase: Juggling HLS phase orderings in random forests with deep reinforcement learning. In MLSys, 2020.
[36] Moshe Looks, Marcello Herreshoff, DeLesley Hutchins, and Peter Norvig. Deep learning with dynamic computation graphs. arXiv, 2017.
[37] Zhihao Jia, Oded Padon, James Thomas, Todd Warszawski, Matei Zaharia, and Alex Aiken
TASO: Optimizing deep learning computation with automatic generation of graph substitu- tions. In SOSP, 2019.
[38] Yizhi Liu, Yao Wang. Ruofei Yu, Mu Li, Vin Sharma, and Yida Wang. Optimizing CNN model inference on CPUs. In ATC, 2019.
[39] Yanqi Zhou, Sudip Roy, Amirali Abdolrashidi, Daniel Wong, Peter Ma, Qiumin Xu, Hanxiao Liu, Mangpo Phitchaya Phothilimtha, Shen Wang, Anna Goldie, Azalia Mirhoseini, and James Laudon. Transferable graph optimizers for ML compilers. In NeurlPS, 2020.
[40] Yaoyao Ding, Ligeng Zhu, Zhihao Jia, Gennady Pekhimenko, and Song Han. IOS: Inter- operator scheduler for CNN acceleration. In MLSys, 2021.
[41] Zhen Zheng. Pengzhan Zhao, Guoping Long, Feiwen Zhu, Kai Zhu, Wenyi Zhao, Lansong Diao, Jun Yang, and Wei Lin. FusionStitching: Boosting memory intensive computations for deep learning workloads. arXiv, 2021.
[42] Yichen Yang. Phitchaya Phothilimthana, Yisu Wang. Max Willsey, Sudip Roy, and Jacques Pienaar. Equality saturation for tensor graph superoptimization. In MLSys, 2021.
[43] Jie Zhao, Xiong Gao, Ruijie Xia, Zhaochuang Zhang, Deshi Chen, Lei Chen, Renwei Zhang, Zhen Geng, Bin Cheng, and Xuefeng Jin. Apollo: Automatic partition-based operator fusion through layer by layer optimization. In MLSys, 2022.
[44] Riyadh Baghdadi, Jessica Ray, Malek Ben Romdhane, Emanuele Del Sozzo, Abdurrahman Akkas, Yunming Zhang, Patricia Suriana, Shoaib Kamil, and Saman Amarasinghe. Tiramisu: A polyhedral compiler for expressing fast and portable code. In CGO, 2019.
[45] Bastian Hagedorn, Archibald Samuel Elliott, Henrik Barthels, Rastislav Bodik, and Vinod Grover. Fireiron: A aitta-mvvemet-any scheduling language for GPUs. In PACT, 2020.
12

---

[46] Jian Weng, Animesh Jain, Jie Wang, Leyuan Wang. Yida Wang, and Tony Nowatzki. UNIT: Unifying tensorized instruction compilation. In CGO, 2021.
[47] Rui Li, Yufan Xu, Aravind Bantuumrran-jjjaa Atanas Rountev, and P. Sadayappan. Analytical characterization and design space exploration for optimization of CNNs. In ASPLOS, 2021.
[48]
Wookeun Jung, Thanh Tuan Dao, and Jaejin Lee. DeepCuts: A deep learning optimization framework for versatile GPU workloads. In PLDI, 2021.
[49] Yaoyao Ding, Cody Hao Yu, Bojian Zheng. Yizhi Liu, Yida Wang, and Gennady Pekhimenko.
Hidet: mask-mapping programming paradigm for deep learning tensor programs. In ASPLOS, 2023.
[50] Size Zheng, Yun Liang, Shuo Wang. Renze Chen, and Kaiwen Sheng. FlexTensor: An auto- matic schedule exploration and optimization framework for tensor computation on heteroge- neous system. In ASPLOS, 2020.
[51] Baptiste Roziere, Jonas Gehring, Fabian Gloeckle, Sten Sootla, Itai Gat, Xiaoqing Ellen Tan, Yossi Adi, Jingyu Liu, Romain Sauvestre, Tal Remez, et al. Code LJama: Open foundation models for code. arXiv, 2023.
[52] Anton Lozhkov, Raymond Li, Loubna Ben Allal, Federico Cassano, Joel Lamy-Poirier, Noua- mane Tazi, Ao Tang, Dmytro Pykhtar, Jiawei Liu, Yuxiang Wei, et al. StarCoder 2 and The Stack v2: The next generation. arXiv, 2024.
[53] Yuxiang Wei, Zhe Wang. Jiawei Liu, Yifeng Ding, and Lingming Zhang. Magicoder: Empow-
ering code generation with OSS- instrucc.. arXiv, 2023.
[54] Qinkai Zheng, Xiao Xia, Xu Zou, Yuxiao Dong, Shan Wang, Yufei Xue, Lei Shen, Zihan Wang, Andi Wang, Yang Li, et al. CodeGeeX: A pre-trained model for code generation with multilingual benchmarking on HumanEval-X. In KDD, 2023.
[55] Yue Wang, Hung Le, Akhilesh Deepak Gotmare, Nghi DQ Bui, Junnan Li, and Steven CH Hoi. CodeT55: Open code large language models for code understanding and generation. arXiv, 2023.
[56] Daya Guo, Qihao Zhu, Dejian Yang, Zhenda Xie, Kai Dong, Wentao Zhang, Guanting Chen, Xiao Bi, Yu Wu, YK Li, et al. DeepSeek-Coder: When the large language model meets programming the rise of code intelligence. arXiv, 2024.
[57] Chunqiu Steven Xia, Matteo Paltenghi, Jia Le Tian, Michael Pradel, and Lingming Zhang. Universal fuzzing via large language models. arXiv,, 2023.
[58] Chunqiu Steven Xia, Yuxiang Wei, and Lingming Zhang. Automated program repair in the of large pre-trained language models. In ICSE, 2023.
[59] Alexander Shypula, Aman Madaan, Yimeng Zeng, Uri Alon, Jacob R. Gardner, Yiming Yang. Milad Hashemi, Graham Neubig, Parthasarathy Ranganathan, Osbert Bastani, and Amir Yaz- danbakhsh. Learning performance-improoing code edits. In ICLR, 2024.
[60] Chris Cummins, Volker Seeker, Dejan Grubisic, Mostafa Elhoushi, Youwei Liang, Baptiste Roziere, Jonas Gehring, Fabian Gloeckle, Kim Hazelwood, Gabriel Synnaeve, and Hugh Leather. Large language models for compiler optimization. arXiv. 2023.
[61] Chris Cummins, Volker Seeker, Dejan Grubisic, Baptiste Roziere, Jonas Gehring, Gabriel Syn- naeve, and Hugh Leather. LLM Compiler: Foundation language models for compiler optimiza- tion. In CC, 2025.
13

---

A LLM Prompt Example
Below show example prompt used in LLM-Guided MCTS framework (refer Figure 1
Example Code to Optimized:
@t.m. script...... ir_module
class MyModule:
(T or.prin_func
def main
A: T. T.Buffer (1, 16, 7168), "float32"),
3: T.Buffer (7168, 2048), "float32"),
C: T.Buffer (11, 16, 2048), "float32"),
):
T. T.func_atten (( goslobal_symboll
"main", "tir.noalias": True))
for b, t, j, k in T.grid 16, 2048, 7168): with r.block ("moe"):
vb, vt, vj, vk T.axis. remap ("SSSR", [b, t, j, kl)
with T.init():
Clvb, vt, vi] T.float32 (0)
Clvb, vt, vil += Alvb, vt, vk] B(vk, vi]
Prompt
You are a code optimization assistant performing Monte Carlo Tree Search (MCIS) on given code to improve performance. Each code has corresponding history of transformations and predicted cost. You are given the code of the selected node and its immediate parent. Their main differences include:
Loop shapes:
Current: for b_i, t_1, 1-1 in (1, 2, 8), and then j_2_init in I.grid(1), and later
1_2 in I.grid(1).
Index example: Parent:
1-1 64 i_2 64
for b_i, t_1, i_1 in (l 2, 2), then 1_2iinit in 1.grid(())
and later 2 in (4)
Index example:
i 256 i.2 64
Tile decisions:
Current: sample_pefect_tillef decision-[4, [4, 8, 1, 64])
Parent:
wilee.eetteeee.......eeee.e_eeee decision=[4, 2, 4, 641)
Performance estimates:
Current: 0.773
Parent: 0.313
Avail lable transformations:
reiiiize, Parallel, Computellocation, Unroll
Task
Analyze the IR, trace, and predicted scores.
Then propose sequence transformations (you may repeat any) to potentially improve performance.
Output your reasoning and your suggested transformations.
For example, your answer should Reasoning: This code still has large loop extents, so I'd tile it twice differently, then unroll.
the following format:
Cransformations to apply: TileSize, TileSize, Unrell.
14

---

LLM Response
Reasoning: The current schedule tiles the j-axis 2048=4x8x1x64,
whereas the parent current node (0.773 va. 0.313) suggests that expanding outer parallel granularity (i_1-8) helped, but collapsing j_2 to 1 removed small register-level tile, limiting unrolling and hurting reuse within each i_3=64 chunk.
4x2x4x64. The higher predicted score of the
I would (1) reintroduce a non-unit j_2 while shrinking i-1 so that i_li__ =8; for instance, [4,4, 64] keeps enough outer tasks but creates j_2xj_3=128-wide microtile that unrolls cleanly. (2) Tile the 7168-step k-reduction to align with i_3=64, e.g., 7168 7x16x64; the 16-step inner slice can unrolled improve register reuse across the i_3 tile. (3) Move the compute of
cut loop overhead and
block "moe" to the 1_1 level so each 1.1 tile C locally, shortening live ranges and reducing write-back pressure. (4) Parallelize to utilize cores without touching the reduction. (5) Unroll the
outer 1-1 for t_1 if 1-1 becemes too small)
small i_2 loop and the k-inner microtile. These changes retain the beneficial outer structure of the current node while restoring
compact inner tile for ILP and cache locality, which should further improve throughput.
framatormations to apply: TileSize, TileSize, Computelocation, Parallel, Unroll, Unroll.
15

---

B Sample Efficiency and Speedup Results
Table presents the relative speedup of three melhods- Search, MCTS, and the REA- SONING CoMPILER--evlluated across the different benchmarks. Speedup is measured as the ratio of execution time for the unoptimized code that of the optimized code after applying given number of transformation proposals. The table captures performance as a function of the number of samples explored. Higher values indicate more effective optimization. For instance, the REASON- ING COMPILER consistently achieves higher speedups with fewer samples, demonstrating superior sample efficiency and faster convergence compared to MCTS and Evolutionary Search. This table corresponds to Figure 3 in the paper.
Table 3: Speedup unoptimized code across varying numbers of samples for different compiler optimization methods.
Number of Samples
18
72
144
192
600
900
1632
5952
Evalutioary Search
4.67
5.70
7.74
7.98
9.40
9.54
11.20
12.04
13.18
Llama-3-88 Attention Layer
4.14
4.68
8.11
8.50
9.94
12.44
12.63
REASONING
4.52
7.08
8.38
8.79
10.56
11.33
12.10
12.57
12.87
Number of Samples
36
54
72
144
192
600
900
1632
3000
Evolutionary Search
2.11
2.27
3.90
4.10
5.07
6.60
6.62
9.31
9.13
DeepSeek-RI R1 MoB Layer
MCTS
5.93
6.33
6.79
6.87
6.93
7.24
8.18
8.84
8.90
REASONING COMPILER
7.05
8.34
8.53
9.10
9.45
11.74
11.74
Number Samples
36
72
150
200
600
1000
1500
3000
Evolutionary Search
2.22
2.44
2.73
2.73
4.64
4.71
5.61
FLUX Attention Layer
3.62
3.79
3.85
4.04
4.37
5.09
5.56
5.40
5. 5.64
REASONING COMPILER
4.48
4.67
4.89
5.37
5.42
5.43
5.59
5.60
5.67
Number of Samples
36
72
1.50
200
400
600
1000
1600
3000
Evolutionary Search
2.08
2.11
2.15
2.19
2.29
2.32
2.44
2,45
2.55
FLUX Convolution Layer
MCTS
2.11
2.13
2.18
2.37
2.38
2.38
2,44
2.44
2.45
REASONING COMPILER
2.21
2.30
2.29
2.36
2.47
2.47
2.51
2.55
Samples
20
1500
Evolutionary Search
1.36
2.28
3.61
5.59
5.75
5,94
5.94
Llama-4-Scoul MLP Layer
MCTS
1.76
2.51
4.05
5,41
7.83
8.13
8.58
8.90
8.90
REABONING COMPILER
12.74
12.74
12.74
12.75
12.75
13.24
13.26
13.52
13.79
16

---

C Impact of LLM Choice and Reasoning Strategy
As continuation of Figure a), Table reports speedup over unoptimized code on three additional benchmarks: DeepSeek-F1 R1 MoE Layer, FLUX Attention Layer, and FLUX Convolution Layer. Each block of the table corresponds a different benchmark and shows the best speedup achieved by the REASONING COMPILER as function of the number of schedules sampled using the reasoning model listed in the table. Rows compare different reasoning models used for API call genera- tion, including both proprietary (e.g., GPT-4o mini, OpenAl ol-mini) and open-source models (e.g., Llama3.3-Instruet, SeepSeek-Distilly Across all benchmarks, the results show that more capable models-those that are larger or samples. For example, 3--naa,.-Instuuct (70B) and DeepSeek-Diss (32B) achieve near-maximal speedup within the first 72-150 samples, while smaller models such as DeepSeck-Diss Llama3.I-Instruce (8B) reach similar performance more gradually. These results validate the gen- erality of findings: the use of context-aware LLMs accelerates convergence of the REASONING COMPILER across diverse code domains. Moreover, the performance of open-source models is com- petitive with proprietary alternatives, further supporting the accessibility and reproducibility of our method.
-on-soooseds--sseeeellel achieve higher speedups with fewer
(7B) or
Table 4: Speedup over unoptimized code across varying numbers of samples for different choices API call models.
Number of Samples
18
36
72
150
200
600
GPT-4o mini
4.52
7.08
8.38
8.79
10.56
11.33
Llama3-8B Attention Layer
OpenAl o1-mini
4.63
4.64
7.37
9.14
9.15
11.77
L 3-Instruct (70B)
5.15
9.68
9.69
9.80
9.80
9.81
DeepSeck-Disttll-- Bestill-Qwen&e (328)
7.07
8.14
8.23
8.77
8.78
9.98
-lamm3.1-Instree (8B)
3.60
5.87
6.28
8.46
8.63
10.52
DeepSeek-Distill-- weenn (7B)
4.06
4.86
6.68
Number of Samples
18
36
72
150
200
600
GPT-4o mini
6.14
7.05
8.33
8.53
9.10
9.45
DeepSeek-R1 MoB. Layer
OpenAl ol-mini
4.56
6.65
8.59
9.29
10.55
11.56
Llama3.3- --nnttsrtttt (70B)
7.30
7.70
7.96
8.06
8.60
9.22
DeepSeek-Distill-Ouee (32B)
5.56
8.11
9.49
10.17
11.02
12.02
Llama3. Innstruct (8B)
4.29
4.31
6.98
8.70
9.18
9.21
DeepSeek- SaaSSeek-Disilll--e Distill-Qwen (7B)
6.89
7.35
7.35
10.22
10.34
10.44
Number of Samples
18
36
72
150
200
600
GPT-4o mini
4.09
4.48
4.89
5.37
5.42
5.43
FLUX Attention Layer
OpenAl ol-mini
3.29
2.99
5.27
5.53
5.65
5.67
Llama3.3-Instree (70B)
2.67
3.12
4.82
4.86
5.71
5.71
DeepSeek-Disill (32B)
3.56
4.29
4.29
4.54
4.99
5,21
Llama3. Inntrruct (8B)
2.01
3.43
3.55
3.80
3.87
5.21
DeepSeck- Distill-Qwen (7B)
3.02
3.76
3.83
4.54
4.94
5.17
Number of Samples
18
36
72
150
200
600
GPT-4o mini
1.65
2.21
2.30
2.29
2.36
2.47
FLUX Convolution Layer
OpenAl o1-mini
2.37
2.37
2.38
2.39
2.45
2.54
Llama3.3-Instruue (70B)
2.30
2.35
2.47
2.51
2.56
2.57
DbepSeck-Distill--5 (32B)
1.41
2.26
2.32
2.35
2.40
2.45
Llama3. I-Instruct (8B)
2.11
2.30
2.39
2.55
2.56
DeepSeck Distill-Qwes (7B)
1.56
2.18
2.42
2.44
2.46
2.45
17

---

D Impact of Historical Trace Depth on Optimization Efficiency
As a continuation of Figure b). Table presents the data for the ablation study on the depth of historical trace included in the prompt sent to the LLM. Specifically, we compare two configurations: the "Parent + Grandparent" setting, where the prompt contains information from the current node and its two immediate ancestors, and the "Parent + Grandparent Great Great- Grandparent" setting, where the prompt additionally includes the great-grandparent node. These variations allow us to assess impact of deeper context windows on the effectiveness of the REASONING COMPILER.
Results show that increasing the historical context generally improves sample efficiency across all benchmarks. For example, on DeepSeek-R1 MoE Layer, adding one more ancestral node boosts early performance significantly, achieving 9.39x speedup at just 18 samples compared to 6.14x for the shallower context. Similarly, on Llama-3-8B Attention Layer, the extended context leads to higher final speedup (1 (11.87x vs. 11.33 x) and earlier convergence. The performance gains, while smaller, are also consistent ments observed across all sample budgets. These findings confirm that providing richer historical context enables the LLM to make more informed decisions at each step of the search, ultimately enhancing the sample efficiency of the REASONING COMPILER
FLUX Attention Layer and FLUX Convolution Layer, with improve-
Table 5: Speedup over unoptimized code across varying numbers of samples for different context lengths.
Number of Samples
18
36
72
150
200
600
Llama-3-88 3-8B Attention Layer
Parent Grandparent
4.52
7.08
8.38
8.79
10.56
11.33
Parent + Grundparent Grandprra
3.63
7.13
11,36
11.86
11.86
11.87
Numher of Samples
18
36
72
150
200
600
DeepSeek-P R1 MoE Layer
Parent Grandparent
6.14
7.05
8.33
8.53
9.10
9.45
Parent + Grandparent Grand-Grandnrren
9.39
10.31
10.31
10.49
10.59
10.65
Number of Samples
18
36
72
150
200
600
FLUX Attention Layer
Parent Grandparent
4.09
4.48
4.89
5.37
5.42
5,43
Parent + Grandparent Great- Gract--ranndareat
4.21
4.55
4.81
5.47
5.53
5.61
Number of Samples
18
36
72
150
200
600
FLUX Convolution Layer
Parent Grandparent
1.65
2.21
2,30
2.29
2.36
2,47
Parent Grandparent + Great Grand-rranndr
1.73
2.22
2.32
2.35
2,49
2.50
18

---

E Ablations of MCTS Branching Factor
To determine the value of MCTS branching factor (B), we ablate on 2 and 4. In Table
results show that when branching factor B = 2. the REASONING COMPILER is more sample-efficien than when factor is chosen, then there are more possible next steps, which require more sampling effort (i.e., more simulations) to cover these expanded possibilities
Our choice of B aligns with prior works 22 22 If higher branching
same level of thoroughness.
Table 6: Speedup over unoptimized code across varying numbers of samples for different branching factors.
Number of Samples
18
36
72
150
200
600
Llama-3-8 Attention Layer
B =
4.52
7.08
8.38
8.79
10.56
11.33
4.16
7.88
8.35
8.89
9.86
10.99
Number of Samples
18
72
150
200
600
DeepSeek-RI R1 MoE Layer
B =
6,14
7.05
8,33
8.53
9.10
9.45
B=4
2.98
4.29
4.29
7.28
7.29
9.10
Number of Samples
18
72
150
200
600
FLUX Attention Layer
B=2
4.09
4.48
4.89
5.37
5.42
5.43
B=4 4
2.40
3,48
3.97
4.95
4.97
5.55
Number of Samples
18
72
150
200
600
FLUX Convolution Layer
2
1.65
2.21
2.30
2.29
2.36
2.47
B = 4
1.91
1.97
2.23
2.23
2.25
2.43
19

---

Cost of LLMs Used in Experiments
In Table for each benchmark, we report the API cost of running full experiment with every LLM used to generate transformation proposals. We run high number of samples to understand the boundary of performance improvements and allow the algorithm to saturate. For OpenAl, our main results used GPT-4o mini, the lowest-cost model available at submission time. For open-source models, we used Hugging Face APIS through the Nscale hyperscaler provider. Across benchmarks, these open-source models achieved competitive speedups and sample efficiency relative to GPT- 4o mini, indicating that open-source models are viable alternative when commercial APIs are impractical. Costs of open-source models could be further reduced by local deployment.
Table 7: Cost of different LLM APIs per entire experiment (USD) across layer-wise and end-to- end benchmarks.
Model
Layer / Task
GPT-4o
OpenAl o1-mini
Llama3.3-
DeepSeek- Distill (32B)
Llama3.1- Instruct (8B)
DeepSeek-
mini
Instruct (70B)
Distill (7B)
Llama-3-8B Attention Layer
$0.89
$6.56
$2.07
$1.55
$0.31
$2.07
DeepSeek-R1 MoF. Layer
$0.90
$6.63
$2.09
$1.57
$0.31
$2.09
FLUX Attention Layer
S0.88
$6.47
$2.03
$1.52
$0.30
$2.03
FLUX Convolution Layer
$1.12
$8.25
$2.67
$2.00
$0.40
$2.67
Llama-4-Scout MLP Layer
$0.90
End-to-End -lama-3-8B
$1.59
G LLM Proposal Validity and Fallback Rates
transformations can occasionally be syntactically valid but semantically redundant or performance-regressic During any single MCTS expansion, proposals that fail basic validity checks (c.g., naming or use-context non-compliance) are simply discarded while the remaining valid proposals proceed, and no fallback triggered. A fallback occurs only when all LLM-generater proposals in that expansion are invalid, in which case the search reverts to the default, non-LLM expansion policy and continues without interruption. In Table g we report the fallback rate as the average fraction of expansions that trigger this non-LLMM path (i.e., expansions in which all LLM proposals are invalid). To prevent downstream harm from poor but valid transformations, the cost model evaluates all proposed transformations before they are added to the tree; proposals with low estimated values are naturally pruned. Because the transformation space a known, finite set of legal rewrites, most correctness issues reduce to naming compliance and use-context. which modern instruction-tunce LLMs typically handle well. Empirically, commercial models (GPT-4o mini and OpenAI ol ol-mini show 0% fallback rates, larger open-souree models perform similarly (Llama3.3 Instrrctt 70B at 0.08% and DeepSeek SeepSeck-Distill 32B at 0.17%), whereas smaller models exhibit higher fallback rates (Llama3. Instrrctt 8B at 10.50% and DeepSeek-Distill 7B 17.20%)
Table 8: Fallback rate by model used as the transformation proposal generator.
Model
Fallback Rate
GPT-4o mini
0%
OpenAl ol-mini
0%
Llama3.33 ---ma33.IInstruct (70B)
0.08%
DeepSeck-Dise (32B)
0.17%
Llama3. I IInstruct (8B)
10.50%
DeepSeek Diitsel (7B)
17.20%
20

---

NeurIPS Paper Checklist
1. Claims
Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Question:
Answer: [Yes]
Justification: We demonstrate accurately the paper's contributions and scope in the abstract, $
(introduction). and $ (results) to support the claims.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize other settings. It is fine to include aspirational goals motivation long it is clear that these goals are not attained by the paper.
2. Limitations
Question:
Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification: The authors acknowledge that their method currently depends on external APIs for querying large language models, which may pose reproducibility and scalability concerns due to cost and access restrictions. They also recognize that the system's 's perfor- mance can vary across model types and that the evaluation is limited to six representative oatt-of-theeas benchmarks. Moreover, since the approach relies on prompt formatting and reasoning traces, its effectiveness may degrade in settings where context length or LLM interpretability is constrained.
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those not discussed in the paper.
The authors encouraged to create separate "Liimitatonns" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The au- thors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the ap- proach. For example, facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to ad- dress problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't it acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
21

---

3. Theory assumptions and proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [NA]
Justification: This paper focuses on practical compiler optimization techniques rather than theoretical developments. As such, does not present formal theorems or proofs. However, we provide formal problem formulation to clearly define the optimization setting and guide our methodology. No theoretical claims are made that would require formal assumptions or correctness proofs.
in 2
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should clearly stated or referenced in the statement of any theo- rems.
The proofs can either appear in the main paper the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be comple- mented by formal proof' provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclu- sions of the paper (regardless of whether the code and data are provided or not)?
Answer: [Yes]
Justification: We have provided detailed experimental setup and included the link to our GitHub repository. We also described in detail our method in 33. and $32 to make sure our experiments can be reproduced.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, No answer this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided not.
If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is specific model and empirical evaluation, may be necessary to either make it possible for others to replicate the model with the same dataset, provide access to the model. In general. releasing code and data often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how replicate the results, access hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate the research performed.
While NeurIPS does not require releasing code, the conference does require all sub- missions to provide some reasonable avenue for reproducibility, which may depend on the nature the contribution. For example
(a) the contribution primarily new algorithm, the paper should make it clear how to reproduce that algorithm.
(b) If the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
22

---

(c) If the contribution is new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or a way re- produce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case au- thors are welcome to describe the particular way they provide for reproducibility. the case of dlosed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers have some path reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [Yes]
Justification: We included the link to our ourrepository in the abstract. The repository contains instructions how up and the experiments.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://nips.ccpmbbli//
guids//CodSuummisioooPiiii
for more details.
While we encourage the release code and data, we understand that this might not be possible, so "No" an acceptable answer. Papers cannot be rejected simply for including code, unless this is central to the contribution (e.g., for a new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (htips: //nips.cc/public/gee... wtwpb/iccguiieeCooSSuuiiiiiiiiie
for more details.
The authors should provide instructions on data access and preparation, including how access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing much information possible in supplemental material (appended the paper) is recommended, but including URLs to data and code is permitted.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: In $4.1 results.
we specified all the experiment details necessary to understand the
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should presented in the core of the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, as supplemental material.
7. Experiment statistical significance
Question: Does the paper report error bars suitably and correctly defined or other appropri- ate information about the statistical significance of the experiments?
23

---

Answer: [Yes]
Justification: All experiments are repeated 20 times, and the results are averaged to ensure statistical stability, as described in $4.1
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, or statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
It should be clear whether the error bar is the standard deviation or the standard error of the mean.
is OK to report I-sigma error bars, but one should state it. The authors should prefer- ably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that out of range (e.g. negative етгог rates).
If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information the com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification: We mention the machine details in $4.1 and the README in the GitHub repository provides the steps.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported the paper (e.g., preliminary or failed experiments that didn't t make it into the paper).
9. Code of ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurlPS Code of Ethics
Answer: [Yes]
Justification: The research conducted in the paper conforms, every respect, with the NeurlPS Code of Ethics.
Guidelines:
The answer NA means that the authors have not reviewed the NeurlPS Code Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., if there is a special consid- eration due to laws or regulations in their jurisdiction).

---

10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
This work presents compiler optimization framework that leverages LLM reasoning for efficient model serving. The positive societal impacts include reducing the computational cost of deploying large machine learning models, which in turn improves accessibility and scalability, as discussed in abstract, introduction, and conclusion (see | and $
Justification:
Guidelines:
The answer NA means that there no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact spe- cific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitiga- tion strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper deseribe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
The paper does not release any models associated datasets which have high risk of misuse. rather focuses on compiler-leve optimizations for efficient ML model serving, which poses direct safety misuse concerns that would warrant safeguards.
Justification:
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by re- quiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
25

---

Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification: Our method is integrated with Apache TVM v0.20.0 [0]] an open-source machine learning compiler stack released under the Apache License 2.0. We properly cite the original work [8] [T]] and ensure full compliance with its licensing terms. We also use OpenAl HuggingFace' model serving and utilize their APIs access the models.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset used and, if possible, include URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset. For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should provided.
assets are released, the license, copyright information, and terms of use the pack- age should be provided. For popular datasets, rated licenses for some datasets. Their licensing guide can help determine the license of dataset.
http:/ttcoode.commattaattttt has
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
If this information is not available online, the authors are encouraged to reach out to the asset's 's creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documenta- tion provided alongside the assets?
Answer: [Yes]
Justification: We integrate the proposed approach into open-source TVM scheduling and also make our code open source, as discussed in $4.1
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create anonymized URL or include an anonymized zip file.
14. Crowdsourcing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the pa- per include the full text instructions given participants and screenshots, applicable, as well as details about compensation (if any)?
Answer: [NA]
Justification:
The paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing research with human subjects.
Including this information in the supplemental material is fine, but if the main contri- bution of the paper involves human subjects, then as much detail as possible should be included in the main paper.
26

---

According to the NeurIPS Code of Ethics, workers involved in data collection, cura- tion, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals or equivalent for research with human subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects. and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: The paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Depending on the country in which research conducted, IRB approval (or equiva- lent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
16. Declaration of LLM usage
Does the paper describe the usage of LLMs if it is important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing, editing, formatting purposes and does not impact the core methodology, scientific rigorousness, or originality of the research, declaration is not required.
Question:
Answer: [Yes]
Justification: Large language models are an integral part of our method. We use them in the program optimization process to guide transformation proposals in compiler optimization search. This use of L.LMs is central, and described detail 3.1 All of our usage com- plies with responsible Al guidelines, and models used (e.g., OpenAl's models, LLaMA-3, DeepSeek) are publicly accessible using APIs.
Guidelines:
The answer NA means that the method development in this research does not involve LLMs as any important, original, or non non-standard components.
Please refer to our LLM policy futps//hhuriisscccooooterro/Dsst for what should should not be described.
27


Reviews:
[{"review_id": "mBSw3LxWC0", "reviewer": "Reviewer_4gxN", "summary": "This paper introduces Reasoning Compiler, which focuses on neural optimization, and does so via monte carlo tree search (MCTS) and reasoning. The paper shows how fewer samples can be used to guide the optimization direction, obtaining results that are better than a an unoptimized baseline compiled via evolutionary search.", "questions": "- Where is reasoning really needed? Can a rule-based MCTS with a big of engineering effort replace the reasoning portion of the tool?\n- Comparison with super-optimization.\n- Any additional details of the surrogate model?", "limitations": "Yes.", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "+ Important domain, runtime performance driven\n+ Strong evaluation and results\n+ Good explanation of methodology and approach.\n\n- Related work is not entirely well explained. Is reasoning necessary?\n- Ablation study is more a discussion section, than an ablation on where the improvements arise.\n\nOverall, I like this work, and appreciate the idea of using reasoning guided MCTS for compiler performance enhancement. The authors do a good job at explaining their approach in detail, how they run their experiments, and motivating the importance of their solution.\n\nMy main weaknesses from the work have to do with related work, and a subnote about the surrogate model for prediction. Regarding related work: the authors focused extensively on ML-guided approaches, but it was not clear that an ML-guided approach is truly needed (or at least, it was fully explained to this reader). The use of TVM and discussion of challenging search space exploration makes it more interesting as to whether reasoning really is required for MCTS to succeed. The authors themselves highlight that an LLM is not the centerpiece; this leads me to wonder if a rule-based MCTS guided approach would have done just as well without the large overheads involved with an LLM?\n\nAnother related work has to do with superoptimization as a field in general. Can the authors discuss how their approach compares/contrasts to compiler superoptimization? Is Reasoning Compiler just a unique subset, or does it actually have more to offer?\n\nFinally, this reviewer was very intrigued about the surrogate model used in the end of section 3, but it did not get enough real-estate in the paper! Such a model seems super useful in general, and could potentially be it's own contribution. Do the authors think so, or is this a simple model, and hence not a big component?", "was_revised": true, "final_justification": "Rebuttal was decent, and more results are added/promised for final paper.", "rebuttal": "We thank the reviewer for the thoughtful and constructive feedback. We’re glad you found our methodology and experimental results compelling, and we appreciate the opportunity to clarify several points. \n\n---\n### **Q1 Where is reasoning really needed**\n\nThank you for this insightful question. To clarify, our statement that *“the LLM is not the centerpiece of our contribution,* ***but a necessary enabler of effective search”*** was meant to emphasize that the key novelty lies in combining structured search (via MCTS) with **contextual, history-aware reasoning**—not in developing a new language model for compilation or optimization. However, we  do view the reasoning component as essential and a main enabler.\n\nYour question rightly asks whether a rule-based MCTS could be sufficient. Nonetheless, **contextual reasoning is a must** for compiler optimization due to the deep interdependence of transformations. Rule-based systems—even if embedded within MCTS—historically struggle in this space, particularly because they lack the flexibility to generalize across workloads and transformation paths.\n\nThis challenge is evident in the **superoptimization literature**, where early approaches (e.g., Massalin’s *Superoptimizer*, ASPLOS 1987) focused on exhaustive enumeration of transformation sequences. Even these exhaustive enumerations were found to be limited in scalability as the **“Stochastic Superoptimization”** paper (Schkufza et al., ASPLOS 2013) states and therefore shifts towards **randomized stochastic search (MCMC)**. \n\nRelated works such as Stochastic Superoptimization and TVM are a testament to the ineffectiveness of rule-based optimizations as the ASPLOS 2013 paper is the work that makes the leap to use random search for compiler optimization. TVM follows suit and uses evolutionary genetic search and simulated annealing algorithms. \n\n> For example, the caption of Figure 4 in the “Stochastic Superoptimization” (ASPLOS 2013) paper states that *“O0 and O3 optimized codes occupy a densely connected part of the space which is easily traversed. Expert code occupies an entirely different region of the space which is reachable only by way of an extremely low probability path.”*\n\nThis resonates with our experience in compiler optimization: **expert-designed heuristics are brittle**, and the performance-optimal regions are often isolated and hard to discover without global context.\n\n*Although these works overcome the limitation of rule-based approaches, they suffer from sample inefficiency that limits their exploration in practice and results in sub-optimal solutions.* As such we go even further beyond stochastic search and contribute **LLM-guided semantic reasoning** to propose structured, context-sensitive search of program transformations that incorporates \n- transformation history,  \n- hierarchical code structure,  \n- hardware cost feedback.\n\nThis enables the compiler to reason about interactions that rule-based or myopic policies would miss. Could this be replicated by a handcrafted rule-based MCTS? Possibly, but it would require significant engineering effort, domain-specific tuning, and still lack the adaptability and generality of a learned reasoning model.\n\nIn short, the LLM and reasoning **are not merely a convenience**: it provides *flexible, context-aware decision-making* that is critical to achieving high sample efficiency and generalization, particularly in settings where manual rules fall short—as the stochastic superoptimization literature has already demonstrated.\n\n--- \n### **Q2: Comparison to superoptimization**\n\nWe appreciate the reviewer’s suggestion to situate our work more explicitly relative to superoptimization. \n\nWhile our high-level goal of discovering highly efficient program variants shares motivation with superoptimization (e.g., Massalin ASPLOS 1987, Bansal and Aiken, ASPLOS 2006, Schkufza et al., ASPLOS 2013), the formulation and tractability of our problem differ substantially.\n\nSuperoptimization typically aims to find the *globally optimal* instruction-level program, often via exhaustive or stochastic search over low-level assembly variants. In contrast, our system operates over a constrained space of **high-level, legality-preserving transformations** (e.g., tiling, fusion, unrolling). That is, we do not attempt to synthesize arbitrary instruction sequences or perform unconstrained equivalence-preserving rewrites.\n\nInstead, our problem is best understood as sequencing legal high-level transformations over structured intermediate representations—closer in nature to a **planning problem amenable to contextual reasoning**, whereas superoptimization is closer to **low-level program synthesis**. \n\nOur approach is framed as a sequential decision-making process over a defined space of high-level transformations (e.g., tiling, fusion, vectorization), applied to structured intermediate representations such as TVM’s `IRModule`. This formulation—explicitly casted as a Markov Decision Process (MDP) in Section 2—lends itself naturally to structured search methods like MCTS and supports contextual reasoning over transformation sequences. In contrast, superoptimization aims to synthesize semantically equivalent low-level (often loop-free) assembly programs, typically using program synthesis techniques such as enumeration, symbolic reasoning, or stochastic search (as seen in Massalin ASPLOS 1987, Bansal and Aiken, ASPLOS 2006, Schkufza et al., ASPLOS 2013). These fundamental differences in abstraction level and search space shape the respective trade-offs between generality, scalability, and reasoning complexity. \n\nWhile our formulation sacrifices the generality of instruction-level search and optimization across all possible domains and programs, it significantly improves scalability and practical applicability to emerging neural workloads.\n\nDue to these foundational differences, we did not explicitly cite superoptimization literature in the original draft. We agree that a discussion contrasting objectives and search constraints would improve contextualization, and we will include it.\n\n---\n### **Q3: Surrogate model details**\n\nThank you for highlighting interest in the surrogate model. In our system, the surrogate (or cost) model is used to estimate hardware performance without requiring each candidate program to be compiled and executed—a well-known bottleneck in practical compiler optimization. \n\n*We use the default model provided by the TVM repository, which is based on XGBoost. We did not modify this model in our implementation. It has been widely adopted in both academic and commercial settings.*\n\nAs the AutoTVM paper (Chen et al., NeurIPS'18) discusses, the use of XGBoost allows the cost model to learn efficiently from relatively few samples—making it a practical choice for rapid autotuning. Additionally, Sections 4 and 6.2 of their paper explicitly explores transfer learning, where knowledge from previously tuned workloads is reused to accelerate optimization on new workloads. While XGBoost itself does not support transfer learning natively, the AutoTVM framework applies transfer across similar tasks by initializing the model with prior data and retraining—demonstrating strong generalization in practice.\n\nWhile surrogate modeling is indeed a powerful and broadly useful tool in this domain, it is not a novel or central contribution of our work. We will clarify better and include more details .\n\nOur use of their unmodified cost model is commensurate with the extensive literature on compiler optimizations and autotuning (Gibson and Cano. PACT’22, Ahn et al. DAC’22, Zhang et al. ICLR’21).\n\n1. We provide new results for two benchmarks (Llama-3-8B Attention Layer and Deepseek-R1 MoE Layer)  **on five platforms from four different hardware vendors (Apple M2 Pro, Amazon Graviton2, AMD EPYC 7R13, Intel Core i9, and Intel Xeon E3)** here and will include them in the paper with discussions. We have to acknowledge TVM, as it provides the hardware-agnostic implementation to build optimization innovations. We inherit the hardware agnosticism from its implementation, although we fundamentally change the optimization algorithm.\n\n2. Moreover, we report **end-to-end execution latency improvement for Llama3‑8B**, covering all *five platforms* from *four hardware vendors.* \n\n3. We also provide results of the *MLP layer from Llama‑4‑Scout‑17B‑16E‑Instruct*, showcasing performance improvements in **representative large-model layers and support for layers beyond what was originally included** in the paper on all these *five different platforms*.\n\n**Three tables for these new results are reported in response to “Reviewer secC.”**\n\nIn the tables, we measure **sample efficiency** using $\\frac{Speedup}{\\text{No.~} of Samples}$ and report the relative improvements of our Reasoning Compiler compared to TVM. The reduction in the absolute number of samples is also reported.\n\nAs the results in all three tables show, the Reasoning Compiler consistently achieves higher speedups in substantially fewer samples across five different platforms, for all the three sets of experiments mentioned above.\n\nFor **Llama-3-8B Attention Layer and Deepseek-R1 MoE Layer across the five platforms**, our Reasoning Compiler achieves $6.1\\times$ average speedup while TVM archives $3.3\\times$. Our compiler uses $6.5\\times$ fewer samples and exhibits $11.9\\times$ higher sample efficiency as the first table shows.\n\nFor **end-to-end execution of Llama3-8B across the five platforms**, the sample efficiency improvement compared with TVM ranges from $3.2\\times$ on AMD EPYC to $11.8\\times$ on Intel Core i9. The end-to-end speedups range from $2.2\\times$ on AMD EPYC to $5.1\\times$ on Amazon Graviton2. Reasoning Compiler consistently achieves significantly higher speedup ($4.0\\times$ geomean vs $2.8\\times$ geomean) with an average of $3.9\\times$ fewer samples.\n\n---\nOnce again, thank you for your detailed and encouraging feedback."}, {"review_id": "Xh3TXfCGqW", "reviewer": "Reviewer_secC", "summary": "The paper introduces reasoning compiler, a novel framework that integrates large language model (LLM) reasoning with Monte Carlo Tree Search (MCTS) to guide compiler optimization for accelerating model inference. Traditional neural compiler techniques struggle with the vast and interdependent transformation space and are often sample-inefficient. \n\nTo address the problem, Reasoning compiler treats optimization as a sequential, context-aware decision-making process. It uses LLMs to propose transformations based on program state and performance history, while MCTS enables structured exploration of optimization paths. Empirical results show that this hybrid approach achieves significant speedups (e.g., up to 7×–10×) over pre-optimized code using orders of magnitude fewer samples than black-box methods like Evolutionary Search, highlighting its superior sample efficiency and practical value for compiler tuning.", "questions": "1. Clarification on the cost model: The framework heavily relies on a learned hardware-aware cost model to simulate downstream performance. However, the paper doesn't explain how this model is trained and validated across different hardware platforms. Can the authors elaborate on how the cost model is built, Whether it is specific to one hardware target or transferable across platforms, The accuracy of the surrogate model compared to actual runtime?\n\n2. Can the authors discuss how well the approach generalize to the other layers of LLMs such as MLP layers and also report the end-to-end inference latency improvement? \n\nThe anonymized repository link doesn't seem to work for me.", "limitations": "yes", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strengths: \n+ Compiler optimization for neural network inference is of high practical importance\n+ The proposed approach is technically sound and well-implemented. It integrates large language model (LLM) reasoning with Monte Carlo Tree Search (MCTS) for compiler optimization. The problem is clearly formalized as a Markov Decision Process, and the empirical evaluation is rigorous.\n+ The core idea of casting compiler optimization as a context-aware reasoning task is interesting. It shows that the semantic understanding of LLMs can help navigating the non-linear optimization space \n\nWeakness: \n- While the high-level flow is clearly described in the work, details on the interaction between LLM proposals and MCTS explorations are unclear to me. Figure 2 is not that helpful. In Figure 2(b), what are the difference between Prog. i+1, .., Temp i+1, 1, ... Prog Sim.?   It seems that the work use both v_i and p_i to represent a program? What is and how to get a terminal program that can help estimate local cost without hardware execution? What is the cost model? \n- Although sample efficiency improvements are impressive, the final absolute performance gains relative to baseline methods (e.g., Evolutionary Search) are sometimes modest. The evaluation is limited to four kernels and does not explore how well the method generalizes to diverse hardware or larger compilation tasks.", "was_revised": true, "final_justification": "The proposed approach is technically sound and well-implemented. It integrates large language model (LLM) reasoning with Monte Carlo Tree Search (MCTS) for compiler optimization. The problem is clearly formalized as a Markov Decision Process, and the empirical evaluation is rigorous. I remain positive of the work.", "rebuttal": "We thank the reviewer for the thoughtful review and for recognizing the novelty, technical soundness, and practical significance of our approach. We address the comments in detail.\n\n---\n### **Q1: Clarification on the cost model**\nAs correctly noted, the theoretical framework relies on a cost model. This has become standard in compiler autotuning.\n\n*We use the default model provided by the TVM repository, which is based on XGBoost. We did not modify this model in our implementation. It has been widely adopted in both academic and commercial settings.*\n\nAs the AutoTVM paper (Chen et al., NeurIPS'18) discusses, the use of XGBoost allows the cost model to learn efficiently from relatively few samples—making it a practical choice for rapid autotuning. Additionally, Sections 4 and 6.2 of their paper explicitly explores transfer learning, where knowledge from previously tuned workloads is reused to accelerate optimization on new workloads. While XGBoost itself does not support transfer learning natively, the AutoTVM framework applies transfer across similar tasks by initializing the model with prior data and retraining—demonstrating strong generalization in practice.\n\nWhile surrogate modeling is indeed a powerful and broadly useful tool in this domain, it is not a novel or central contribution of our work. We will clarify better and include more details.\n\nOur use of their unmodified cost model is commensurate with the extensive literature on compiler optimizations and autotuning (Gibson and Cano. PACT’22, Ahn et al. DAC’22, Zhang et al. ICLR’21).\n\nWe evaluate all candidate schedules using this model, and only the final selected schedules are measured with actual hardware execution. While we do not claim new contributions in cost model design, we note that the model has been validated in prior work to provide reliable performance estimates for guiding the search.\n\nWhile the original AutoTVM paper does not report explicit accuracy metrics (e.g., R^2 or MAE) for the cost model, this seems intentional. In autotuning, the goal of the cost model is not to predict absolute runtime precisely, but rather to rank program candidates effectively so that search can focus on promising regions of the space. Thus, the utility of the cost model is typically validated through its impact on search efficiency and final performance.\n\n1. We provide new results for two benchmarks (Llama-3-8B Attention Layer and Deepseek-R1 MoE Layer)  **on five platforms from four different hardware vendors (Apple M2 Pro, Amazon Graviton2, AMD EPYC 7R13, Intel Core i9, and Intel Xeon E3)** here and will include them in the paper with discussions. We have to acknowledge TVM, as it provides the hardware-agnostic implementation to build optimization innovations. We inherit the hardware agnosticism from its implementation, although we fundamentally change the optimization algorithm.\n\n2. Moreover, we report **end-to-end execution latency improvement for Llama3‑8B**, covering all *five platforms* from *four hardware vendors.* \n\n3. We also provide results of the *MLP layer from Llama‑4‑Scout‑17B‑16E‑Instruct*, showcasing performance improvements in **representative large-model layers and support for layers beyond what was originally included** in the paper on all these *five different platforms*.\n\n|**Hardware Platform**|**Benchmark**|**TVM # Samples**|**TVM Speedup**|**Ours # Samples**|**Ours Speedup**|**Reduction in # Samples**|**Improvement in Sample Efficiency**|\n|---------------------|-------------|---------------|----------------|----------------|----------------|---------------------------|------------------------------------|\n|Apple M2 Pro|Llama-3-8B Attention Layer|1010|3.3|190|9.7|5.3×|15.6×|\n||Deepseek-R1 MoE  Layer|1040|2.8|230|4.8|4.5×|7.8×|\n|Amazon Graviton2|Llama-3-8B Attention Layer|510|3.9|60|5.1|8.5×|11.1×|\n||Deepseek-R1 MoE  Layer|980|2.67|150|5.9|6.5×|14.4×|\n|AMD EPYC 7R13|Llama-3-8B Attention Layer|1400|2.1|200|12.1|7.0×|40.3×|\n||Deepseek-R1 MoE  Layer|2290|1.7|330|2.3|6.9×|9.4×|\n|Intel Xeon E3|Llama-3-8B Attention Layer|2760|3.9|320|5.8|8.6×|12.8×|\n||Deepseek-R1 MoE  Layer|1000|3.7|180|4.4|5.6×|6.6×|\n|Intel Core i9|Llama-3-8B Attention Layer|920|10.5|130|11|7.1×|7.4×|\n||Deepseek-R1 MoE  Layer|1632|9.1|192|9.1|8.5×|8.5×|\n|**Geomean**|||**3.3×**||**6.1×**|**6.5×**|**11.9×**|\n\n|**Hardware Platform**|**Benchmark**|**TVM # Samples**|**TVM Speedup**|**Ours # Samples**|**Ours Speedup**|**Reduction in # of Samples**|**Improvement in Sample Efficiency**|\n|---------------------|-------------|------------------|----------------|-------------------|----------------|-----------------------------|------------------------------------|\n|Apple M2 Pro|End-to-End Llama-3-8B|4820|2.2×|1770|3.9×|2.7×|4.8×|\n|Amazon Graviton2|End-to-End Llama-3-8B|4560|3.7×|1440|5.1×|3.2×|4.4×|\n|AMD EPYC 7R13|End-to-End Llama-3-8B|410|2.0×|140|2.2×|2.9×|3.2×|\n|Intel Xeon E3|End-to-End Llama-3-8B|4640|5.0×|670|5.0×|6.9×|6.9×|\n|Intel Core i9|End-to-End Llama-3-8B|3800|2.2×|720|4.9×|5.3×|11.8×|\n|**Geomean**| |—|**2.8×**|—|**4.0×**|**3.9×**|**5.6×**|\n\n|**Hardware Platform**|**Benchmark**|**TVM # Samples**|**TVM Speedup**|**Ours # Samples**|**Ours Speedup**|**Reduction in # of Samples**|**Improvement in Sample Efficiency**|\n|---------------------|-------------|------------------|----------------|-------------------|----------------|-----------------------------|------------------------------------|\n|Apple M2 Pro|MLP Layer Llama‑4‑Scout‑17B‑16E‑Instruct|2460|2.2×|440|3.4×|5.6×|8.6×|\n|Amazon Graviton2|MLP Layer Llama‑4‑Scout‑17B‑16E‑Instruct|1630|1.7×|500|4.0×|3.3×|7.7×|\n|AMD EPYC 7R13|MLP Layer Llama‑4‑Scout‑17B‑16E‑Instruct|510|6.4×|100|10.2×|5.1×|8.1×|\n|Intel Xeon E3|MLP Layer Llama‑4‑Scout‑17B‑16E‑Instruct|1200|2.0×|300|6.1×|4.0×|12.2×|\n|Intel Core i9|MLP Layer Llama‑4‑Scout‑17B‑16E‑Instruct|230|5.6×|20|12.7×|11.5×|26.1×|\n|**Geomean**| |—|**3.1×**|—|**6.4×**|**5.3×**|**11.1×**|\n\n---\n### **Q2: Generalization to other layers and end-to-end results**\n\nWe appreciate the reviewer’s insightful question. Our technique is not limited to any specific layer type. As long as a layer is supported by TVM, our method can be applied without modification. This flexibility stems from our implementation being built directly on top of TVM's abstraction layers, enabling us to target a wide variety of architectures and operators.\n\nIn the tables, we measure **sample efficiency** using $\\frac{Speedup}{\\text{No.~} of Samples}$ and report the relative improvements of our Reasoning Compiler compared to TVM. The reduction in the absolute number of samples is also reported.\n\nAs the results in all three tables show, the Reasoning Compiler consistently achieves higher speedups in substantially fewer samples across five different platforms, for all the three sets of experiments mentioned above.\n\nFor **Llama-3-8B Attention Layer and Deepseek-R1 MoE Layer across the five platforms**, our Reasoning Compiler achieves $6.1\\times$ average speedup while TVM archives $3.3\\times$. Our compiler uses $6.5\\times$ fewer samples and exhibits $11.9\\times$ higher sample efficiency as the first table shows.\n\nFor **end-to-end execution of Llama3-8B across the five platforms**, the sample efficiency improvement compared with TVM ranges from $3.2\\times$ on AMD EPYC to $11.8\\times$ on Intel Core i9. The end-to-end speedups range from $2.2\\times$ on AMD EPYC to $5.1\\times$ on Amazon Graviton2. Reasoning Compiler consistently achieves significantly higher speedup ($4.0\\times$ geomean vs $2.8\\times$ geomean) with an average of $3.9\\times$ fewer samples.\n\nFor the **large MLP layer** from **Llama‑4‑Scout‑17B‑16E‑Instruct**, on Intel i9, Reasoning Compiler achieves a $12.7\\times$ speedup in just 20 samples, while TVM reaches only $5.6\\times$ after 230 samples. This corresponds to an $11.5\\times$ reduction in the number of samples and a $26.1\\times$ improvement in sample efficiency compared with TVM. Similar trends are observed across all the five platforms. For AMD EPYC, our compiler achieves $10.2\\times$ speedup in 100 samples while TVM achieves $6.4\\times$ in 510 samples. \n\nThese results support the broad applicability and practical benefits of our method across layers, models, and hardware.\n\n---\n### **W1: Clarification on the interaction between LLM and MCTS (Figure 2):**\n\nThank you for pointing this out. We agree with your comments and will revise Figure 2.\n\nWe use $v_i$ to denote a node in the tree, and $p_i$ to denote the program stored in node $v_i$. In Figure 2, we used notations such as $\\textsf{Prog. i}$ instead of $p_i$ for better visuals, which seems to have caused confusion instead. We apologize.\n\nIn Figure 2(a), the compiler applies a sequence of LLM suggested transformations to $\\textsf{Prog. i}$, which is the plan resulting from LLM reasoning to transform $\\textsf{Prog. i}$ to $\\textsf{Prog. i+1}$. The compiler applies transformations in this sequence one-by-one, generating temporary intermediate programs denoted by $\\textsf{Temp i+1,1}$, $\\textsf{Temp i+2,2}$, etc. After all of these temporary states, the compiler reaches $\\textsf{Prog. i+1}$. This is the MCTS Expansion stage.\n\nThe next stage (Figure 2(b)) is the MCTS simulation stage, where the algorithm tries to evaluate the value of $\\textsf{Prog. i+1}$ while considering its potential future transformations. Therefore, the MCTS algorithm generates a sequence of random transformations to simulate and assess how valuable $\\textsf{Prog. i+1}$ is. The terminal program ($\\textsf{Prog. Sim.}$) is attained after the compiler applies a sequence of random transformations one-by-one to $\\textsf{Prog. i+1}$ until no more valid transformations could be applied. It was an oversight to denote the temporary programs as $\\textsf{Temp i+1,1}$, same as Figure 2(a)--they should have been $\\textsf{Dump i+1,1}$.\n\n---\n### **Repository**\nHovering your mouse over the word \"anonymized\" shows the link in the bottom corner of the browser.\n\n---\nThank you for your insightful inquiries to our work."}, {"review_id": "Fv1ptTT53T", "reviewer": "Reviewer_SxhP", "summary": "This paper proposes the reasoning compiler, a framework that optimizes compilation with LLM and Monte Carlo tree search. Using the reasoning compiler, the authors perform compiler optimization as a sequential search process with contextual awareness. Experimental results reveal that the proposed method achieves comparable or superior performance compared to previous evolutionary search method with significantly smaller sample sizes.", "questions": "Please see the weaknesses above. On top of the two weaknesses, can the authors also discuss about whether it is possible to use the generated optimization traces to further fine-tune models for better optimization performance?", "limitations": "yes", "rating": 4, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strengths:\n\n1. **Clarity** The paper is well written and easy to follow. The method is described clearly and concisely, and results are clearly presented.\n\n2. **Significance** The topic is important. It is important to optimize compiler performance for better downstream machine learning task performance.\n\n3. **Originality** The proposed method is novel.\n\n4. **Quality** The authors conduct carefully designed, comprehensive experiments to show that the proposed reasoning compiler works better than baseline methods.\n\nWeaknesses:\n\n1. Lack of varied hardware environments: The experiment is fixed to one hardware environment, namely \"Intel Core i9 CPU using Apache TVM v0.20.0\" (lines 245-246). Is the advantage of the proposed method agnostic to hardware environments?\n\n2. Lack of discussion about downstream implications: Can the authors discuss in more detail the implications of this work to the downstream applications of machine learning model tuning or inference?", "was_revised": true, "final_justification": "My concerns are resolved.", "rebuttal": "We thank the reviewer for the positive and constructive comments, and we appreciate the recognition of our work’s clarity, originality, and significance. Below, we address the two primary comments and the additional question regarding fine-tuning.\n\n---\n### **W1: Hardware agnosticism**\n\nWe appreciate your comment and we emphasize that our approach is hardware agnostic. We have to acknowledge TVM, as it provides the hardware-agnostic implementation to build optimization innovations. We inherit the hardware agnosticism from its implementation, although we fundamentally change the optimization algorithm. \n\n*We use the default surrogate cost model provided by the TVM repository, which is based on XGBoost. We did not modify this model in our implementation. It has been widely adopted in both academic and commercial settings.*\n\nAs the AutoTVM paper (Chen et al., NeurIPS'18) discusses, the use of XGBoost allows the cost model to learn efficiently from relatively few samples—making it a practical choice for rapid autotuning. Additionally, Sections 4 and 6.2 of their paper explicitly explores transfer learning, where knowledge from previously tuned workloads is reused to accelerate optimization on new workloads. While XGBoost itself does not support transfer learning natively, the AutoTVM framework applies transfer across similar tasks by initializing the model with prior data and retraining—demonstrating strong generalization in practice.\n\nWhile surrogate modeling is indeed a powerful and broadly useful tool in this domain, it is not a novel or central contribution of our work. We will clarify better and include more details.\n\nOur use of their unmodified cost model is commensurate with the extensive literature on compiler optimizations and autotuning (Gibson and Cano. PACT’22, Ahn et al. DAC’22, Zhang et al. ICLR’21).\n\n1. We provide new results for two benchmarks (Llama-3-8B Attention Layer and Deepseek-R1 MoE Layer)  **on five platforms from four different hardware vendors (Apple M2 Pro, Amazon Graviton2, AMD EPYC 7R13, Intel Core i9, and Intel Xeon E3)** here and will include them in the paper with discussions.\n\n2. Moreover, we report **end-to-end execution latency improvement for Llama3‑8B**, covering all *five platforms* from *four hardware vendors.* \n\n3. We also provide results of the *MLP layer from Llama‑4‑Scout‑17B‑16E‑Instruct*, showcasing performance improvements in **representative large-model layers and support for layers beyond what was originally included** in the paper on all these *five different platforms*.\n\n|**Hardware Platform**|**Benchmark**|**TVM # Samples**|**TVM Speedup**|**Ours # Samples**|**Ours Speedup**|**Reduction in # Samples**|**Improvement in Sample Efficiency**|\n|---------------------|-------------|---------------|----------------|----------------|----------------|---------------------------|------------------------------------|\n|Apple M2 Pro|Llama-3-8B Attention Layer|1010|3.3|190|9.7|5.3×|15.6×|\n||Deepseek-R1 MoE  Layer|1040|2.8|230|4.8|4.5×|7.8×|\n|Amazon Graviton2|Llama-3-8B Attention Layer|510|3.9|60|5.1|8.5×|11.1×|\n||Deepseek-R1 MoE  Layer|980|2.67|150|5.9|6.5×|14.4×|\n|AMD EPYC 7R13|Llama-3-8B Attention Layer|1400|2.1|200|12.1|7.0×|40.3×|\n||Deepseek-R1 MoE  Layer|2290|1.7|330|2.3|6.9×|9.4×|\n|Intel Xeon E3|Llama-3-8B Attention Layer|2760|3.9|320|5.8|8.6×|12.8×|\n||Deepseek-R1 MoE  Layer|1000|3.7|180|4.4|5.6×|6.6×|\n|Intel Core i9|Llama-3-8B Attention Layer|920|10.5|130|11|7.1×|7.4×|\n||Deepseek-R1 MoE  Layer|1632|9.1|192|9.1|8.5×|8.5×|\n|**Geomean**|||**3.3×**||**6.1×**|**6.5×**|**11.9×**|\n\n|**Hardware Platform**|**Benchmark**|**TVM # Samples**|**TVM Speedup**|**Ours # Samples**|**Ours Speedup**|**Reduction in # of Samples**|**Improvement in Sample Efficiency**|\n|---------------------|-------------|------------------|----------------|-------------------|----------------|-----------------------------|------------------------------------|\n|Apple M2 Pro|End-to-End Llama-3-8B|4820|2.2×|1770|3.9×|2.7×|4.8×|\n|Amazon Graviton2|End-to-End Llama-3-8B|4560|3.7×|1440|5.1×|3.2×|4.4×|\n|AMD EPYC 7R13|End-to-End Llama-3-8B|410|2.0×|140|2.2×|2.9×|3.2×|\n|Intel Xeon E3|End-to-End Llama-3-8B|4640|5.0×|670|5.0×|6.9×|6.9×|\n|Intel Core i9|End-to-End Llama-3-8B|3800|2.2×|720|4.9×|5.3×|11.8×|\n|**Geomean**| |—|**2.8×**|—|**4.0×**|**3.9×**|**5.6×**|\n\n**The table for the MLP layer from Llama‑4‑Scout‑17B‑16E‑Instruct is reported in response to “Reviewer secC.”**\n\nIn the tables, we measure **sample efficiency** using $\\frac{Speedup}{\\text{No.~} of Samples}$ and report the relative improvements of our Reasoning Compiler compared to TVM. The reduction in the absolute number of samples is also reported.\n\nAs the results in all three tables show, the Reasoning Compiler consistently achieves higher speedups in substantially fewer samples across five different platforms, for all the three sets of experiments mentioned above.\n\nFor **Llama-3-8B Attention Layer and Deepseek-R1 MoE Layer across the five platforms**, our Reasoning Compiler achieves $6.1\\times$ average speedup while TVM archives $3.3\\times$. Our compiler uses $6.5\\times$ fewer samples and exhibits $11.9\\times$ higher sample efficiency as the first table shows.\n\nFor **end-to-end execution of Llama3-8B across the five platforms**, the sample efficiency improvement compared with TVM ranges from $3.2\\times$ on AMD EPYC to $11.8\\times$ on Intel Core i9. The end-to-end speedups range from $2.2\\times$ on AMD EPYC to $5.1\\times$ on Amazon Graviton2. Reasoning Compiler consistently achieves significantly higher speedup ($4.0\\times$ geomean vs $2.8\\times$ geomean) with an average of $3.9\\times$ fewer samples.\n\nFor the **large MLP layer** from **Llama‑4‑Scout‑17B‑16E‑Instruct**, on Intel i9, Reasoning Compiler achieves a $12.7\\times$ speedup in just 20 samples, while TVM reaches only $5.6\\times$ after 230 samples. This corresponds to an $11.5\\times$ reduction in the number of samples and a $26.1\\times$ improvement in sample efficiency compared with TVM. Similar trends are observed across all the five platforms. For AMD EPYC, our compiler achieves $10.2\\times$ speedup in 100 samples while TVM achieves $6.4\\times$ in 510 samples. \n\n---\n\n### **W2: Downstream implications**\n**Cost efficiency:** These improvements directly translate into meaningful reductions in the operational cost of LLM inference—measured not only in dollars but also in energy consumption and environmental impact. Because our Reasoning Compiler consistently achieves $2$–$5\\times$ in required samples and end-to-end runtime across diverse hardware platforms, the compute workload—and consequently, the billing for cloud or on-premise usage—is reduced by a similar factor. This is highly impactful, especially as LLM inference costs now constitute a significant portion of infrastructure budgets for startups, enterprises, and research labs alike.\n\n**Energy and environmental impact:** Inference runtime is directly tied to energy usage. Every millisecond reduction in execution time lowers the energy consumed per query, thereby contributing to a smaller carbon footprint. Given the growing scale and adoption of LLMs, this efficiency gain scales to material environmental benefits when deployed at production volume.\n\n**Responsiveness and usability:** Our method also improves system responsiveness, which is critical for latency-sensitive applications like real-time chat, AI assistants, and interactive agents. Faster model execution enhances user experience and expands the viability of deploying large models in constrained environments such as edge devices or mobile apps.\n\n**Agile deployment and finetuning:** By significantly reducing the number of samples required for optimization, our approach enables more agile model deployment, faster iteration cycles, and lower cost per tuning run in practical machine learning workflows. \n\n**Self-optimizing inference.** One particularly exciting direction is the possibility of bootstrapping model inference itself. For instance, when the LLM guiding compilation is optimizing inference for its own architecture, a virtuous cycle is created: faster inference enables lower-latency querying of the model, which in turn accelerates the compilation process. This feedback loop opens up new possibilities for self-improving systems and continual performance refinement.\n\nWe thank the reviewer for the thoughtful prompt that opened the door to this discussion, and we will include it.\n\n---\n### **Q1: Possibility of using the generated optimization traces to fine-tune models for better optimization**\n\nThis is a valuable question. While we did not fine-tune the LLM in our current work, we believe that fine-tuning using optimization traces is a promising direction.\nOur current system demonstrates strong performance without any fine-tuning, enabling plug-and-play integration into compiler infrastructures without requiring retraining or labeled data. This makes our method broadly applicable and easy to adopt across different hardware and workloads.\nThat said, fine-tuning could likely yield even stronger optimization capabilities, particularly when targeting specific hardware or workload distributions. However, this introduces a tradeoff: fine-tuning on one hardware target may reduce portability and generalization to others — a limitation we deliberately avoided to preserve flexibility.\nNonetheless, it is intuitive to fine-tune the model using optimization traces generated by MCTS. These traces can be filtered to produce high-quality training data, enabling a form of self-improvement where the model learns from its own refined experience.\n\n---\nThank you again for your insightful questions and feedback. We will revise the paper to clarify these points, incorporate the additional cross-hardware results, and expand the discussion of downstream implications and future work."}, {"review_id": "x5zLVymocz", "reviewer": "Reviewer_jGo8", "summary": "This paper presents the REASONING COMPILER, a novel compiler optimization framework that integrates large language model (LLM) reasoning with Monte Carlo Tree Search (MCTS) to improve the sample efficiency and performance of code optimization for model serving workloads. Recognizing that modern neural compilers face challenges due to the vast and interdependent space of transformation sequences, the authors cast compilation as a sequential decision-making process and use LLMs to propose context-aware transformations. These LLM-guided proposals are integrated into MCTS, which balances exploration and exploitation while leveraging a learned cost model to guide search without incurring real hardware evaluations.\n\nThe authors evaluate their method across four diverse neural network operators from LLaMA3, DeepSeek, and Stable Diffusion workloads. Results show that REASONING COMPILER significantly outperforms traditional black-box autotuners like Evolutionary Search, achieving up to 15–16× improvements in sample efficiency. Ablation studies further validate the contributions of the LLM proposal engine, the effect of model scale, and the utility of historical transformation context in prompts.", "questions": "- Can the authors report on the cost (e.g., time or monetary) of LLM queries per program? Would open-source models suffice in settings where commercial API use is impractical?\n- How well does the learned surrogate cost model generalize across unseen programs or hardware architectures? Is it retrained per workload?\n- Have the authors evaluated performance on full end-to-end inference pipelines, beyond isolated operators? If not, what are the key barriers?\n- Do LLMs sometimes propose transformations that are syntactically valid but semantically redundant or harmful? How often is fallback to random search triggered?", "limitations": "N/A", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 4, "clarity": 3, "significance": 3, "originality": 4, "strengths_and_weaknesses": "Strengths\n- Demonstrates a novel combination of LLM reasoning with MCTS for compiler optimization (Section 3).\n- Clearly articulates the motivation for contextual, history-aware transformation proposals, with concrete examples of transformation dependencies (e.g., Section 3.1, Figure 1).\n- Empirical results show strong improvements in sample efficiency and performance across four diverse benchmarks, with up to 7–10× speedups using fewer than 40 samples (Section 4.2, Figure 3).\n- Ablation studies substantiate the importance of LLM quality, instruction tuning, and prompt history depth (Section 4.3, Figure 5).\n- Open-sourced implementation enables reproducibility and adoption by the broader community (Section 4.1).\n- Clarity in problem formalization, with a structured MDP and integration of LLMs into each MCTS stage (Section 2, Section 3.2).\n\nWeaknesses\n- The paper does not perform end-to-end evaluations on full model inference pipelines—only individual operators are benchmarked (Section 4.1).\n- LLM proposal quality is only evaluated in terms of downstream performance; there is no analysis of proposal diversity, failure modes, or qualitative reasoning quality.\n- The cost of querying commercial LLM APIs (e.g., OpenAI) is not discussed, which could affect deployment feasibility (Section 4.1).\n- Claims around open-source LLMs being competitive could benefit from more head-to-head comparisons on the same operator and hardware budgets (Section 4.3.1).\n- The simulation step in MCTS uses a learned surrogate cost model, but the generalization accuracy and training procedure of this model are underdescribed (Section 3.2, Section 4.1).", "was_revised": true, "final_justification": "I’m maintaining an accept recommendation because the paper cleanly integrates LLM-guided proposals with MCTS for compiler optimization, with a clear MDP setup and strong sample-efficiency gains over black-box autotuners. The rebuttal addressed my main concerns by adding cross-hardware results (five platforms) and end-to-end Llama-3-8B runs that show higher speedups with fewer trials than TVM. The authors clarified that the surrogate cost model is TVM’s standard XGBoost model used only for ranking candidates, which matches common autotuning practice. They also reported LLM usage costs with per-experiment tables and showed low fallback rates, which reduces risk from poor proposals. Some presentation issues remain (e.g., figure clarity) and the main paper still emphasizes kernel-level evaluations, but the new results and planned edits help. I encourage the authors to tighten figure captions and expand the end-to-end evaluations in the final version. On balance, the novelty, evidence, and practical relevance for model serving outweigh the remaining limitations, so I keep my positive rating.", "rebuttal": "We sincerely thank the reviewer for the thoughtful and detailed feedback. We are glad the novelty, empirical rigor, and potential impact of our work came through clearly. Below, we address the specific comments and questions.\n\n---\n### **Q1: Cost of LLMs**\nWe appreciate this important practical consideration. From the OpenAI models, we only used the cheapest and not the most capable model at the time of submission (GPT-4o mini) for our main results. For open-source models, because of the time constraints during submission, we used the Hugging Face APIs through Nscale hyperscaler provider.  \n\nAs the results in Section 4.3.1 show, the open-source models (especially Llama3.3-Instruct (70B) and DeepSeek-Distill (32B)) provide competitive sample efficiency and speedup gains in comparison with GPT-4o mini. Their cost can be minimized by local deployment. \n\nAnd yes, the competitive results suggest that the open-source models can suffice when commercial API use is not practical.\n\nNonetheless, the following table reports the cost of every experiment for each benchmark. We were not particularly cost conscious about running the experiments and let the experiments run a high number of samples to understand the boundary of performance improvements and allow the algorithm to saturate. \n\nOverall, we have spent about 350 USD with OpenAI models which includes all the experiments, development, and debugging runs for the submission and rebuttal. With Hugging Face, we have spent about 90 USD that also includes all the experiments, development, and debugging.\n\n|**Cost per Entire Experiment**|**GPT-4o mini**|**OpenAI o1-mini**|**LLaMA3.3-Instruct (70B)**|**DeepSeek-Distill (32B)**|**LLaMA3.1-Instruct (8B)**|**DeepSeek-Distill (7B)**|\n|-----------------------------|---------------|------------------|---------------------------|---------------------------|--------------------------|--------------------------|\n|Llama-3-8B – Attention Layer|$0.89|$6.56|$2.07|$1.55|$0.31|$2.07|\n|DeepSeek-R1 – MoE Layer|$0.90|$6.63|$2.09|$1.57|$0.31|$2.09|\n|FLUX – Attention Layer|$0.88|$6.47|$2.03|$1.52|$0.30|$2.03|\n|FLUX – Convolution|$1.12|$8.25|$2.67|$2.00|$0.40|$2.67|\n|End-to-End Llama-3-8B|$1.59||||||\n|Llama‑4‑Scout‑17B‑16E‑Instruct – MLP Layer|$0.90|||||||\n\n\n---\n### **Q2: Surrogate cost model generalization**\n\nWe appreciate the reviewer’s attention to the cost model. The cost model generalizes to many platforms, but is not our work or contribution. It is a native part of TVM and one of its important contributions.\n\n*We use the default model provided by the TVM repository, which is based on XGBoost. We did not modify this model in our implementation. It has been widely adopted in both academic and commercial settings.*\n\nAs the AutoTVM paper (Chen et al., NeurIPS'18) discusses, the use of XGBoost allows the cost model to learn efficiently from relatively few samples—making it a practical choice for rapid autotuning. Additionally, Sections 4 and 6.2 of their paper explicitly explores transfer learning, where knowledge from previously tuned workloads is reused to accelerate optimization on new workloads. While XGBoost itself does not support transfer learning natively, the AutoTVM framework applies transfer across similar tasks by initializing the model with prior data and retraining—demonstrating strong generalization in practice. The XGBoost model gets trained on the fly although it benefits from transfer learning across runs. The available documentation suggests that it is not necessary to train the model per workload.\n\nWhile surrogate modeling is indeed a powerful and broadly useful tool in this domain, it is not a novel or central contribution of our work. We will clarify better and include more details.\n\n\nOur use of their unmodified cost model is commensurate with the extensive literature on compiler optimizations and autotuning (Gibson and Cano. PACT’22, Ahn et al. DAC’22, Zhang et al. ICLR’21).\n\n1. We provide new results for two benchmarks (Llama-3-8B Attention Layer and Deepseek-R1 MoE Layer)  **on five platforms from four different hardware vendors (Apple M2 Pro, Amazon Graviton2, AMD EPYC 7R13, Intel Core i9, and Intel Xeon E3)** here and will include them in the paper with discussions. We have to acknowledge TVM, as it provides the hardware-agnostic implementation to build optimization innovations. We inherit the hardware agnosticism from its implementation, although we fundamentally change the optimization algorithm.\n\n2. Moreover, we report **end-to-end execution latency improvement for Llama3‑8B**, covering all *five platforms* from *four hardware vendors.* \n\n3. We also provide results of the *MLP layer from Llama‑4‑Scout‑17B‑16E‑Instruct*, showcasing performance improvements in **representative large-model layers and support for layers beyond what was originally included** in the paper on all these *five different platforms*.\n\n**Three tables for these new results are reported in response to “Reviewer secC.”**\n\nIn the tables, we measure **sample efficiency** using $\\frac{Speedup}{\\text{No.~} of Samples}$ and report the relative improvements of our Reasoning Compiler compared to TVM. The reduction in the absolute number of samples is also reported.\n\nAs the results in all three tables show, the Reasoning Compiler consistently achieves higher speedups in substantially fewer samples across five different platforms, for all the three sets of experiments mentioned above.\n\nFor **Llama-3-8B Attention Layer and Deepseek-R1 MoE Layer across the five platforms**, our Reasoning Compiler achieves $6.1\\times$ average speedup while TVM archives $3.3\\times$. Our compiler uses $6.5\\times$ fewer samples and exhibits $11.9\\times$ higher sample efficiency as the first table shows.\n\n---\n### **Q3: End-to-end evaluation and barriers**\n\nOur approach and TVM can be used for end-to-end optimizations but engineering challenges make experimentation limited. Converting PyTorch models to TVM IR is difficult because several of the operators are not supported. Other challenges include inter-operator dependencies that may affect scheduling, more complex memory reuse patterns and layout constraints, and increased runtime variability due to kernel fusion or asynchronous execution.\n\nFor **end-to-end execution of Llama3-8B across the five platforms**, the sample efficiency improvement compared with TVM ranges from $3.2\\times$ on AMD EPYC to $11.8\\times$ on Intel Core i9. The end-to-end speedups range from $2.2\\times$ on AMD EPYC to $5.1\\times$ on Amazon Graviton2. Reasoning Compiler consistently achieves significantly higher speedup ($4.0\\times$ geomean vs $2.8\\times$ geomean) with an average of $3.9\\times$ fewer samples.\n\n\n---\n### **Q4: Redundant transformations and fallback**\n\nThank you for this insightful question. Yes, LLM proposals can occasionally be syntactically valid but semantically redundant or performance-regressive. During one expansion process, if all the transformations proposed by LLM are invalid, the process falls back to MCTS with a default (non-LLM) policy, effectively continuing the search. However, if only some of the transformations are invalid, those ones are skipped and the rest are still applied and no fall back is triggered. To prevent downstream harm from poor but valid transformations, the cost model evaluates all proposed transformations before they are added to the tree, so semantically poor proposals are naturally pruned due to a poor estimated value. Our transformation space is constrained to a known set of valid transformations, so correctness is primarily a matter of naming compliance and context usage, both of which the LLM handles well in most cases. \n\n|**Model**|**Fallback Rate**|\n|---------|-----------------|\n|GPT-4o mini|0%|\n|OpenAI o1-mini|0%|\n|LLaMA3.3-Instruct (70B)|0.08%|\n|DeepSeek-Distill (32B)|0.17%|\n|LLaMA3.1-Instruct (8B)|10.50%|\n|DeepSeek-Distill (7B)|17.20%|\n\nThe above table summarizes the average fallback rates for all the LLM models used for Reasoning Compiler. As the table shows, OpenAI models (GPT-4o mini and o1-mini) and open source models with a larger number of parameters (Llama3.3-Instruct (70B) and DeepSeek-Distill (32B)) fair similarly with 0% for the commercial models and 0.08% and 0.17% for the open-source models. On the other hand, smaller models have higher fallback rates with Llama3.1-Instruct (8B) at 10.50% and DeepSeek-Distill (7B) at 17.20%. \n\nThese results are consistent with our comparative study of different LLMs for Reasoning Compiler in Section 4.3.1 and Supplementary Material Section D.1, Table 2. \n\n---\n### **W4: Open-source models vs proprietary models**\nWe thank the reviewer for pointing more head-to-head comparisons between open-source models and proprietary ones. We agree and to that end, besides the study in Section 4.3.1 and Figure 5 (a) of the submission, we had added Supplementary Material Section D.1, Table 2 that reports the gains on the rest of the benchmarks using all the open-source LLMs. These additional results reconfirm that larger open-source models are competitive with proprietary alternatives, further supporting the accessibility and reproducibility of our Reasoning Compiler.\n \n---\n### **Minor notes and formatting**\n\nWe appreciate the feedback on figure legends and spacing. We will improve font size in Figure 1 and Figures 3–5, and adjust math formatting in Equation (1) accordingly.\n\n---\nOnce again, we thank the reviewer for their encouraging review and constructive questions, which have helped us improve both the scope and clarity of our work."}]

Author rebuttal:

Meta-review: This paper introduces a novel framework for compiler optimization that combines LLMs with MCTS. Reviewers were consistently positive about the technical soundness and sample efficiency. They raised valid concerns about the limited evaluation on a single hardware platform and the lack of end-to-end benchmarks. The authors provided a very thorough rebuttal, presenting significant new results across multiple hardware platforms and for end-to-end inference. This rebuttal effectively resolved the reviewers' primary concerns and solidified the paper's contributions. The resulting work presents a significant and practical advance in compiler optimization. The paper should be accepted.
Decision: Accept (poster)