Behavioral Embeddings of Programs: A Quasi-Dynamic Approach for Optimization Prediction
Authors: Haolin Pan
Dong Jinyuan
Hongbin Zhang
Hongyu Lin
Mingjie Xing
Yanjun Wu
Venue: ICLR 2026 Poster (2026)
Keywords: Program Representation
Compiler Optimization
Behavioral Embedding
Primary area: unsupervised, self-supervised, semi-supervised, and supervised representation learning

Abstract: Learning effective numerical representations, or embeddings, of programs is a fundamental prerequisite for applying machine learning to automate and enhance compiler optimization. Prevailing paradigms, however, present a dilemma. Static representations, derived from source code or intermediate representation (IR), are efficient and deterministic but offer limited insight into how a program will behave or evolve under complex code transformations. Conversely, dynamic representations, which rely on runtime profiling, provide profound insights into performance bottlenecks but are often impractical for large-scale tasks due to prohibitive overhead and inherent non-determinism. This paper transcends this trade-off by proposing a novel quasi-dynamic framework for program representation. The core insight is to model a program's optimization sensitivity. We introduce the Program Behavior Spectrum, a new representation generated by probing a program's IR with a diverse set of optimization sequences and quantifying the resulting changes in its static features. To effectively encode this high-dimensional, continuous spectrum, we pioneer a compositional learning approach. Product Quantization is employed to discretize the continuous reaction vectors into structured, compositional sub-words. Subsequently, a multi-task Transformer model, termed PQ-BERT, is pre-trained to learn the deep contextual grammar of these behavioral codes. Comprehensive experiments on two representative compiler optimization tasks---Best Pass Prediction and -Oz Benefit Prediction---demonstrate that our method outperforms strong static baselines. Our code is publicly available at https://github.com/Panhaolin2001/PREP/.

Published as a conference paper at ICLR 2026
BEHAVIORAL EMBEDDINGS OF PROGRAMS: A QUASI- DYNAMIC APPROACH FOR OPTIMIZATION PREDIC- TION
Haolin Pan Jinyuan Hoongbin Hoogyu
Hongyu Lin Mingjie Xing 1.23; Yanjun Wu
"University of Chinese Academy of Sciences, China
!Institute of Software Chinese Academy of Sciences, China
Hangzhou Institute for Advanced Study at UCAS, China
[panhaolin2: 1ow.rjiyyann24 (eaayyuuaannaaamlllaaeaa ucas ac. y
[hongbin2019
ABSTRACT
Learning effective numerical representations, or embeddings, of programs is a fundamental prerequisite for applying machine learning to automate and enhance compiler optimization. Prevailing paradigms, however, present dilemma. Static representations, derived from source code or intermediate representation (IR), are efficient and deterministic but offer limited insight into how a program will be- have evolve under complex code transformations. Conversely, dynamic repre- sentations, which rely on runtime profiling, provide profound insights into perfor- mance bottlenecks but are often impractical for large-scal tasks due to prohibitive overhead and inherent non-determinism. This paper transcends this trade-off by proposing novel quasi-dynamic framework for program representation. The core insight to model a program's optimization sensitivity. We introduce the Program Behavior Spectrum, a representation generated by probing program's IR with a diverse set of optimization sequences and quantifying the resulting changes in its static features. To effectively encode this high-dimensional, continuous spec- trum, we pioneer compositional learning approach. Product Quantization is em- ployed to discretize the continuous reaction vectors into structured, compositional sub-words. Subsequently, multi-task Transformer model, termed PQ-BERT, is pre-trained to learn the deep contextual grammar these behavioral codes. Com- prehensive experiments Pass Prediction and -Oz Benefit Prediction demonstrate that our method outper- forms strong baselines. Our code is publicly available at
two representative compiler optimization tasks Best
INTRODUCTION
Applying machine learning to automate and enhance compiler optimization has
emerged as promising direction to un-
Specific Tasks
LUV Code
lock the full potential of modern complex
Program Classification
hardware Ashouri et al. (2018); Pan et al.
2025a b Chen al. (2021): Ansel al. 2014;; Deng et al. (2025). The success of this paradigm hinges on a fundamen- tal prerequisite: learning effective nu- merical representation, or embedding, of a program. A powerful program embedding acts as a universal semantic interface, cap- turng the essential properties of the source
Model
Pass Prediction
Comptter
Bug Detection
Dotaset
Program
Figure 1: Example of machine learning for compiler optimization task.
"Corresponding author.
Chad://tttp/gggtttt..o/////aeaap /

---

Published as a conference paper at ICLR 2026
code in dense vector. As illustrated in Figure [] such representation can serve as a foundational component for a diverse array of high- impact downstream tasks, ranging from optimization prediction and code classification to bug de- tection and performance analysis.
However, learning representation that is both rich in semantics and practical for real-world compil- ers presents a fundamental dilemma, forcing a choice between two prevailing but flawed paradigms: (I) Static Representations: This dominant approach extracts features from various static pro- gram representations, including source code, intermediate representation (IR), abstract syntax trees (ASTs), and control or data flow graphs (CFGs/DFGs) Wei al. (2020); Hellendoorn et al. (2019); Guo et al. (020). Methods range from handcrafted feature vectors like Autophase Haj-Ali et al. (202bb) to deep learning models that operate on sequences (e.g.. IR2Vec Venkatakeerthy et al. (0200) graph structures Cummins et al. (2021a); Guo et al. (2020). The primary advantage of static methods their efficiency and determinism. However, their core limitation their myopic nature: they describe what a program is, structurally, but offer limited insight into how it will be- have or evolve under complex code transformations. (2) Dynamic Representations: An alternative approach involves profiling the program during execution to collect runtime features, such hard- ware performance counters (HPCs) et al. (2023). These representations offer profound insights performance bottlenecks. Nevertheless, they are often impractical for large- scale tasks due to their prohibitive overhead and inherent non-determinism. This trade-off between the efficiency of static analysis and the insightfulness of dynamic profiling has created significant bottleneck, limiting the capabilities of current learning-basee -ased compilers.
into program's
In this work, we transcend this dilemma by proposing novel quasi-dynamic framework for program representation. Our core insight is that an effective representation for optimization can model pro- gram's 's optimization sensitivity- its intrinsic propensity to react to different code transformations. We introduce the Program Behavior Spectrum, new representation generated by probing the program's IR with a set of diverse optimization sequences and quantifying the resulting changes in its static features. To ensure scale-invariance, these reactions are captured using logarithmic rela- tive difference. We then pioneer a compositional learning approach to encode this high- nigh-dimensional spectrum: Product Quantization (PQ) discretizes the continuous reaction vectors into structured sub-words, and a multi-task Transformer model (PQ-BERT) is pre-trained to learn their deep con- textual grammar.
Our main contributions are threefold:
We are the first to propose a quasi-dynamic program representation, the Behavioral Spec- trum. It captures program's 's optimization sensitivity by measuring changes in static fea- tures under carefully designed optimization probes.
We present a compositional encoding methodology using Product Quantization (PQ) and tailored multi-task Transformer (EQ-)ERT). This combination effectively learns the deep grammar of program behavior while addressing the ccale-vs-preciste trade-off in repre- sentation learning.
We introduce a program embedding specifically designed for compiler optimization, and demonstrate through comprehensive experiments that our method, Behavioral-Pe outperforms other baselines on two representative compiler optimization tasks.
2 METHODOLOGY
Our proposed framework learns program representations by modeling their quasi-dynamic reactions to compiler optimizations. The process consists of three main stages, as illustrated in Figure 2 (1) Behavioral Spectrum Extraction, where we quantify a program's optimization sensitivity;
(2) Structured Vocabulary Construction, where encode the continuous spectra into discrete, compositional vocabulary: and (3) Behavioral Grammar Learning, where a Transformer model learns the deep contextual relationships within these vocabulary.
2

---

Published as a conference paper at ICLR 2026
By dshawkora Edcctntttee
( Structueed Moodialeyy Contaaes Construtionn
Id Bahavkand Grunnnr Learling
100
great
UPA
dy taku oay
1AA, Tay 232-
1E3 2ud (escs 236, 422
cell mow
1011 1a
177, 136, US, 18, 14L 67, EX, 330, 11N,
99 177, 136, 155, (N00E) D4), 07, 60, 191,
With PeolPeeiie
Enpactine
Fostora
Trasstanie Croade
bltoinllpcaa Caossic
Figure 2: The overall architecture of program representation learning framework. It transforms LLVM IR into a Behavioral Spectrum, encodes it using Product Quantization, and learns its under- lying grammar via a pre-trained Transformer model.
2.1 STEP BENIAIORAL SPECTRUM EXTRACTION
The foundation our approach to represent a program not by its static structure alone, but by its Behavioral Spectrum: a high-dimensional footprint that characterizes its reactions to a diverse set of optimization transformations.
2.1.1 PROBING PROGRAM BEHAVIOR.
To elicit a program's behavior, we apply a set of carefully designed optimization sequences, termed probes, its LLVM IR. The choice of probes is critical: random single-pass selections cannot reveal the nuanced interactions present in realistic compilation.
We construct probes systematically in data-driven way. First, each program in the pre-training corpus is represented by its baseline Autophase feature vector, horig- We adopt Autophase because it is 56-dimensional feature set (e.g., counts of arithmetic instructions, basic blocks, and branches) that provides a stable summary of program structure, and its changes under optimization directly capture behavioral variation. We cluster these representations into P groups, under the assumption that programs with similar structural features exhibit similar optimization sensitivities Haj-Ali et al.) (2020b). For each cluster, we employ heuristic search Garciarena & Santana (2016) (e.g., genetic algorithm greedy strategy) to find one fixed-length sequence that maximizes the average instruc- tion count reduction across the cluster. Instruction count reduction is chosen as the optimization objective because it is reliable indicator of code size improvement.
This process yields P distinct sequences, each tuned to a category of program structures. The resulting probe set is both powerful, since each probe is empirically optimized, and diverse, since different clusters give rise to different strategies. For any program we then compute its behavioral spectrum by first extracting its baseline vector harig R5 and then applying each probe i 1,... P to obtain optimized versions Pi with corresponding vectors hopt, i. Comparing horig with hope. across all probes produces rich spectrum of behavioral transformations.
2.1.2 SCALE-INVARIANT REACTION QUANTIFICATION
significant challenge in comparing program behaviors their inherent scale sensitivity. A trans- formation that removes 100 instructions may be monumental for small kernel but trivial for million-line application. To address this, we quantify the reaction not as an absolute difference, but a logarithmic relative difference, ensuring scale-invariant representation. For each feature dimension j E (1,...,56 56), the reaction di,j for probe is calculated as:
dej log(1 max(0, kn)n log(1 max(0, horig.
3

---

Published as a conference paper at ICLR 2026
where max(0,.) ensures that the imput the logarithm is non-negative, robustly handling poten- tial minor negative values arising from feature extraction artifacts. The log(1 + z) transformation (or loglp) gracefully handles zero-valued features and compresses the effect of large absolute changes, focusing on multiplicative, of shifts. The complete Behavioral Spectrum for program p is thus matrix Sp R RPx50 where each row is scale-invariant reaction vector. This corpus of spectra forms the basis for all subsequent learning.
2.2
STEP 2: STRUCTURED VOCABULARY CONSTRUCTION VIA PQ
2.2.1
MOTIVATION FOR DISCRETIZING BEHAVIORAL SPECTRA
The Behavioral Spectrum, Sp. provides rich, continuous representation of program's optimiza- tion sensitivities. However, to learn the complex grammar and long-range dependencies within this P-step sequence, need leverage powerful sequence models like Transformers. These models traditionally operate on discrete tokens, necessitating a bridge from our continuous vector space to discrete vocabulary.
The motivation for this discretization stems from a core hypothesis in compiler optimization: gen- eralization. A given optimization sequence often elicits a similar type of behavioral change across a range of different programs. For example, a loop-unrolling pass might consistently produce a re- action characterized by an increase arithmetic instructions and decrease in control-flow instruc- tions, regardless of the specific program's 's details. It is therefore desirable to group these similar continuous reaction vectors into finite set of discrete behavioral archetypes or words. Clustering is natural approach for discovering such archetypes.
However, a naive hard clustering method, which assigns each vector to single, indivisible cluster ID, may suffer from information loss. A vector lying on the boundary between two clusters is forced into a single choice, losing the valuable information that it shares characteristics with both archetypes. Furthermore, this approach struggles to capture the fine-grained internal structure of the 56-dimensional reaction vectors. To overcome these limitations, we employ Product Quanti- zation Jegou al. (2010), structured vector quantization technique that performs clustering in more granular, compositional manner.
2.2.2 PQ FOR STRUCTURED BEHAVIORAL ENCODING
The core insight of PQ is to abandon the search for monolithic, high-dimensional prototypes and instead learn a set of low-dimensional, reusable building blocks or primitives. It posits that any complex reaction vector can be approximately reconstructed by composing these simpler primitives. This is analogous to representing a complex color not single entry in giant color palette, but as a combination fundamental R, G, B values.
To implement this, PQ decomposes each D 56 dimensional reaction vector d into M disjoint sub-vectors. our work, we choose M = 8, resulting in 8 sub-vectors {dl,.... dg each of dimension Dunh D/M = 7. A separate, small-scale K-Means quantizer, q. is then trained for each of the M subspaces Ahmed et al. (2020); Ravuri & Amarasinghe Each quantizer qi has its own codebook (or nub-vocabulary)
(c.,,..., containing k 256 low-dimensional
centroids (where k* corresponds to Thbits = 8).
An arbitrary reaction vector d then encoded by quantizing each of its sub-vectors d, indepen- dently:
c = (d.) = arg min ||d, - cr,,
(2)
The final representation is compositional code: a tuple of M = 8 integer IDs, c= (C1, ......, c8), where 6 E (O...., 255]. The original vector d can be approximately reconstructed by concatenat- ing the corresponding centroids: d = [c1,e]> C2.c,...., C.
This compositional approach allows us to represent vast number of distinct vectors (a virtual vocabulary of size 256a) with a compact set of learned centroids (8 x 256), thereby retaining fine- grained structural information with minimal loss.
4

---

Published as a conference paper at ICLR 2026
2.3 STEP 3: LEARNING THE BEHAVIORAL GRAMMAR WITH PQ-BERT
The PQ step transforms each program's 's continuous Behavior Spectrum into a structured, discrete article of size P x M. This encoding preserves fine-grained information but does not yet capture the rich contextual dependencies between the P different reactions. For instance, program's 's strong reaction to a loop-unrolling probe is often correlated with its reaction to a vectorization probe. To effectively learn this deep, underlying grammar of compiler optimization behavior, we require sufficiently powerful and expressive sequence model.
We design Transformer-basee model, which call PQ-BERT, tailored to our compositional codes. Standard language models like BERT are trained to predict a single token from its context. However, our PQ representation is multi-faceted. where each reaction is described by M sub-word IDs. A naive approach might concatenate these sub-words into a longer sequence, but this may disrupt the inherent synchrony of the M subspaces. Therefore, method is to treat the prediction of the M sub-word IDs as multi-task learning problem. where the model is forced to learn the intricate correlations between subspaces simultaneously. We pre-train PQ-BERT using a multi-task Masked Language Model (MLM) objective.
Model Architecture. The PQ-BERT architecture is designed to process the P x M matrix of com- positional codes. The input to the model is a sequence of reaction codes, C {C1, .,,..., Cp). where each C2 = (ct.1 (C..1....,.C C.MM) is tuple of M sub-word IDs. The model first projects these discrete codes into a continuous vector space using M independent sobb-mmeddingg layers. For each code C₁2 the i-th sub-embedding layer, E,, maps the sub-word ID Cr. to now-dimensional vector i = ).. These M sub-embeddings are then concatenated to form a single high-dimensional embedding x, [e,, 64,2; -.. ;..M for the 1-th reaction, where its dimension is D 256. This sequence of P fused embeddings, X = {xI..... .pp). is then augmented with positional encodings and processed by a standard multi-layer Transformer Encoder Vaswani et al. (2017).
The encoder uses self- atf-attention to produce a sequence of contextually
output representations
H (hi,.... hr]. Finally, to facilitate the multi-task objective, the model employs M indepen- dent linear output heads. The i-th head, O., takes the entire output sequence H and produces distribution logits over the hs* possible sub-words for the i-th subspace.
Pre-training Task and Objective Function. We pre-train PQ-BERT using multi-task MLM objective. Let C be the set of all P M code sequences in corpus. During training, for each sequence C, we randomly select a set of indices Imask to be masked, which constitutes approximately 15% of the P M total sub-word positions. The masked input sequence is denoted Cmasked- The model's objective is to predict the original sub-word IDs, Corig. at these masked positions.
The total loss C is defined the sum of the average cross-entropy losses from all A output heads, calculated only over the masked positions. For a single sequence, the loss is:
((C)
log P(cr, Cmusked)
Immskk
(t.i)
where Imaak,i are the masked positions corresponding to the i-th subspace, and the probability (CusseCCmseed)
P(cr,s (muuseed is computed from the logits produced by the i-th output head O after softmax operation. This multi-task setup forces the model to learn the intricate correlations between differ- ent subspaces of the reaction vectors. For instance, it might learn that certain type of reaction in the first 7 dimensions (e.g., related to scalar instructions) often co-occurs with specific reaction in the last 7 dimensions (e.g., related to memory behavior). This process yields powerful encoder capable of understanding the deep, compositional grammar of program optimization behavior. The pre-trained Transformer Encoder part of the model then used to generate embeddings for down- stream tasks.
EXPERIMENTAL SETUP
Pre-training Dataset. For self-supervised pre- treiniinnngg we use a dataset of over 220,000 LLVM IR files constructed for the task of classifying programs by functionalities Mou al. (016).. Fach file
5

---

Published as a conference paper at ICLR 2026
Table 1: Composition of downstream datasets from CompilerGym.
Uncurated Datasets
Curated Datasets
Type
Dataset
Train
Val
Test
Type
Dataset
Train
Val
Test
blas-v0
133
28
29
github-v0
7,000
1,000
obench-v1
11
linux-vO
4,906
1,000
mibench-vl
0
40
Uncurated
Curated
openev-vO
149
32
32
chstone-v0
0
12
poj104-v1
7,000
1,000
npb-v0
121
tensorflowvvO
415
89
90
Total
19,603
3,149
335
corresponds competitive programming solution, providing rich diversity in algorithmic struc- tures and computational patterns. This allows the model to acquire gencral-purpose program seman- ties that independent of specific optimization tasks. We train for 30 epochs with learning rate of 10-4 and batch size of 32.
Downstream Task Datasets. To evaluate transferabilityy adopt benchmarks from Compiler- Gym Cummins et al. (2021b). Importantly, these benchmarks are entirely different from our pre- training corpus, ensuring strict out-of-domain evaluation Fursin (2009); Guthaus al. (2001) > Bailey et al. ); Hara et al. (2008); Culjak et al. y; Abadi et al. (2016). Following the official split, uncurated benchmarks (e.g.. linux, github) are used for training, while curated benchmarks (e.g., cbench, mibench) are reserved for testing. Table summarizes the dataset composition.
Downstream Tasks. We consider two widely studied tasks in compiler auto-tuning: (1) predict- ing which optimization pass among 124 candidates yields the largest instruction reduction, and (2) predicting the instruction reduction ratio under the -Oz optimization pipeline. These tasks are rep- resentative of classification and regression challenges in compiler optimization and capture both fine-grained and holistic optimization effects. Performance reported using Top-1/Top-5 accuracy and Mean Absolute Error (MAE), respectively.
Baselines. We compare against both feature-based and embedding-basee based methods. Feature-based baselines include Autophase Alaj-Aliee instruction opcode counts), which can be used directly without further training. Embedding-basse baselines include Ve2 Venkata et al. (2020) and inst2vec Ben-Nun et al. (2018), where we use their released pre-trained embeddings. Since inst2vec only provides instruction-level em- beddings rather than whole-progran representations, we apply an LSTM encoder to aggregate them for downstream tasks. In addition, we include the GNN-based ProGraMI. (Crmmmns et al. (2021a), which is pre-trained on the same algorithm classification dataset ours, ensuring a fair comparison.
Haj--AaaeHaajAli-eee et al. (2020b) (56 handcrafted features) and InstCount (LLVM
Implementation Details. For a fair comparison, all methods are trained on NVIDIA A100 GPUs under the same optimization settings (Adam with learning rate 1 x 10 batch size 128, 100 epochs). For Best Pass Prediction, the models (except inst2vec) use two-layer MLP (512 256 1 124), while the -02 regression task uses a deeper MLP gradually shrinking from 256 to dimen- sion. For 2 we directly adopt an LSTM-based model for both downstream tasks, instead of the MLP used by other methods.
4 EXPERIMENTS
In this section, we present the empirical evaluation of our proposed behavioral embedding, Behavioral-PO. Our experiments are designed to answer three key research questions:
RQ1: Does our proposed behavioral representation outperform state-of-the-a static representa- tions compiler optimization tasks?

---

Published as a conference paper at ICLR 2026
RQ2: What is the contribution of each key component in framework, such iivee-invrriant quantification and compositional encoding?
RQ3:
Does the learned embedding space of our behavioral representation exhibit a meaningful geometric structure that aligns with semanties of compiler optimization tasks?
Additionally, to evaluate the practical utility and cross-domain generalization of our method, provide extended analyses on execution runtime, cycle-level performance, and device mapping in Appendix C
On reporting validation vs. test performance. We report both validation and test results. The validation set is not used for model selection or hyperparameter tuning: it reflects the original split and contains programs mostly from the same benchmark families as the training set, leading to generally higher performance. In contrast, the test set is composed mainly of programs from suites outside the training data, providing a stricter out-of-domain evaluation. Thus, validation measures in-distribution generalization, while test primarily assesses cross-domain robustness.
(Accuracy
00
Ma 1i
20
112
Performance comparison across two prediction tasks: Best Pass Prediction, evaluated in terms of accuracy (higher values indicate better performance). and -Oz Benefit Prediction, evaluated in terms of Mean Absolute Error (lower values indicate better performance).
Figure 3:
4.1 MAIN RESULTS: SUPERIOR PERFORMANCE ON DOWNSTREAM TASKS (RQ1)
To evaluate the practical utility and versatility of the learned program representations, we select two downstream tasks that central to the field of compiler optimization. These tasks were chosen to assess two distinct but equally critical capabilities: fine-grained, single-step decision making (classification) and holistic, mangee performance estimation (regression).
4.1.1 TASK 1: BEST PASS PREDICTION
This is 124-class classification problem. evaluate performance using Top-1 and Top-5 accuracy. Top-1 accuracy measures the percentage of cases where the model's single highest-probability prediction the correct best pass. Top-5 accuracy measures the percentage of cases where the correct best pass is included within the model's 's top five predictions, metric that reflects the practical utility the model in narrowing down the search space for autotuners.
Figure presents the results on both the validation and test sets. Our method, Behavioral-Pe -PQ. achieves a Top-1 accuracy of 64.48% % and Top-5 accuracy of 89.55 on the held-out test set. This represents substantial improvement over all baseline methods. Notably, it surpasses the strongest static embedding baseline, inst2vec (39.27% Top- 1), by a large margin of over 25 absolute percent- age points in Top-1 accuracy. This result strongly suggests that the behavioral spectrum captures critical information about optimization sensitivity that is not readily available in purely static repre- sentations.

---

Published as a conference paper at ICLR 2026
4.1.2 TASK 2: OO BENEFIT PREDICTION
We further evaluate the representations on the task of predicting the benefit of the -Oz optimization pipeline. This regression task assesses representation's 's ability to model the cumulative effect of a long sequence of interacting transformations. The results, measured by Mean Absolute Error (MAE), are presented in Figure
Our Behavioral-FO method achieves a Mean Absolute Error of 8.19% on the test set, with corresponding validation MAE of 2.22%. This is the lowest error among all tested methods. For comparison, the next best baseline, inst2vec, yields an MAE of 16. 6,23% while other static repre- sentations such as Ve2 and Autophase result in MAEs of 25.40% and 25.92% respectively. The performance difference between methods more pronounced on this task than on the single-pass prediction task. The results suggest that while static representations can be effective for modeling immediate, single-pass effects, they are less suited for predicting the outcomes of complex opti- mization sequences. Our Behavioral Spectrum, by directly encoding the program's 's reactions to such transformations, appears to provide a more effective signal for this type of long-range predic- tive task.
Table 2: Ablation study results on both downstream tasks. For Best Pass, Top-5 Acc. (%) is reported. For -02 Pred., MAE (%) is reported. Best test set results for each task in bold.
Best Pass (Top-5 %)
-Oz Pred. (MAE %)
Model Variant
Validation
Test
Validation
Test
Ours (KMeans)
98.48
93.43
3.19
8.24
Ours (No-Relative)
99.05
94.33
2.81
10.96
Ours (No-Transformez)
98.73
87.46
2.31
10.08
Ours (Rehavioral-PQ., Full)
99.08
89.55
2.22
8.19
4.2 ABLATION STUDIES (RQ2)
To answer RQ2, we conduct series of ablation studies to dissect our framework and validate the contribution of each key design component. We compare our full Rehavi aaa model against three variants: (1) Ours (KMeans), which discards Product Quantization and instead directly clusters the full behavioral vectors using standard K-Means; (2) Curs Moo-Relativen which uses absolute feature differences instead of the scale-invariant logarithmic ratio; and (3) Ours (No-Tran) ans former), which removes the Transformer encoder and directly pools the encoded sub-embeddings.
The results are presented in Table 2 For the Best Pass Prediction task, our full Rehavioral-Pg model achieves the highest Top-5 accuracy
the validation set (99.08%), while the No-Relative variant achieves the highest Top-5 accuracy on the test set (94.33%). This sug- gests that for this classification task, coarser-grained signals captured by absolute differences or monolithic clusters can provide reasonably strong predictive power, although our full model remains competitive on the test set (89.55%).
For the more complex -Oz Benefit Prediction task. the superiority of our full model is clear. On both the validation and test sets, Behaviorall aalPOQ achieves the lowest MAE (2.22% and 8.19%, respectively). significantly outperforming all ablated versions. Notably, the No-Rel l ve and No-Trans former variants reach higher MAE values on the test set (10.96% and 10.08%, respec- tively), confirming that methodscale-invarics regression tasksmethod, and that the methoddeep contextual reasoning provided by the Transformer is essential for modeling long-range optimization effectsmethod.
quantification is critical for generalization to complex
4.3 EMBEDDING SPACE ANALYSIS (RQ3)
To answer RQ3, we investigate whether our learned embedding space exhibits a meaningful geometric structure that aligns with the semantics of compiler optimization. A well-structured space should similar optimization needs into coherent clusters. We evaluate this property both quantitatively, using K- Nearest Neighbors (K-NN) classifier, and qualitatively, through t-SNE visualization.
group programs with
8

---

Published as a conference paper at ICLR 2026
Table 3: Top-1 accuracy (%) of K-Nearest Neighbors classifier (k=5) the Best Pass Prediction task, evaluated on the test set. This metric reflects the semantic structure of each embedding space. Embedding Method
Autophase
InstCount
IR2Vec
inst2vec
ProGraMI.
Ours
K-NN Top-1 Acc. (%)
46.57
75.82
74.33
72.15
70.75
79.70
The results provide converging evidence of our method's superiority. Quantitatively, we use a K-NN classifier (k = 5), whose performance directly reflects the local semantic coherence of space. As shown in T our Behavio oral -PQ embedding achieves a Top-1 accuracy of 79.70%, significantly outperforming all baselmns, including the next best method, InstCount (75.82%). Qualitatively, this strong result is visually corroborated by the t-SNE projection of the embedding spaces, presented in Figure 4 The visualization shows that our Behavioral-PO space is the only one to exhibit distinct, well-separated clusters of programs that share the same optimal pass. For instance, programs for which -inst combine is optimal are naturally grouped to- gether. Together, these quantitative and qualitative results suggest that our behavioral approach generally learns representation space that is reasonably well-structured and semantically aligned with the task of compiler optimization.
Behavioral-PQ
ProGraML
InstCount
20
10
Dimenston 2
Dimansion Inst2vec
Dimension IRZVec
Dimansion Autophase
le H
Bimensoon
-10
-20
-35
Dimension1
Dimenston
-30
Dimension
Figure 4: t-SNE visualization of embedding spaces for the test set. Each point is program, colored by its true best optimization pass.
5 RELATED WORK
Program representations for machine learning in compiler optimization have evolved from handcrafted features to learned embeddings Zhu et al. (2024); Cummins et al.] (2023); Gong et al. (2025); Liu et al. 22021) Park ct al. (2022). Early approaches relied on manually designed level metrics such as Autophasc Haj-Ali et al. (2020b) and InstCount Lattner & Adve (2004): [Cmmmins et al. (202Tb). which count instructions, branches, and memory operations. These features are simple and interpretable but lack the expressiveness needed capture complex behaviors, limiting generalization across diverse code bases.
Learned representations instead encode semantics automatically. inst2vec Ben-Nun et al, (0018) applics skip-gran models on LLVM IR, capturing local context but ignoring control/data flow. IR2Vec Venkata et al. 2020) extends this via graph-based flow analysis, while ProGraMI. (Cummins al. (202 unifies control, data, and call graphs in multigraph for message passing, enabling richer analyses. These approaches better capture non-local dependencies and are less sensitive to superficial variations. Dynamic profiling has also proven useful [u et al. (2023): Duesterwald et al. (2003), recording runtime behaviors such as memory accesses and input/output values. Such signals complement static embeddings. motivating quasi-dynamic approaches that integrate both perspectives for more effective optimization.
5.1 LIMITATIONS AND FUTURE WORK
Limitations. Our approach has four main limitations. First, the diversity of optimization probes may be insufficient for some program classes, although the selected probes maintain reasonably good optimization performance. Second, inference requires computing / + 1 Autophase features per program, introducing some preprocessing overhead (about 0.2s per program). although this remains faster and more stable compared with collecting full dynamic features. Third, our evaluation currently focused compiler optimization tasks,

---

Published as a conference paper at ICLR 2026
with limited validation on other downstream tasks such program classification. Fourth, while behavioral vocabularies provide interpretability, their semantic meaning and coverage are still limited, which may affect the generalizability of the representations.
Future directions include developing adaptive probe selection strategies to better suit dif- ferent program classes, reducing preprocessing costs through more efficient Autophase computation, and ex- ploring limited integration of dynamic information to improve prediction accuracy. Additionally, we plan to validate the approach broader range of downstream tasks beyond compiler optimizations, and to enhance the interpretability and coverage of behavioral vocabularies more explainable program representations.
Future Work.
6 CONCLUSION
This work introduces quasi-dynamic paradigm for program representation, termed as Behavioral Embed- dings, which characterize programs by modeling their responses to a series carefully designed optimization probes. By capturing these optimizatioon-senitiiee behaviors, the approach encodes deep semantics that are dif- ficult to extract from static structure alone. Compared with purely static embeddings, it effectively balances the efficiency of static amalysis with the richer, task-relevant insights typically provided by dynamic profiling, Em- pirical results demonstrate clear advantages: our model substantially improves accuracy on best pass prediction and reduces error in optimization benefit prediction.
ETHICS STATEMENT
The research presented in this paper adheres to ethical principles for academic work. All program corpora used for pre-training and evaluation, such as the OJ dataset and CompilerGym benchmarks, are derived from publicly available, open-source codebases intended for research purposes. We acknowledge that the large-scale pre-training of our Transformer models requires significant computational resourees, which has associated environmental impact. We argue that this cost is justified by the foundational nature of this research, which aims to establish a new, more efficient paradigm for compiler optimization that could, in the long term, reduce overall computational waste. While our work focuses on benign compiler optimization, we recognize that ad- vanced program representation techniques could potentially be misused for analyzing or optimizing malicious software. However, our proposed method does not introduce any capabilities uniquely suited for such applica- tions beyond those already present in existing program analysis tools. We advocate for the responsible use of these technologies within the academic and industrial communities.
REPRODUCIBILITY
To support the reproducibility Behavioral-PQ., we make the complete source code and experimen- tal configuration publicly accessible. All models, training datasets, and scripts can be found at: https//anonymous.44 hwtp:ymwwwuuuuuoo.osooooo wwwpeeeccnenee//PP-1111111
The repository provides step-by-step instructions for setting up the environment, running the experiments, and reproducing the results standard benchmarks. By provid- ing these resources, we aim to enable independent verification and replication of our findings, fostering further progress in compiler optimization research.
REFERENCES
Martín Abadi, Paul Barham, Jianmin Chen, Zhifeng Chen, Andy Davis, Jeffrey Dean, Matthieu Devin, Sanjay Ghemawat, Geoffrey Irving, Michael Isard, et al. (TensorFlow): a system for (Large-Scale) machine learning. In 12th USENIX symposium operating systems design and implementation (OSDI 16), pp. 265-283, 2016.
Mohiuddin Ahmed, Raihan Seraj, and Syed Mohammed Shamsul Islam. The k-means algorithm: A compre- hensive survey and performance evaluation. Electronics, 9295 2020.
Jason Ansel, Shoaib Kamil, Kalyan Veeramachaneni, Jonathan Ragan-Kelley, Jeffrey Bosboom, Una-May O'Reilly, and Saman Amarasinghe. Opentuner: An extensible framework for program autotuning. In Pro- ceedings of the 23rd international conference on Parullel architectures and compilation, pp. 303-316, 2014.
Amir H. Ashouri, William Killian, John Cavazos, Gianluca Palermo, and Cristina Silvano. A survey on compiler autotuning using machine learning, ACM Computing Surveys (CSUR), 51(5):1-42, 2018
David H Bailey. Eric Barszez, John T Barton, David S Browning, Robert L Carter, Leonardo Dagum, Rod A Fatoohi, Paul O Fredericksonn, Thomas A Lasinski, Rob S Schreiber, et al. The nas parallel benchmarks. The International Journal of Supercomputing Applications, 5())633-3, 1991.
10

---

Published as a conference paper at ICLR 2026
Tal Ben-Nun, Alice Shoshana Jakobovits, and Torsten Hoefler. Neural code comprehension: learnable representation of code semantics. Advances in neural information processing systems, 31, 2018.
Junjie Chen, Ningxin Xu, Peiqi Chen, and Hongyu 7hang. Efficient Ffficientcoomplerrr autotuning via bayesian optimiza- tion. 202/ IEEF/ACM 43rd International Conference on Software Engineering (ICSE), pp. 1198-1209.
IEEE, 2021.
Ivan Culjak, David Abram, Tomislav Pribanic, Hrvoje Dzapo, and Mario Cifiek. A brief introduction openev. In 2012 proceedings of the 35th international convention MIPRO. pp. 1725-1730 JEEE, 2012.
Chris Cummins, Pavlos Petoumenos, Zheng Wang, and Hugh Leather. End-to-end deep learning of optimiza- tion heuristics. 2017 26th International Conference Parallel Architectures and Compilation Techniques (PACT), pp. 219-232. IEEE, 2017.
Chris Cummins. Zacharias V Fisches. Tal Ben-Nun, Torsten Hoefler, Michael FP O' O'Boyle, and Hugh Leather. Programl: A graph-based program representation data flow analysis and compiler optimizations. In International Conference on Machine Learning, pp. 2244-2253. PMLR, 2021a.
Chris Cummins, Bram Wasti, Jiadong Guo, Brandon Cui, Jason Ansel, Sahir Gomez, Somya Jain, Jia Liu, Olivier Teytaud, Benoit Steiner, Yuandong Tian, and Hugh Leather. Compilergym: Robust, performant compiler optimization environments for ai research, 2021b. URL 08267
Chris Cummins, Volker Seeker, Dejan Grubisic, Mostafa Elhoushi, Youwei Liang, Baptiste Roziere, Jonas Gehring. Fabian Gloeckle, Kim Hazelwood, Gabriel Synnaeve, et al. Large language models for compiler optimization. arXiv preprint vrXiv:2309.07062. 2023.
Chaoyi Deng, Jialong Wu, Ningya Feng, Jianmin Wang, and Mingsheng Long. Compilerdream: Learning a compiler world model for general code optimization, 2025. URL
16077
Evelyn Duesterwald, Calin Cascaval, and Sandhya Dwarkadas. Characterizing and predicting program behav- ior and its variability. In 2003 12th International Conference Parallel Architectures and Compilation Techniques, pp. 220-231011221 IEEE, 2003.
Grigori Fursin. Collective tuning initiative: automating and accelerating development and optimization of computing systems. In GCC Developers' Summit, 2009.
Unai Garciarena and Roberto Santana. Evolutionary optimization of compiler flag selection by learning and ex- ploiting flags interactions. In Proceedings the 2016 on Genetic and Evolutionary Computation Conference Companion, pp. 1159-1166, 2016.
Jingzhi Gong, Vardan Voskanyan, Paul Brookes, Fan Wu, Wei Jic, Jie Xu, Rafail Giavrimis, Mike Basios, Leslic Kanthan, and Zheng Wang. Language models for code optimization: Survey, challenges and future directions, 2025.
httttps://axxiv.oort
Daya Guo, Shuo Ren, Shuai Lu, Zhangyin Feng, Duyu Tang, Shujie Liu. Long Zhou, Nan Duan, Alexey Svyatkovskiy, Shengyu Fu, et al. Grapheodebert: Pre-training code representations with data flow. arXiv preprint wr.iv:2009.0co.mm 08366, 2020.
Matthew Guthaus, Jeffrey S Ringenberg, Dan Ernst, Todd Austin, Trevor Mudge, and Richard B Brown. Mihench: free, commercially representative emhedded benchmark suite. In Proceedings of the fourth annual IEFE international workshop on workload characterization. WWC-4 (Cat. No. 0IEX538), pp- 3-14. IEEF, 2001.
Ameer Haj-Ali, Nesreen K. Ahmed, Ted Willke, Yakun Sophia Shao, Krste Asanovic, and lon Stoica. Neurovec- torizer: End-to-end vectorization with deep reinforcement learning. In Proceedings of the 18th ACM/IEEE International Symposiun on Code Generation and Optimization. pp.
2020a.
Ameer Haj-Ali, Qijing Jenny Huang, John Xiang, William Moses, Krste Asanovic, John Wawrzynek, and Ion Stoica. Autophase: Juggling his phase orderings in random forests with deep reinforcement learning. Proceedingy of Machine Learning and Systems, 2:70-81, 2020b.
Yuko Hara. Hiroyuki Tomiyama, Shinya Honda, Hiroaki Takada, and Katsuya Ishii Chstone: A benchmark program suite for practical c-based high-level synthesis. In 2008 IEEE International Symposium on Circuits and Systems (ISCAS), pp. 1192-1195 IEEE, 2008
Vincent J Hellendoom, Charles Sutton, Rishabh Singh, Petros Maniatis, and David Bieber. Global relational models of source code. In International conference on learning representations, 2019.
11

---

Published as a conference paper at ICLR 2026
Herve Jegou, Matthijs Douze, and Cordelia Schmid. Product quantization for nearest neighbor search. IEEE transactions on pattern analysis and machine intelligence, 33 2010.
Chris Lattner and Vikram Adve. Livm: A compilation framework for lifelong program analysis & transforma- tion. In International symposiun on code generation and optimization, 2004. CGO 2004., pp. 75-86, IEEE, 2004.
Hongzhi Liu, Jie Luo, Ying Li, and Zhonghai Wu. Iterative compilation optimization based on metric learning and collaborative filtering. ACM Transactions on Architecture and Code Optimization (TACO), 19(1):1 2021.
Lili Mou, Ge Li, Lu Zhang, Tao Wang, and Zhi Jin. Convolutional neural networks over tree structures for pro- gramming language processing. In Proceedings of the AAAI conference on artificial intelligence, volume 30, 2016.
Haolin Pan, Hongyu Lin, Haoran Luo, Yang Liu, Kaichun Yao, Libo Zhang, Mingjie Xing, and Yanjun Wu. Compiler-rl: Towards agentic compiler auto-tuning with reinforcement lcarning, 2025a. URL https:
//arxiv.crg abs/2506. 15701
Haolin Pan, Yuanyu Wei, Mingjie Xing. Yanjun Wu, and Chen Zhao. Towards efficient compiler auto-tuning Leveraging synergistic search spaces. In Proceedings of the 23rd ACM/IEEF International Symposium on Code Generation and Optimization, pp. 614-627, 2025b.
Sunghyun Park, Salar Latifi, Yongjun Park, Armand Behroozi, Byungsoo Jeon, and Scott Mahlke, Srtuner: Effective compiler optimization customization by exposing synergistic relations. 2022 IEEE/ACM Inter- national Symposium on Code Generation and Optimization (CGO), pp. 118-130. IEEE, 2022.
Indraneil Paul, Goran Glavaš, and Tryna Gurevych. Ircoder: Intermediate representations make language mod- els robust multilingual code generators. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 15023-15041, 2024. 2024.
Chaitamya Ravuri and Saman Amarasinghe. Eliminating haltiaaaaaiaiiiiiiinonnnccdd errors in Ilm code generation with functional clustering, 2025. URL
11021
Ashish Vaswani, Noam Shazcer, Niki Parmar, Jakob Uszkorcit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. Advances in neural information processing systems, 30, 2017.
S VenkataKeerthy, Rohit Aggarwal, Shalini Jain, Maunendra Sankar Desarkar, Ramakrishna Upadrasta, and YN Srikant. Tr2vec: Llvm based scalable program embeddings. ACM Transactions Architecture and Code Optimization (TACO), 17(4):1-27. 2020.
Jiayi Wei, Maruth Goyal, Greg Durrett, and Isil Dillig. Lambdanet: Probabilistic type inference using graph neural networks. arXiv preprint arXiv:2005.0216/ 2020.
Xiangzhe Xu, Zhou Xuan, Shiwei Feng, Siyuan Cheng. Yapeng Ye, Qingkai Shi, Guanhong Tao, Le Yu, Zhuo Zhang, Xiangyu Zhang. Pem: Representing binary program semantics similarity analysis via a probabilistic execution model. In Proceedings of the 31st ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering, pp. 401-412, 2023.
Mingxuan Zhu, Dan Hao, and Junjie Chen. Compiler autotuning through multiple-phar learning. ACM Transactions on Software Engineering and Methodology, 33(4):1-38. 2024.
12

---

Published as a conference paper at ICLR 2026
APPENDIX
A DETAILED CASE STUDY ON BLAS-VO.1277
To provide more concrete, miero-level example of model's behavior, we conduct deep-dive analysis into a successful prediction case: the program blas- -aaaas-v.1...... .11 from the test set.
Scenario Overview. For this program, the empirically determined single best optimization pass -inst Our Behavioral based model successfully includes this pass in its Top-5 predic- tions, with -inst combine being the highest-ranked (Top-1) recommendation. It is worth noting that baseline model relying solely static Autophase features failed include -instcembine its Top-5 predictions for this case. The respective predictions are shown Table
Table 4: Model predictions for the case study program blas
Item
Details
Ground Truth Best Pass Our Model's Top-5
inst combine (ID: 53)
-instcombine, -early-cse--mmmss, -ipscop, etc.
Analyzing the Relationship between Predictions and Probe Reactions. To better understand the information available model. investigate the relationship between the program's 's Behavioral Spectrum the model's correct prediction. reaction strength is defined as the magnitude of reduction in key Autophase feature (feature #51).
identify which of the probes elicits the strongest reaction. The
Our analysis shows that Probe #10, a complex 50-pass sequence, clicited the single strongest reaction. We then cross-reference model's Top-5 predicted passes against the composition of this most impactful probe sequence.
The results of this analysis are detailed Table found sequence contain a total 12 instances passes that were also present in our model's Top-5 prediction list. Most notably, the ground truth best pass, - isst combine, appears times within this single most reactive probe.
passes within strongest probe
Table 5: Analysis of the alignment between the model's Top-5 predictions and the composition of the most reactive probe sequence (Probe #10) for blas -v0122. 11.
Analysis Item
Finding
Most Reactive Probe Index
Probe #10
Model's Top-5 Predictions
- -instcomiine
(Top-1), ipsccp.
eearly-ycss manemy-, memasa, -globalopt, mergefunc
Occurrences of Top-5 Passes within Probe #10
-instcombine
3 times
-early-cse-messsa
4 times
-ipsccp
times
-globalopt -mergefunc
3 times
times
Total Alignment
12 out of 50 passes in the strongest probe are from the model's s Top-5 list.
Discussion. This analysis reveals strong correlation: the program reacts most intensely to probe quence that richly populated with passes the model identified highly effective. While this correlation does not establish direct causal link as the probe's 's overall effect result of the entire 50-pass sequence, not just the individual passes- it does provide valuable insight. It suggests that the Behavioral Spectrum contains dis- cemiille signals related to the efficacy of certain optimization types. The pre- pre-trained PQ-BERT model appears to be capable of identifying these signals within the complex, high-dimensional spectrum and associating them with correct individual pass recommendations.
13

---

Published as a conference paper at ICLR 2026
Table 6: Performance comparison with general-purpose Large I. Language Models (LLMs) in zero- shot setting. For the Best Pass Prediction task, we report Top-1 and Top-S accuracy (%). For the -Oz Benefit Prediction task, we report Mean Absolute Error (MAE, %). The performance of our specialized model and the best traditional baseline are included for reference.
Model / Method
Task 1(Tpp11 %
Task 1(Top-5
Task 2
Gemeral-Purport Large Language Models (Zero-Shot)
gpt-5-mini
35.52
56.12
23.87
2ai-org/GLMM4555
34.93
59.10
22.81
DeepSeck- V
34.03
54.93
24.05
BaiduERNIEE44--300--4411
2.39
23.28
23.70
tencen//Hnnyuan-A13BB Instruct
1.30
2.60
23,52
Tonggi-Zhivven/QvennLonnLLLL
23.54
48.25
22.74
Specialized Models (for reference)
Autophase (Best Baseline)
38.51
56.72
25.92
Ours (Behavioral-PQQ)
64.48
89.55
8.19
B COMPARISON WITH GENERAL-PURPOSS LARGE LANGUAGE MODELS
Recent advancements Large Language Models (LLMs) have demonstrated remarkable capabilities in code understanding and generation. To situate our work within this modern context, we conducted an experiment to evaluate the zero-shot performance of several state-of-the-art general-purpose LLMs on our two downstream tasks. We provided the models with the full LLVM IR, its Autophase features, and the list of candidate passes, then prompted them return their predictions in structured JSON format.
The results, summarized in Table reveal clear trend. While some the best-perforning L.MMs (e-g.- gpt-5-m ni, GI.M44 DeepSeek-V3] achieve respectable Top-5 accuracy of around 55-60% on the Best Pass Prediction task, their performance is still substantially lower than specialized Behavi ora 1-11PP model (89.55%) Notably, handcrafted Autophase baseline (3).51%) On more complex -Oz Benefit Prediction task, the performance gap is even more stark. The best- pess-eeforming LLM (GLM-4. 4.5) yields an MAE of 22.81 %, an error rate nearly three times higher than that of our method 19%%.
the best LLM's Top-1 accuracy (35.52%) does not surpass that of the simple,
These findings provide crucial insight: while general-purpose LLMs possess a broad understanding of code, this knowledge does not readily translate the highly specialized, quantitative, and nuanced domain of com- piler optimization. Their zero-shot reasoning struggles match the performance of a smaller, domain-specific model that has been pre-trained on representation-ouv Behavioral Spectrum-that is intrinsically aligned with the task of predicting optimization outcomes.
C EXTENDED GENERALIZATION AND EMPIRICAL ANALYSIS
To further validate the robustness, practical utility, and across-task generality of our proposed Behavioral-PQ embeddings, we present additional experimental results. These evaluations extend beyond the code-size metrics used in the main text to include execution runtime, cycle-level performance, and cross cross-domain generalization tasks.
C.I EVALUATION ON SUNTIME-RELATE METRICS
While instruction-count reduction serves stable proxy for optimization quality, execution runtime is the ultimate performance metric in production environments. We conducted two experiments evaluate how well our embeddings capture runtime-related characteristics.
Task A:
Predicting -03 Runtime on Neuro Vectorizer Haj-Ali et al. (200aa). task to predict the absolute execution time of programs after -03 optimization. As shown in Table Behavioral- PQ achieves the lowest MAE and the highest Rº among all tested static representations.
We formulated a regression
Task B: Predicting Sycle-Reduction Percentage via LLVM-MCA. We evaluated the percentage of cycle reduction as estimated by 11vm-mca, static analysis tool that models the execution of machine code on a specific CPU microarchitecture. This metric provides a high-fidelity simulation of hardware-leve execution efficiency and pipeline utilization without requiring physical hardware execution. As shown in Table g our method consistently outperforms static haselines in predicting these simulated microarchitectural performance gains under the -O3 optimization pipeline.
14

---

Published as a conference paper at ICLR 2026
Table 7: Performance predicting absolute execution time dataset).
Representation
MAE 4
R 1
IR2Vec
94.11
0.746
Autophase
86.87
0.808
ProGraML
89.50
0.741
Ours (Behavioral-PQQ
85.54
0.807
Table 8: Performance on predicting cycle reduction percentage (CompilerGym))
Representation
MAE (%) +
IR2Vec
10.86%
ProGraML.
10.34%
Autophase
8.85 %
Ours (Behavioral-PQQ)
8.78 %
C.2 COMPARISON WITH DYNAMIC BASELINES
A key advantage of our quasi-dynamic approach is its ability to capture behavioral signals without the over- head of physical execution. To position method, compared it against purely dynamic haseline using hardware performance counters (HPCs) collected via linux perf.
For this baseline, we extracted 28-dimensional dynamic feature vector during program execution. This vector consists of 20 raw hardware event counts and derived performance ratios, providing comprehensive profile of the program's 's runtime behavior.
Dynamic Feature Composition (28 Dimensions):
Raw Hardware Events (20): We tracked fundamental metrics including total cycles, instructions re- tired, branches, branch-misses, frontend/hackend stalled cycles, cache references/misses (I.1 ..eecccce,, L1-icache, I.IC),, and Translation Lookaside Buffer (TIB) loads/misses (dTLB, iTL.B), as well system-level events context-switches CPU-migrations.
Derived Performance Ratios (8): To capture architectural efficiency beyond absolute counts, we calculated eight ratio-based features: Instructions Per Cycle (IPC), branch miss rate, L1-dcache miss rate, L1-icache miss rate, LLC miss rate, dTLB miss rate, iTLB miss rate, and the frontend stall ratio.
As shown Table 9 despite having access these high-fidelity execution profiles, the dynamic bascline achieves an MAE of 16.83 (R² 0.726), which is higher than our Behavioral-PQ's MAE of 85.54 (R² 0.807). This result underscores that optimization sensitivity (how program reacts to changes) is more robust indicator of post-optimization performance than single-point runtime snapshot, raw HPCs can be susceptible environmental noise and exhibit limited predictive power for complex code transformations.
Table Comparison with 28 28-dimensional dynamic HPC baseline on -03 runtime prediction.
Representation
Type
MAE 4
R² 1
Dynamic (28-dim HPC)
Dynamic
116.83
0.726 0.807
Ours (Behavioral-PQ)
Quasi-dynamic
85.54
C.3 CROSS-TASE GENERALITY: HETEROGENEOUS DEVICE MAPPING
To evaluate whether representation captures general program semantics beyond compiler optimization, we applied the Heterogeneous Device Mapping (DevMap) Cummins al. (017) task. This task requires predicting whether an OpenCI. kernel executes faster on a CPU or a GPU, which demands deep understanding of code-harde hardwaree interactions.
As shown in Table I0 our method achieves the best performance (68.38 38%38% accuracy) among all baseline em- beddings. This demonstrates that optimization-sensiivviy probes elicit fundamental semantic features that generalize other systems-level tasks.
15

---

Published as a conference paper at ICLR 2026
C.4 COMPARISON WITH FINE-TUNED LLMs AND IR IR-BASED METHODS
We further compared our approach against modern Large Language Models (LLMs) and specialized IR-based models under the pre-train + fine-tune paradigm. We fine-tuned Qwen2.5 (at various scales) and IRCoder Paul ct al. (024) on the DevMap task.
Table 10: Comparison with modern models on the DevMap classification task.
Model Representation
Type
Accuracy 1
IR2Vec
Static Embedding
64.71%
Autophase
Static Features
66.91% %
ProGraML
GNN-based
62.26%
Qwen2.5-12 1.5B (Finetuned)
LLM
58.82%
Qwen2.5-3B (Finetuned)
LLM
66.18%
IRCoder (starcoderbase- IB)
IR-Pretrained
61.03%
IRCoder (starcoderbase-7BB
IR-Pretrained
60.29%
Ours (Behavioral-P(Q)
Quasi-dynamic
68.38% %
The results in Table TOhighlhhg that while models like IRCoder excel at code generation by treating IR static text, they may struggle to capture the nuanced behavioral changes critical for performanco-sensitie tasks. In contrast, Bchavioral-PQ' focus on optimization sensitivity provides more effective representation for these domains, even when compared to significantly larger models. This confirms that our quasi-dynamic paradigm complementary to existing IR-based static methods.
D THE USE OF LARGE LANGUAGE MODELS
Large Language Models were utilized to support the enhancement of clarity and coherence throughout the manuscript. They helped with rephrasing. maintaining academic standards, and improving overall readability. It important that their role was confined the writing process, and all material was carefully reviewed finalized by authors themselves.
16


Reviews:
[{"review_id": "YzueLFGr5y", "reviewer": "Reviewer_JrYV", "summary": "This paper introduces a quasi dynamic program embedding that encodes how a program’s static features change in response to compiler optimization probes. The resulting Behavioral Spectrum is discretized via Product Quantization into compositional codes and modeled with a multi task Transformer (PQ BERT). On CompilerGym tasks, Best Pass Prediction and -Oz benefit regression, the approach improves substantially over static baselines, with informative ablations and embedding space analyses.", "questions": "•\tCan you include a dynamic baseline?\n\n•\tCan you use your representation for other tasks? Other than instruction reduction.", "limitations": "", "rating": 4, "confidence": 3, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "•\tNew perspective: optimization sensitivity as representation. \n\n•\tStrong gains vs. static baselines on two tasks. \n\n•\tAblations and qualitative analyses add insight.", "weaknesses": "1.\tThe paper’s central claim is that behavioral embeddings improve optimization prediction. However, neither downstream task measures runtime speedup or other practical metrics (e.g., energy, compile time). This weakens the argument that the representation is task-relevant beyond instruction count reduction. Adding runtime benchmarks or end-to-end pipelines would substantially strengthen the generality of the proposed representation. \n\n2.\tThe work frames itself between static and dynamic profiling, yet no dynamic baselines are included. Including one such baseline would help quantify the “best of both worlds” claim. \n\n3.\tIs the representation created specific to the task of instruction reduction? Or is it general? The representations that the authors compare to (e.g., IR2vec) are general representations that can be used for many tasks. If the proposed representation is specific to the task of instruction reduction, then it is natural that it would be better. Can the authors clarify how general their representation? i.e., how would it perform on other tasks (other than instruction reduction).", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "LWFkEAnSAa", "reviewer": "Reviewer_ijRG", "summary": "This paper aims to address the trade-off between static and dynamic program representations by proposing a quasi-dynamic framework modeling a program’s optimization sensitivity. It introduces the Program Behavior Spectrum, a novel representation constructed by probing program IR using multiple optimization sequences and measuring the resulting changes in static features. These high-dimensional behavioral change vectors are then discretized via Product Quantization, and a transformer model (PQ-BERT) is pre-trained to learn compositional behavioral embeddings. The method is evaluated on two optimization prediction tasks: Best Pass Prediction and -OZ Benefit Prediction, and demonstrates improvements over static baselines.", "questions": "1. What is the computational cost of running the entire pipeline? \n2. Can the learned embeddings support optimization tasks beyond code-size improvement, such as runtime performance optimization?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "- The core idea of deriving representations from optimization-induced behavioral variation is novel and addresses a meaningful gap between static embeddings and full dynamic profiling.\n- The authors validate the contribution of each component in the framework through ablation studies. \n- The authors go beyond downstream accuracy and examine the embedding space, demonstrating an effort towards enhancing interpretability and supporting the utility of their method. \n- The authors have provided the complete code for their experiments.", "weaknesses": "- Limited Evaluation and Generalization: The evaluation is limited to two downstream tasks, which are narrow in scope. \n- No comparison with dynamic baselines. The approach is motivated as more efficient than dynamic profiling, but since it relies on generating multiple optimized variants per program, it would be helpful to include an analysis of the associated computational cost and scalability.\n- Minor fixes:\n- Line 145,146: horig and hopt -> h_orig and h_opt\n- Line 179: ..may suffers from -> may suffer from", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "sbUrAg2rIT", "reviewer": "Reviewer_S2NV", "summary": "The authors introduce the Program Behavior Spectrum, a new representation generated by probing a program's IR with a diverse set of optimization sequences and quantifying the resulting changes in its static features. To effectively encode this high-dimensional, continuous spectrum, the authors  pioneer a compositional learning approach. Product Quantization is employed to discretize the continuous reaction vectors into structured, compositional sub-words. Subsequently, a multi-task Transformer model, termed PQ-BERT, is pre-trained to learn the deep contextual grammar of these behavioral codes. Comprehensive experiments on two representative compiler optimization tasks---Best Pass Prediction and -Oz Benefit Prediction---demonstrate that our method outperforms state-of-the-art static baselines.", "questions": "Please jutify the benefit of the proposed method over pre-train+fine-tuning paradigm.", "limitations": "", "rating": 4, "confidence": 3, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "1. The proposed method has some potential to overcome the limitation of static/dynamic representation method and encode the semantic of a program.\n2. The evaluation is thorough. Ablation studies show that PQ, scale-invariance, and Transformer components each contribute meaningfully.\n3. Open-source implementation is provided.", "weaknesses": "1. The proposed method is limited to compiler optimization tasks, which, while technically meaningful, may be too specialized and fall outside the primary scope of interest for most ICLR readers.\n2. Missing references/comparison to several work on leveraging IR/compiler optimization for improving program embedding:\n    - Unleashing the power of compiler intermediate representation to enhance neural program embeddings ICSE 2022\n    - Ircoder: Intermediate representations make language models robust multilingual code generators ACL 2024\n    - ObscuraCoder: Powering Efficient Code LM Pre-Training Via Obfuscation Grounding ICLR 2025\n3. Building on the point above, the choice of baselines appears outdated. All compared models are from before 2021, while many modern program embedding and code representation models have demonstrated stronger performance and broader applicability. The authors justify their baseline selection by requiring that each method be “pre-trained on the same algorithm classification dataset.” However, this constraint seems unnecessary in the current landscape, as most recent code embedding models are pre-trained on large-scale, diverse corpora. The paper should clarify the practical incentive for adopting the proposed model instead of fine-tuning these stronger, pre-trained representations on the target optimization tasks.\n\nMinor comments:\n1. Autophase should be explained in more details.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)