Can Large Language Models Understand Intermediate Representations in Compilers?
Authors: Hailong Jiang
Jianfeng Zhu
Yao Wan
Bo Fang
Hongyu Zhang
Ruoming Jin
Qiang Guan
Venue: ICML 2025 poster (2025)
Keywords: Large Language Models (LLMs)
Intermediate Representations (IRs)
Code Comprehension
Primary area: deep_learning->large_language_models
TL;DR: Evaluating LLMs’ ability to understand intermediate representations (IRs) from structural, syntactic, semantic, and reasoning perspectives.

Abstract: Intermediate Representations (IRs) play a critical role in compiler design and program analysis, yet their comprehension by *Large Language Models* (LLMs) remains underexplored.  In this paper, we present an explorative empirical study evaluating the capabilities of six state-of-the-art LLMs—GPT-4, GPT-3, DeepSeek, Gemma 2, Llama 3, and Code Llama—in understanding IRs.  Specifically, we assess model performance across four core tasks: *control flow graph reconstruction*, *decompilation*, *code summarization*, and *execution reasoning*.  While LLMs exhibit competence in parsing IR syntax and identifying high-level structures, they consistently struggle with instruction-level reasoning, especially in control flow reasoning, loop handling, and dynamic execution.  Common failure modes include misinterpreting branching instructions, omitting critical operations, and relying on heuristic reasoning rather than on precise instruction-level logic.  Our findings highlight the need for IR-specific enhancements in LLM design. We recommend fine-tuning on structured IR datasets and integrating control-flow-sensitive architectures to improve the models’ effectiveness on IR-related tasks.  All the experimental data and source code are publicly available at [https://github.com/hjiang13/LLM4IR](https://github.com/hjiang13/LLM4IR).

Can Large Language Models Understand Intermediate Representations in Compilers?
Hailong Jiang" Jianfeng Zhu !
Yao Wan
Bo Fang²
Hongyu Zhang*
Ruoming Jin
Qiang Guan"
Abstract
et al., 2020; Webb et al., 2021; Sbirlea al., 2015). Like- wise, IRs enable efficient transformations, optimizations, and analyses that are independent of specific programming languages, thereby promoting portability across different architectures (Chow, 2013).
Intermediate Representations (IRs) play a criti- cal role in compiler design and program analy- sis, yet their comprehension by Large Language Models (LLMs) remains underexplored. paper, we present an explorative empirical study evaluating the capabilities of six state-of-the-art LLMs-GPT-4 GPT-3, DeepSeek, Gemma 2, Llama 3, and Code Llama-in understanding IRs. Specifically, we assess model performance across four core tasks: control flow graph reconstruc- tion, IR decompilation, code summarization, and execution reasoning. While LLMs exhibit compe- tence in parsing IR syntax and identifying high- level structures, they consistently struggle with instruction-level reasoning, especially in control flow reasoning, loop handling, and dynamic ex- ecution. Common failure modes include misin- terpreting branching instructions, omitting criti- cal operations, and relying on heuristic reason- ing rather than on precise instruction-level logic. Our findings highlight the need for IR-specific en- hancements in LLM design. We recommend fine- tuning on structured IR datasets and integrating control-flow-sensite architectures to improve the models' effectiveness on IR relreade tasks.'
this
Beyond their role in compilation, IRs are crucial for va- riety of code intelligence tasks, including vulnerability de- lection (Zhu al., 2024; Jiang al., 2024b), code com- prehension and generation (Yuan et al.. 2023; Jiang et al., 2024c), clone detection (Paul et al., 2024), and binary-to- source matching (Mao et al., 2023; Collyer et al., 2023). Unlike high-level programming languages, IRs adopt an instruction-leww format that encodes low-level details such control flow dependencies, instruction transformations, and memory operations. These characteristics introduce unique challenges for structural analysis, syntactic process- ing, semantic understanding, and execution reasoning, mak- ing IR comprehension fundamentally distinct from model- ing natural language or high-level source code.
Current IR processing methods, such as NCC (Venkata- Keerthy et al., 2020), represent IR instructions lin- ear sequences of tokens using Word2Vec-style embed- dings (Church, 2017), and BERT-style pretraining (Devlin, 2018; VenkataKeerthy et al., 2020; Niu et al., 2024). How- ever, these approaches lack an understanding of control flow structures and execution semanties, limiting their abil- ity generalize to IR-specific tasks such as Control Flow Graph (CFG) reconstruction, bug detection, and execution reasoning. Since IR execution is governed by low-level op- erational semantics rather than surface syntax, models that fail to capture control flow relationships often struggle with execution-aware IR tasks.
1. Introduction
Intermediate Representations (IRs) play pivotal role in compiler design by structuring the compilation process into front-end, middle-end, and back-end phases (Reissmann
Our Study Recently, with the remarkable advancements in LLMs for source code understanding and generation (Jiang et al., 2024d; Wang al., 2023; Liu al., 2023a), natural question arises: Can LLMs understand IRs in compilers? To investigate this, we systematically evaluate LLMs' compre- hension of IRs by assessing their performance across three dimensions. An overview of our evaluation framework and its alignment with the RQs is illustrated in Figure
"Equal contribution KKen State University, USA 'Huzhoong University of Science and Technology. China Pacific Northwest National Laboratory, USA 'Chongqing University, China. Cor- respondence to: Yao Wan < <wanyao@ ed Bo Fang <bo.fang @ ppnnl.gov> Qiang Guan <qguan @ edu...
Proceedings of the 4nnd International Conference on Machine Learning, Vancouver, Canada. PMLR 267, 2025. Copyright 2025 by the author(s).
All the experimental data and source cade are publicly avail- able at https: / / thub. com/h jinj/ijiaaa a.gl1//1.. /LLMMMI IR.
We first investigate whether LLMs can accurately recon-

---

Can Large Language Models Understand Intermediate Representations in Compilers?
Task 1: CPG Reconstruction
RQ 1: Can LLMss accurately reconstruct CFGs?
Reconstruct
Source Code
Compiler
IRs
Prompt
LLMs
Control Flow Graphs
Task IR Decompilation
Source Codes
Decompile
IRs
Pass
Reasoning
RQ 2: Can LLMs understand the
LLMs
Prompt
syniax and semantics of IRs?
Assertion Statements
Prompt
LLMs
Fail
IR Semanties
Summarize
RQ3: Can LLMs reason over IRs to simulate program
execution and predict correct outcomes?
LLMs
Prompl
Task 3: Code Summarization
Task 4: Execution Reasoning
Figure I. Overview our evaluation framework for assessing LLM understanding of IRs across four tasks.
struct CFGs from IRs (RQ1). To evaluate structural un-
into how LLMs process IRs:
derstanding (RQ1), we design Task to prompt LLMs to
reconstruct CFGs from a LLVM IR. This directly tests their
RQI (Task 1): LLMs can recover basic blocks and par- tial graph topology, but often misconstruct control flow
ability to identify basic blocks, interpret control-flow in-
structions (e.g., branches and loops), and infer execution
edges-especcally in branches and levps-revealing lim- ited structural reasoning in CFG reconstruction.
paths-keys for IR analysis and compiler optimization.
RQ2 (Tasks 2-3): LLMs can parse IR syntax and iden- tify basic structural patterns, but frequently skip critical instructions or rely on heuristic approximations when gen- erating code or summaries cunnmares--ndiiating a semantic gap in IR understanding.
We further examine whether LLMs understand the syntax
and semantics of IRs (RQ2). This RQ targets both syntactic recognition and semantic comprehension, which are critical for understanding IR beyond its surface form. We evaluate this through two tasks: decompilation (Task 2: converting IR into high-level source-liks like code) and semantic summa- rization (Task 3: generating natural language descriptions of IR behavior).
RQ3 (Task 4): LLMs approximate exccution reasoning through high-level semantic steps instead of simulating instruction-level behavior, leading to failures in tracking control flow, variable updates, and iteration logic-eves even with chain-of-thought prompting.
Lastly, we investigate whether LLMs can reason over IRs to simulate program execution and predict correct outcomes (RQ3). This RQ probes the models' capacity for dynamic reasoning- tracking control flow, variable states, and in- struction effects under specific inputs. We design exe- cution reasoning task where LLMs are provided with IR code and input values, and are asked to determine whether assertions on output states will hold (Task 4).
The primary contributions of this paper summarized as follows.
Contributions
To the best of our knowledge, this is the first study on LLMs' capacity to comprehend and reason over IRs, cov- ering structural analysis, syntactic and semantic under- standing, and execution reasoning.
We empirically evaluate state-of- the-art LLMs across mul- tiple IR-processing tasks, introducing targeted tasks, rig- orous evaluation metrics, and systematic analyses.
In our experiments, we evaluate six tate-of-the--ae art LLMs: GPT-4 (OpenAI, 2023), GPT-3 (Brown et al., 2020), DeepSeek (Guo al., 2024), Llama 3 (Touvron et al., 2023). Gemma 2 (Team et al., 2024), and Code Llama (Rozière et al., 2024). All evaluations are conducted on benchmark dataset derived from HumanEyal (Zheng et al., 2023), con- sisting of 164 C++ programs paired with their corresponding LLVM IRs. Each program compiled using Clang at four optimization levels--00, -01,- -02, and -03-to gener- ate diverse set of LLVM IRs that capture both unoptimized and progressively optimized code structures.
Our findings pinpoint critical limitations- -uch as insuf- ficient control flow awareness, shallow instruction- semantics, and flawed execution reasoning laying the foundation for future IR- specifi model designs.
2. Preliminaries
LLVM and IRs LLVM is a widely used compiler infras- tructure that optimizes code via IRs (Lattner & Adve, 2004).
Takeaway Findings Our results reveal several key insights
LLVV IRs are lowelevel, statically typed, and follow the
2

---

Can Large Language Models Understand Intermediate Representations in Compilers?
Static Single Assignment (SSA) form, offering a language- agnostic representation. They support control flow analysis and enable custom compiler passes.
3.1. Evaluation Tasks
CFG Reconstruction (Task 1) This task evaluates whether LLMs can accurately reconstruct the CFG from a given IR function. As RQI focuses structural understanding, this task serves as direct probe: generating a CFG necessitates interpreting branches, loops, and control transfers-key components of IR structure. A model that successfully reconstructs the CFG demonstrates a substantive grasp of the structural semantics inherent in the code.
Program
Examppee
life
Module
G++
1ift sum
by
Rosts Block
Complleer
Instruction
LUM
Fadd
add nsw 1.2 50, %0
do load 132, pir Ka, align
IR Decompilation (Task 2) To investigate whether LLMSs understand the syntax of IR (RQ2), we evaluate their ability to translate IR back into high-level source code. Decompi- lation serves as a rigorous test of syntactic comprehension, as it demands the recognition of structural patterns- such as loops and conditionals- a flattened IR format, and their accurate reconstruction into readable, high-level con- structs. A successful decompilation indicates that the model has sufficient grasp of IR syntax to recover valid source- level structures.
Oupul
Opeicter
Types
Operands
[1 load 132, ptn Rb, aliggn
ldwriter
Kadd
132 58, 33 store 132 Sadd, ptr Rsun, wllign
Figure 2. An illustration of LLVM IR with the corresponding hier- archical structure.
Figure 2 shows the hierarchical structure of a program's is LLVM IR, from modules down to individual instructions. An LLVM IR program consists of several hierarchical com- ponents. At the top level, a module encapsulates a set of re- lated functions, serving as a compilation unit. Each function contains a sequence of basic blocks, which are straight-ling instruction sequences with a single entry and a single exit. These basic blocks form the nodes in control flow graphs and are the fundamental units of control transfer. Inside each block, instructions define operations with explicit opcodes, operand types, and values, governed by LLVM's strict static typing system.
Code Summarization (Task 3) This task complements Task 2 by shifting the focus toward semantic understand- ing. In this setting, L.LMs are required to generate concise natural language summaries of IR functions. Addressing RQ2 involves more than surface-level parsing; it demands deeper comprehension of the IR's behavior and purpose. Summarization serves as proxy to evaluate whether the model can accurately capture functional intent, behavior, and
revealing its capacity to reason about function-level semantics.
Control Flow Graphs (CFGs) A CFG is a fundamental program representation that captures all possible execution paths of a program. Each node in the CFG corresponds to basic block-a straight-line sequence of instructions with a single entry and exit point-while edges represent possible control transfers between these blocks. CFGs serve as the backbone for numerous compiler and analysis tasks. In vulnerability detection, they help identify unreachable code, malicious branches, and control hijacks (Zhou al., 2019; Anju et al., 2010). In compiler optimizations, CFGs enable transformations such dead code elimination, loop unrolling, and branch prediction (McConnell, 1993). Addi- tionally, in broader program analysis. CFGs support sym- bolic execution, slicing, and structural reasoning (Fechete et al., 2008), making them indispensable for understanding and manipulating program semantics at the IR level.
Execution Reasoning (Task 4) To address RQ3, we eval- uate whether LLMs can simulate the execution of IR given specific input values. This task requires the model to deter- mine whether assertions on the output state hold, necessitat- ing accurate tracking of control flow, data dependencies, and variable updates. Unlike simple pattern matching, execution reasoning demands that the model interpret IR as a dynamic process, reflecting deeper level of semantic understanding.
Together, these tasks provide comprehensive and struc- tured benchmark aligned with the three core RQs, offering insights into how well LLMs can understand, interpret, and reason over compiler IRs.
3.2. Studied LLMs
In this paper, we investigate six state-of-the-art LLMs that span a range of model families, training objectives, and application scopes. These models are selected to provide balanced coverage of proprietary and open- open-source mod- els, general-purpose and code-specialized LLMs, as well as diverse training paradigms (c.g., instruction tuning, multi- modal pretraining), ensuring representative and fair evalu- ation across usage scenarios.
3. Study Design
As shown in Figure 1, to address our RQs (RQ1-3), we
design four evaluation tasks (Task 1-4), each targeting a specific dimension of IR comprehension and aligned with the reasoning skills required to answer the corresponding research question.
3

---

Can Large Language Models Understand Intermediate Representations in Compilers?
GPT-4 (OpenAI, 2023) and GPT-3 (Brown al., 2020), developed by OpenAl have demonstrated strong perfor- mance across a wide range of code-related tasks. GPT-4, in particular, is known for its enhanced reasoning capabil- ities and broader context window.
Table I. Key statistics of HumanEval IRs under different optimiza- tion levels (-00 to -03). BB: Basic Block.
opt Level
LoC
Tokens
Functions
Loops
BBs
-00
162,389
1,079,290
5,195
463
13,228
DeepSeek R1 (Guo et al., 2024) is an open-source, instruction-tuneds model trained on both code and nat- ural language corpora. Notably, it comes with built-in support for Chain-of-Thoure
-01
66,598
454,954
393
577
7,708
02
69,102
478,002
374
625
8,048
03
75,519
526,395
367
827
8,917
(CoT) prompting, enabling multi-step reasoning without manual CoT demonstrations. Ilama 3 (Touvron et al., 2023), developed by Meta, is a family of high-performance transformer models with open weights.
and are given detailed, domain-specifii instructions.
To enhance response precision and consistency, we adopt Expert Meta-Templat Prompt format:
Code Llama (Rozière et al., 2024) extends Llama with additional pretraining on source code, making more capable in code synthesis and static analysis.
Prompt
Gemma 2 (Team et al., 2024) is instruction-tuns tuned model released by Google, offering competitive perfor- mance in general-purpose and technical reasoning tasks. These models cover a spectrum training objectives, access modes, and design philosophies, providing a com- prehensive basis for evaluating LLMss' ability to reason over compiler IRs.
You are expert in Irelevant fiel))
Input Data: [IR snippet or relevant context]
Task: [Task description]
Output Format: [Expected output structure]
This template ensures that LLMs receive explicit role as- signments, domain knowledge, task objectives, and output formatting requirements collectivvey guiding them toward more accurate and well-structured responses.
3.3. Datasets
Following (Zheng et al., 2023; Tan et al., 2024), we use the HumanEval benchmark, which comprises 164 programming tasks introduced by OpenAl (Chen al., 2021). Each task includes function signature, natural language description, reference implementation, and an average of 7.7 assertion statements. In our experiments, the C++ functions serve as the basis for generating LLVM IRs across various compiler optimization levels. For further discussion on dataset gener- alizability and task complexity, please refer to Appendix B.
For each of the four tasks (i.e., CFG reconstruction, IR decompilation, code summarization, and execution reason- ing), iteratively refine prompts using strategies such as few-shot learning and CoT prompting (Wei et al., 2022; Xie et al., 2025). Multiple prompt variants are evaluated to maximize accuracy, consistency, and task alignment.
3.5. Implementations
Task 1: CFG Reconstruction We prompt LLMs to gener- ate CFGs from IR snippetsusing structured meta- veta-template that defines graph syntax and node semantics (see Sec- tion 3.4). The prompt includes explicit guidance and erative refinement. Following (Ma et al., 2024), we evaluate CFG reconstruction along multiple axes. Node construction accuracy measures the correctness of identified basic blocks, while edge construction accuracy evaluates the accuracy of control -ontrol-flow edges. Based on these, we define two aggregate metrics. Full CFG accuracy refers to the case where both nodes and edges match the ground truth, whereas partial CFG accuracy applies when nodes are correct but edges contain errors. To capture finer- mnner-griieed structural reason- ing, additionally report loop edge accuracy, which mea- sures correct reconstruction of loop structures (e.g., for, wh l e), and conditional branch accuracy, which evaluates the correctness of conditional branches (c.g., if-else).
Table 1 summarizes the key IR characteristics across differ- ent optimization levels from -00 to -03. At -00 the IR remains verbose due to the lack of optimizations, preserving duplicated functions such as linkonce_odz to facilitate debugging. As the optimization level increases to -01 and -02, redundant structures are pruned, and moderate loop constructs begin emerge as a result of early optimization passes. The -03 level applies more aggressive transfor- mations that substantially restructure the IR, introducing deeper and more complex loop-level optimizations.
3.4. Prompt Design
Effective prompt design is crucial for optimizing LLM per- formance (Liu et al., 2022). Traditional single-turn prompts that embed role instructions and task descriptions into sin- gle input often lack clarity and structured guidance. Recent studies (Liu et al., 2023b; Park et al., 2023) demonstrate that LLMss perform more reliably when treated as expert agents
Task 2: IR Decompilation We prompt LLMs to decompile into C++ source code and compare the results generated

---

Can Large Language Models Understand Intermediate Representations in Compilers?
to the original ground truth. The task probes whether LLMs recover syntactically valid and semantically faithful source code from IRs, across optimization levels (00-03), Following prior work (Tan et al., 2024; Armengol-Eston Estapé et al., 2024; Wong et al., 2023; Jiang et al., 2024a), we evaluate decompiled outputs using three hierarchical met- rics. task completions assesses whether the model produced syntactically valid code. execution completion, checks whether the code compiles and runs. Among these, we dis- tinguish between re-execution success (all assertions pass) and re-execution mismatch (at least one assertion fails).
Table 2. LLM performance on CFG construction.
Task Comp.
Node
Full Acc.
Partial Acc.
LLMs
Acc.
Loop Condition
GPT-4
164
50
39
2
GPT-3
133
18
14
3
1
DeepSeek R1
77
64
57
6
1
Gemma 2
98
18
16
2
0
Llama 3.1
76
17
15
2
0
Code Ilama
67
0
0
0
0
Task 3: Code Summarization We prompt LLMs with IR code snippets and request function-leve summaries. To improve output quality, we apply expert-designed meta- prompts with few-shot examples and, where appropriate, CoT guidance. The generated summaries are compared to ground-truth or human-curated references. Following (Wu et al., 2024), we evaluate summary quality using BLEU (Pa- pineni et al., 2002) for surface fluency, ROUGE-L (Lin, 2004) for content coverage, and METEOR (Denkowski & Lavie, 2014) for semantic alignment.
4.1.1. STRUCTURAL REASONING OVER CFGS
From Table 2, observe substantial variation in both task completion and CFG generation accuracy across LLMs. GPT-4 completes all 164 cases but produces only 39 fully correct CFGs. In contrast, DeepSeek RI handles just 77 cases yet yields 57 accurate CFGs, suggesting stronger per- instance reasoning capabilities. This improvement likely attributable to its use of CoT prompting, which explicitly guides step-by-step control flow reasoning. Other models, including GPT-3 (14 conrect), Gemma 2 (16), and Llama 3.1 (15), show weaker performance, while Code Llama fails to generate any correct CFGs.
Execution Reasoning We present LLMs with IR snippets along with embedded assertion statements and prompt them to predict which assertions evaluate to "true." This task requires the model to simulate program behavior through static reasoning over control flow and variable as- signments. We evaluate execution reasoning by checking assertion-Ievel correctness: pass if all are correct, par- tial pass if some are correct, and failure if none are. We also report the overall pass rate, the proportion of correctly predicted assertions across all cases.
Task 4:
Structural errors further confirm these limitations. GPT- alone incurs 1 14 node-level mistakes, with GPT-3 and Gemma 2 each exceeding 100. Loop and conditional edge recovery remains poor across the board: GPT-4 reconstructs only 9 loops and 2 branches; DeepSeek RI achieves 6 and 1, respectively, despite fewer inputs. These results highlight persistent difficulties in accurately identifying instruction boundaries and modeling control dependencies.
4. Experimental Results and Analysis
Finding 1: LLMs can identify basic control instruc- tions and construct partial graph structures, but often misrepresent branches and loops-revealing limited structural reasoning and poor control dependency un- derstanding. Chain-of- th-tuugght prompting improves accuracy by enabling stepwise CFG reconstruction.
To assess how effectively LLMs understand IRs, we struc- ture our analysis around the three RQs outlined in Figure This section presents the quantitative results and highlights key findings. Each model output is categorized as either Task Completed, indicating that the model produced valid response, or Task Failed, typically due to hangs or crashes caused by IRs exceeding the model's 's context window.
4.1.2. IMPACT OF GRAPH DENSITY
4.1. Structural Understanding (RQ1)
Upon further analysis, we observe that applications in dif- ferent performance groups exhibit distinct density patterns, suggesting that density influences model behavior and re- construction outcomes.
In this RQ, we evaluate whether LLMs can reconstruct CFGs from IRs, revealing their structural understanding. We assess LLMs' structural understanding by prompting them to reconstruct CFGs from IRs. Their outputs are com- pared with compiler-generate ground truth using metrics including full and partial CFG accuracy, as well as cor- rectness of loop and conditional edges. Table 2 presents model-wise performance across 164 applications, ensuring robustness through multiple runs.
Graph Density Graph density quantifies how intercon- nected a CFG is, calculated as:
Density = Number of Edges
(I)
X (n 1)
where n is the number of nodes.
5

---

Can Large Language Models Understand Intermediate Representations in Compilers?
Table The number of applications in each category for different .. in decompilation tasks, including task completion, re-exccution, and success across optimization levels (00-03).
LLMs
Task Completed
Re-execution Completed
Re-execution Success
Opts
-00
-01
-02
-03
-00
-01
-02
-03
-00
-01
-02
-03
GPT-4
161
164
164
164
34
34
38
32
15
12
10
14
GPT-3
95
145
144
133
2
7
2
DeepSeck RI
62
76
75
77
31
39
40
38
22
16
17
17
Gemma 2
72
100
100
98
0
12
16
9
0
7
6
6
Llama 3.1
72
97
96
77
0
24
26
23
0
15
15
14
Code Llama
65
88
83
120
14
16
21
19
4
6
6
4
4.2. Syntax and Semantic Comprehension (RQ2)
0.30
CFG Con. Accuracy
Construction
0.25
Example (y=D))1)
In RQ2, we evaluate whether LLMs can comprehend both the syntactic and semantic aspects of IRs. We address this via IR decompilation and code summarization (Tasks 2-3).
Example Two (y=0.020)
0.183
Dendity
0.1.6
0.15
0.10
0.092
0.098
0.102
4.2.1. HEURISTIC IR RECONSTRUCTION
0.045
0.051
We use the IR decompilation task to test syntactic under- standing, where LLMs are prompted to reconstruct high- level C++ code from IR. Outputs are evaluated based on syntactic validity, execution behavior, and correctness of pre- dicted assertions (Tan et al., 2024; Armengol-Estappe et al., 2024; Wong al., 2023). All experiments conducted across optimization levels (-00 -03) test robustness under compiler-induced IR variation.
0.029
0.021
0.00
GPT4
GPT44
Gemmna Lima 3.1
Liama
Gemma
GPT-4
Figure Density metrics comparing Complete CFG Accuracy vs. Node Constraction Error.
Table shows the number of applications in each category for different LLMs decompilation task. From Table 3, most task failures occur -00, where minimally optimized IRs are the longest (Table 1), often exceeding model context limits. GPT-4 handles these cases best with only 3 failures, while Code Llama fails 99 applications. Interestingly, re-execution success peaks at -01 and -02, likely because they preserve structured control flow while reducing redun- dancy-makingg IRs easier for LLMs to interpret. In contrast, C3's aggressive optimizations (e.g., loop unrolling, instruc- tion reordering) obscure control dependencies and degrade decompilation accuracy.
Figure 3 compares complete CFG accuracy (black bars) and node construction error (gray bars) across density levels. The æ-axis groups models by overlapping task success (e.g., "GPT-4 only" or "GPT-4 Gemma 2"), while the y-axis indicates the average density of those programs. Error bars denote the density range per category. The distinct density patterns between successfully reconstructed CFGs and those with high node errors indicate that graph density strongly correlates with LLM performance. This insight can inform model selection. As shown by the two examples the Figure 3, CFG density appears to influence which models succeed or fail the task:
Across multiple examples, observe that LLMs frequently omit segments of IR-such entire i statements nested within loops loops-resulting in incomplete or oversimplified de- compiled code. Instead of following a strict instruction- by-instruction reconstruction process, LLMs often rely on heuristic pattern recognition infer missing logic. This heuristic behavior limits their capacity to capture fine- grained control flow semantics. In particular, loop structures are often approximated rather than faithfully reproduced, leading to missing branches or partially handled conditions. Similar patterns are evident code summarization and ex- ecution reasoning tasks (Section 4.3), where LLMs fail to
Higher-density CFGs (avg. 0.11): GPT-4 and Gemma 2 succeed more often, while Llama 3.1 fails.
Lower-density CFGs (avg. 0.02): Llama 3.1 performs better, while GPT-4 and Gemma struggle.
Finding 2: LLM performance CFG reconstruction varies with graph density, indicating its potential as model selection hcuristic in IR analysis.
6

---

Can Large Language Models Understand Intermediate Representations in Compilers?
Table Performance comparison of L.LMs in the code summariza- tion task, including BLEU (B), ROUGE-L L (R-L), METEOR (M).
Table 5. Execution reasoning performance of L.I.Ms across IRs and Source Code (SC). Pass rates are calculated based on the number of completed tasks.
Task Comp.
High Quallty
Avg. Scores
LLMs
R-L
M
B
R-L.
M
Task Comp.
Pass
Partial Pass
Pass Rate
LLMs
IRs
SC
IRs
SC
IRs
SC
IRs
SC
GPT-4
164
9
21
0.35 0.56 0.63
GPT-4
164
164
59
118
102
44
0.36
0.72
GPT-3
131
1
1
1
0.24
0.47 0.56
GPT-3
164
164
7
26
135
135
0.04
0.158
DeepSeek R1
49
5
10
0.42
0.64
0.705
DeepSeek
164
164
32
119
132
25
0.195
0.804
Gemma 2
88
4
7
0.35
0.52 0.63
Gemma 2
164
164
32
100
132
63
0.195
0,610
Llama 3.1
81
5
11
0.39
0.61
0.67
Llama 3.1
164
164
31
119
114
35
0.189
0.73
Code Llama
123
1
1
0.25
0.43
0.55
Code Llama
163
164
44
131
119
33
0.270
0.80
fully represent execution semantics. As shown in Figure 5 (Appendix F.2), such semantic drift can result in outputs that pass re-execution checks while omitting critical logic.
put/output types, and simple arithmetic-they struggle with deeper semantic elements. The scarcity of high-quality summaries (score > 0.8 across BLEU, ROUGE-L, and METEOR) indicates difficulty in capturing iteration pat- terns, conditional branches, and nontrivial data dependen- cies. This gap implies that current models rely more pat- tern recognition than on reconstructing true program logic. One representative example is provided in Appendix F.3.
Finding 3: LLMs can partially generate syntactically valid C++ code from IR, but often rely on heuristic reconstruction, skipping critical instructions and mis- representing control revealing limited semantic understanding.
Finding 4: LLMs demonstrate an initial understand- ing of IR syntax and semantics, yet their outputs often omit critical instructions, oversimplify control flow, and rely on shallow patterns- revealing limited sen- sitivity fine-grained semantics and an incomplete grasp of underlying program logic.
4.2.2. PATTERN-BB BAASED SUMMARIZATION
To evaluate LLMs' ability to capture the broader semantics of IR, we prompt each model to generate natural language descriptions of IR-derived code snippets (Task 3). This task assesses whether LLMs can articulate a function's 's purpose, inputs, outputs, and internal behavior. Generated summaries are compared to predefined golden references using BLEU,, ROUGE-L, and METEOR. Higher similarity scores indicate stronger semantic comprehension and the ability infer meaningful program behavior.
4.3. Execution Reasoning (RQ3)
In this RQ, aim to evaluate whether LLMs can simulate the behavior of IR code, reasoning over control flow, vari- able state, and memory updates to predict runtime outcomes with certain inputs.
Table 4 presents model performance on Task 3. GPT-4 com- pletes all 164 summaries, while GPT-3, Gemma 2, Llama 3.1, and Code Llama fail on 31, 76, 83, and 41 programs, re- spectively. However, high-quality summaries (score > 0.8) rare across all models. Each produces only one such case in BLEU. For ROUGE-L, GPT-4 leads with 9 high- scoring outputs, followed by Llama 3.1 (5), Gemma 2 (4), and others (1 each). GPT-4 also ranks highest in METEOR with 21 cases, ahead of Illama 3.1 (11) and Gemma (7). In terms of average scores, DeepSeek RI performs best overall, achieving the highest BLEU (0.42), ROUGE-L (0.64), and METEOR (0.705). Llama 3.1 and GPT-4 follow closely, while GPT-3 and Code Llama lag behind, reflecting weaker semantic modeling. These results highlight that while LLMs can generate summaries, their ability to capture fine-grained IR semantics remains limited.
We prompt LLMs with IR snippets annotated with assertion statements, derived from the HumanEval benchmark, and ask them predict which assertions evaluate tuu.. To encourage detailed reasoning, prompts request intermediate execution logs, tracing control flow decisions, and variable states. We further design counterfactual setup, where models are given both IR and the corresponding source code, to compare exccution behavior across representations. Outputs are evaluated using pass/fail metrics based on the correctness of assertion predictions.
1.3.1. APPROXIMATION IN EXECUTION REASONING
Table 5 summarizes LLM performance on execution reason- ing for both IR and source code. All models complete most tasks successfully, with only one failure from Code Llama on an IR case indicating that execution reasoning likely processes IR incrementally, step by step, rather than all at
These results suggest that while LLMs can identify surface- level syntactic features- such as function signatures, in-
7

---

Can Large Language Models Understand Intermediate Representations in Compilers?
once in structural syntactic tasks. This interaction style may alleviate token limitations seen in earlier tasks. Despite this, clear performance gap remains: LLMs consistently perform better on source code, suggesting they are more attuned to high-level constructs. GPT-4 achieves the highest IR pass count (59), while Code Llama leads on source code (131 passes), likely benefiting from code-specific pretrain- ing. These trends imply that IR-spccific training may be essential to close the gap between SC and IR reasoning.
execution traces or reinforcement-guiide prompting.
Finding 5: LLMs approximate program execution through high-level semantic steps rather than faith- fully simulating instruction-level behavior, leading to failures in complex control flows.
4.3.2. HEURISTIC INFERENCE UNDER UNCERTAINTY
In this task, we observe that LLMs often struggle with com- plex control flows, leading to inconsistencies between IR and source code execution. While most models handle simple control logic correctly, assertion failures increase with nested conditions or iterative patterns. For instance, in "CPP_93", an encoding functiin that replaces vowels works for short input (encode ( ""TEST") == "tgst"), but fails on a longer one (encode (*I DoNt KnOw WhAt to WrItE")), ), due to skipped transformation steps during reasoning.
Unlike source code, IR lacks explicit variable names and high-level semantics, making execution reasoning more am- biguous. When faced with such uncertainty, LLMs rarely express a lack of understanding. Instead, they often fall back on heuristic strategies-suce such as inferring behavior from function names or assertion patterns-rather than reasoning from instruction semantics.
This behavior frequently leads to incorrect outcomes. In 35 of 1 164 GPT-4 cases, observe such pattern- guessing. For example, in "CPP 120", a function named maximum leads the model to assume that the IR computes top-k ele- ments, even without evidence from the actual IR instructions. Similarly, in "CPP_109" the model infers that a vector can be sorted through shifts simply because the assertion passes, without reasoning through the IR-defined shift operations.
These failures reveal that LLMs do not simulate exccution instruction by instruction. Instead, they approximate behav- ior using high-level semantic steps. Analysis of reasoning logs shows that models typically organize their execution into abstract phases-suce as function understanding, in- put analysis, operation planning, and assertion compari- son-rather than processing each instruction sequentially. This abstraction introduces gaps when reasoning over con- trol dependencies, particularly loops or conditionals.
These failures highlight LLMs' inability to handle uncer- tainty in IR reasoning. Even with CoT prompting, models fail to track control flow, variable updates, or simulate inter- mediate steps. As a result, they often prioritize plausibility over correctness, revealing a fundamental limitation in oper- ational semantic understanding. These behaviors reveal a potential limitation in how LLMs manage uncertainty dur- ing IR reasoning-a pattern further discussed in Section 5.
We also observe granularity issues. LLMS can identify overall algorithmic intent (c.g., sorting), but miss critical details. In "CPP_149 149", where the program deletes strings of odd length before sorting, the model correctly recognizes sorting but omits the filtering step changing the outcome.
Finding 6:
When facing ambiguity in IR execution, LLMs often rely on heuristics, such function names or output patterns, rather than simulating operations, revealing challenges in semantic alignment.
CPP 149 Function Description:
A function that accepts vector of strings as a parameter deletes the strings that have odd lengths from it, and returns the resulting vector in sorted order.
However, the LLM skipped an 'odd number removal" step, altering execution behavior. Here is a reasoning log snippet:
5. Discussion
Our evaluation reveals fundamental challenges in LLMs' ability to understand and reason over compiler IRs, particu- larly in control flow, execution semantics, and iterative logic. These difficulties stem from both the intrinsic properties of IRs, such as their verbosity and lack of high-level abstrac- tions, and the limitations of current LLM architectures.
Step 3:
Analyzing the LLVM IR code:
The LLVM IR code is complex and involves multiple opera- tions, including comparisons, memory allocations, and calls other functions.
Key parts of the code involve loading strings from the vector, comparing them, and potentially invoking sorting or filtering operations.
IRs typically produce longer token sequences per function compared to source code, intensifying context length con- straints. This not only limits the model's ability to retain global context but also disrupts its handling of structural dependencies. Even with full context, models frequently
This further demonstrates the limitations of high-level ap- proximations and motivates the need for more structured, step-tracking reasoning mechanisms, such as fine-tuning on
8

---

Can Large Language Models Understand Intermediate Representations in Compilers?
fail on tasks that require precise execution tracking fine- grained semantic reconstruction. GPT-4 outperforms other models across tasks, consistent with prior findings code understanding (Ma et al., 2023), but still falls short on com- plex control and semantic reasoning.
contextual information for downstream tasks. Approaches like IR2Vec (Venkatakeerthy al., 2020) address this lim- itation by introducing hierarchical vector representations that enhance the semantic understanding of IRs. Recent work, such as FAIR (Niu et al., 2024), further refines IR representations using graph Transformers to reduce over- smoothing issues while incorporating pre-training tasks that explicitly capture IR token semantics and flow-type informa- tion. Meta's s LLM Compiler (Cummins et al., 2024) aligns with these efforts, offering pre-trained models for code op- timization tasks. While previous work has explored IR representation for code optimization, no study has system- atically examined how LLMs understand IR syntax, CFGs, execution behavior, and semanties. This study provides the first empirical evaluation of LLMs' IR comprehension across these dimensions.
These findings reveal three key limitations in current LLMs and point to corresponding opportunities for improvement. First, control flow comprehension is fragile-LLMs fre- quently misrepresent branching and looping structures, lead- ing to degraded structural fidelity. Second, models often rely on superficial pattern matching rather than simulating execu- tion. In the face of ambiguity, they tend to infer logic from function names or assertion syntax instead of expressing uncertainty or reasoning through operations, highlighting a lack of uncertainty modeling IR reasoning. Third, loop reasoning remains major weakness: models struggle to track induction variables, identify termination conditions, and capture inter-iteration dependencies. Addressing these limitations may require instruction-level fine-tuning, super- vision via execution traces, or prompting strategies that explicitly incorporate uncertainty.
7. Conclusion and Future Work
This study systematically evaluates the capabilities of LLMs in understanding IRs in compilers, across four dimensions: structural analysis, syntax comprehension, semantic rea- soning, and execution simulation. Our findings show that while LLMs can effectively recognize static IR features and basic control flow patterns, they struggle with more complex constructs-ppariicllaat particularly loop reasoning, precise control dependencies, and instruction-level execution be- havior. Furthermore, LLMs often omit key instructions during decompilation, misinterpret control structures, and approximate execution in coarse semantic steps rather than performing fine-grained simulation. This contrasts with their strong performance other code tasks, suggesting a lack of exposure to IR and the need for IR- -peecciciic training.
6. Related Work
LLMs for High-level Programming Languages Recent advancements in LLMs for NLP have extended to code understanding, enabling models to comprehend high-level languages like Python, C++, and Java (Wan et al., 2024). Models such as GPT-3 (Brown et al., 2020). GPT-4 (Ope- nAI, 2023), Llama (Touvron et al., 2023), and Claude 3 (An- thropie, 2024) excel at tasks like code generation, translating natural language into executable code. Specialized models, including StarCoder (Li et al., 2023). Code Llama (Rozière et al., 2024), and Code Gemma (Team, 2024), enhance these capabilities, tackling more complex coding tasks and driv- ing advances in software engineering applications (Zhao et al., 2023; Wan et al., 2024).
To advance IR-level understanding, future work should en- hance control flow awareness, support multi-granularinn semantic reasoning, and improve loop modeling. Further- more, incorporating IR-specific training. optimizing token- efficient representations, and leveraging graph- neural architectures may further strengthen LLMs' reasoning over IR. Bridging the gap between source code comprehension and IR analysis is crucial for applying LLMs to compiler design, program analysis, and binary code understanding.
IR Representation Learning IR representation learning integrates structural and flow-based features, such as token sequences (Peng et al., 2021; Gui et al., 2022), SAT (Guo et al., 2023), CFGs (VenkataKeerthy et al., 2020; Yu et al., 2020; Chu et al., 2024), and control-data flow graphs (CD- FGs) (Ben-Nun et al., 2018; Brauckmann et al., 2020; Cummins et al., 2021). In terms of model architectures, graph neural networks have been widely used to encode the structures of CFGs and CDFGs via message-passing tech- niques (Brauckmann et al., 2020; Cummins et al., 2021; Yu et al., 2020). Other approaches include skip-gram embed- dings, such as inst2vec (Ben-Nun et al., 2018), and relation- based embeddings, like TransE (Bordes et al., 2013), which are trained on CDFGs and CFGs to produce instruction-level embeddings. However, these models lack task-agnostic pre- trained embeddings, limiting their ability to capture essential
Acknowledgments
This research supported by the U.S. Department of Energy, through the Office of Advanced Scientific Com- puting Research's 's "End-to-end co-design for performance, energy efficiency, and security in Al-enabled computational science (ENCODE)" project. Pacific Northwest National Laboratory operated by Battelle Memorial Institute for the U.S. Department of Energy under Contract No. DE- AC05-76RL001330
9

---

Can Large Language Models Understand Intermediate Representations in Compilers?
Impact Statement
J., Winter, C., Hesse, C., Chen, M., Sigler, E., Litwin, M., Gray, S., Chess, B., Clark, J., Berner, C., McCan- dlish, S., Radford, A., Sutskever, I., and Amodei, D. Language models are few-shot learners. arXiv preprint arXiv:2005. 14165, 2020.
This work investigates LLMs' ability to understand IRs, advancing research in code analysis, optimization, and au- tomated reasoning. may benefit software reliability, but it also raises concerns about automated program manipula- tion in security-sensitive contexts. We advocate for further ethical safeguards in applying LLMs to such domains.
Chen, M., Tworek, J., Jun, H., Yuan, Q.. Pinto, H. P. Kaplan, J., Edwards, H., Burda, Y., Joseph, N., Brockman, G., et al. Evaluating large language models trained on code. arXiv preprint urXiv:2107.03377 2021.
Several limitations merit attention. Our study covers a lim- ited set of models and Humaneval-derrveed IRs, which may not fully reflect the diversity of real-world IRs. While we employ expert meta-prompts, we do not explore ad- vanced prompting techniques, which could affect perfor- mance. Moreover, though we identify key challenges in IR comprehension, we do not propose concrete solutions such as IR-spccific fine-tuning or dataset augmentation. Fu- ture work should expand datasets, include newer models, and explore targeted adaptation strategies to improve LLM performance on both static and dynamic IR tasks.
Chow, F. Intermediate representation: The increasing sig- nificance of intermediate representations in compilers. Queue, 11 11(10)30-37,, 2013.
Chu, Z.. Wan, Y., Li, Q., Wu, Y., Zhang, H., Sui, Y., Xu, G., and Jin, H. Graph neural networks for vulnerability detection: A counterfactual explanation. In Proceedings of the 33rd ACM SIGSOFT International Symposium on Software Testing and Analysis, pp. 389-401, 2024.
Church, K. W. Word2vec. Natural Language Engineering, 23(1):155-162, 2017.
References
Anju, S., Harmya, P., Jagadeesh, N., and Darsana, R. Mal- ware detection using assembly code and control flow graph optimization. In Proceedings of the 1st Amrita ACM-W Celebration on Women in Computing in India, pp. 1-4. 2010.
Collyer, J., Watson, T., and Phillips, Faser: Binary code similarity search through the use of intermediate represen- tations. arXiv.org, 2023. doi: (0445550/axxi.2310000666
Cummins, C., Fisches, Z. V., Ben-Nun, T., Hoefler, T. O' O'Boyle, M. F., and Leather, H. Programl: A graph- based program representation for data flow analysis and compiler optimizations. In International Conference on Machine Learning, pp. 2244-225, PMLR, 2021.
Anthropic. Claude model, 2024. Available at https: w/www.anttropicc.o com cwoom/ppoouc......
Frsang J., Woodruff, J., Cummins, C., and O'Boyle, M. F. Slade: portable small language model decompiler for optimized assembly. In 2024 IEEE/ACM International Symposium Code Generation and Opti- mization (CGO), pp. 67-80. IEEE, 2024.
Cummins, C., Seeker, V., Grubisic, D., Roziere, B., Gehring, J., Synnaeve, G., and Leather, H. Meta large language model compiler: Foundation models of compiler opti- mization. arXiv preprint arXiv:2407.02524. 2024.
Ben-Nun, T., Jakobovits, A. S., and Hoefler, Neural code comprehension: A learnable representation of code semantics. Advances in neural information processing systems, 31, 2018.
Denkowski, M. and Lavie, A. Meteor universal: Language specific translation evaluation for any target language. In Proceedings of the ninth workshop on statistical machine translation, pp. 376-380, 2014.
Bordes, A., Usunier, N., Gareia-Duran, A., Weston, J., and Yakhnenko, O. Translating embeddings for modeling multi-relational data. Advances in neural information processing systems, 26, 2013.
Devlin, J. Bert: Pre-training of deep bidirectional trans- formers for language understanding. arXiv preprint arXiv:1810 04805, 2018
Devlin, J., Chang, M.-W., Lee, K., and Toutanova, K. Bert: Pre-training of deep bidirectional transformers for lan- guage understanding. arXiv preprint 181000880 2018.
Brauckmann, A., Goens, A., Ertel, S., and Castrillon, Compiler-based graph representations for deep learning models of code. In Proceedings of the 29th Interna- tional Conference on Compiler Construction, pp. 201- 211,201 2020.
Fechete, R., Kienesberger, G., and Blieberger, J. A frame- work for cfg-hased static program analysis of ada pro- grams. Reliable Software Technologies Europe 2008: 13th Ada- aul--Europe International Conference on Re- liable Software Technologies, Venice, Italy, June 16-20. 2008. Proceedings 13, pp. 130- 143. Springer, 2008
Brown, T. B., Mann, B., Ryder, N., Subbiah, M., Kaplan, J., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G.,
Askell, A., Agarwal, S., Herbert- Voss, A., Krueger, G., Henighan, T., Child, R., Ramesh, A., Ziegler, D. M., Wu,
10

---

Can Large Language Models Understand Intermediate Representations in Compilers?
Feng, Z., Guo, D., Tang, D., Duan, N., Feng, X., Gong, M., Shou, L.,, Qin, B., Liu, T., Jiang, D., et al. Code- bert: A pre-trained model for programming and natural languages. arXiv preprint arXiv:2002.08815. 2020.
Lattner, C. and Adve, V. LLVM: compilation framework for lifelong program analysis & transformation. In CGO, CGO '04, 2004.
Li, R., Allal, B., Zi, Y., Muennighoff, N., Kocetkov, D., Mou, C., Marone, M., Akiki, C., Li, J., Chim, J., al. Starcoder: may the source be with you! arXiv preprint arXiv:2305.06161, 2023.
Gui, Y., Wan, Y., Zhang, H., Huang, H., Sui, Y., Xu, G., Shao, Z., and Jin, H. Cross-language binary-source code matching with intermediate representations. In 2022 IEEE International Conference on Software Analy- sis, Evolution and Reengineering (SANER), pp. 601-612. IEEE, 2022.
Lin, C.-Y. Rouge: package for automatic evaluation of summaries, In Text summarization branches out, pp. 74-81, 2004.
Guo, D., Ren, S., Lu, S., Feng, Z., Tang, D., Liu, S., Zhou, L., Duan, N., Svyatkovskiy, A., Fu, S., et al. Graphcode- bert: Pre-training code representations with data flow.
Liu, J., Xia, C. S., Wang, Y., and Zhang. L. Is your code generated by chatgpt really correct? rigorous evaluation of large language models for code generation. Advances in Neural Information Processing Systems, 36:21558- 21572, 2023a.
arXiv preprint arXiv:2009.088 083366, 2020.
Guo, D., Lu, S., Duan, N., Wang, Y., Zhou, M., and Yin, Unixcoder: Unified cross-modal pre-training for code representation. arXiv preprint arXiv:2203.0388 2022.
Liu, P. Yuan, W., Fu, J., Jiang, Z., Hayashi, H., and Neubig, G. Pre-train, prompt, and predict: A systematic survey of prompting methods in natural language processing. ACM Computing Surveys, September 2022. URLhttps. https:
Guo, D., Zhu, Q. Yang, D., Xie, Z., Dong, K., Zhang, W., Chen, G., Bi, X., Wu, Y., Li, Y., et al. Deepseek-coder: When the large language model meets programming the rise code intelligence. arXiv preprint 20Xiv:2401.14196 2024.
/ // //arxiv. org abs 21077 3586.
Liu, P. Yuan, W., Fu, J., Jiang, Z., Hayashi, H., and Neubig, G. Pre-train, prompt, and predict: systematic survey of prompting methods in natural language processing. ACM Computing Surveys, 55(9):1-35, 2023b.
Guo, J., Liu, J., Liu, X., Wan, Y., and Li, L. Summarizing source code with heterogeneous syntax graph and dual position. Information Processing & Management, 60(5): 103415, 2023.
Ma, W., Liu, S., Lin, Z., Wang. W., Hu, Q., Liu, Y., Zhang, C., Nie, L., Li, L., and Liu, Y. Lms: Understanding code syntax and semantics for code analysis. arXiv preprint arXiv:2305.n 12138, 2023.
Han, X., Zhang, Z., Ding, N., Gu, Y., Liu, X., Huo, Y., Qiu, J., Yao, Y., Zhang, A., Zhang, L., et al. Pre-trained models: Past, present and future. Al Open, 2:225 250, 2021.
Ma, W., Liu, S., Zhao M., Xie, X., Wang, W., Hu, Q., Zhang, J., and Liu, Y. Unveiling code pre-trained mod- els: Investigating syntax and semantics capacities. ACM Transactions on Software Engineering and Methodology. 33(7):1-29, 2024.
Jangg B., Xie, Y., Hao, Z., Wang. X., Mallick, T. Su, W. J., Taylor, C. J., and Roth, D. A peek into token bias: Large language models are not yet genuine reasoners. In Pro- ceedings of the 2024 Conference on Empirical Methods in Natural Language Processing (EMNLP), pp. 4722- 4756, November 2024a. Also available as arXiv preprint arXiv:2406.1 1050.
Mao, J., Tang, Z., and Rao, W. Cross-language binary- source code matching based on rust and intermediate representation, 2023.
Jingg H., Zhu, J., Fang, B., Barker, K., Chen, C., Jin, R., and Guan, Q. Happa: modular platform for hpc application resilience analysis with lims embedded. In 2024 43rd International Symposium on Reliable Distributed Systems (SRDS), pp. 51. IEEE, 2024b.
McConnell, C. D. Tree-based code optimization. University of Illimois Urbana-Chunn Chaaaaaanaa 1993.
Mikolov, T. Efficient estimation of word representations in vector space. arXiv preprint 1rXiv:1301.377 2013.
Jiang, H., Zhu, J., Fang, B., Chen, C., and Guan, Q. Investi- gating resilience of loops in hpc programs: A semantic approach with 1lms. In 2024 IEEE High Performance Extreme Computing Conference (HPEC). pp. 1-10. IEEE, 2024c.
Niu, C., Li, C., Ng. V., Lo. D., and Luo, B. Fair: Flow type-aware pre-training of compiler intermediate repre- sentations. In Proceedings of the 46th IEEE/ACM Inter- national Conference on Software Engineering, pp. 1-12, 2024.
Jiang, J., Wang, F., Shen, J., Kim, S., and Kim, S. A survey on large language models for code generation. arXiv preprint 40Xiv:2406.0m 00555 2024d.

---

Can Large Language Models Understand Intermediate Representations in Compilers?
OpenAl. Gpt-4 technical report, 2023. URL https:// openai com/ http:/reeeerrc.ttetreeeeepppee
Tan, H., Luo, Q. Li, J., and Zhang, Y. LIm4decompile: Decompiling binary code with large language models. arXiv preprint arXiv:2403.0522
2024.
Papineni, K., Roukos, S., Ward, T., and Zhu, W.-J. Bleu: method for automatic evaluation of machine transla- tion. In Proceedings of the 40th annual meeting of the Association for Computational Linguistics, pp. 311-318, 2002.
Team, C. Codegemma: Open code models based on gemma. arXiv preprint arXiv:2406. 1409, 2024.
Team, G., Riviere, M., Pathak, S., Sessa, P. G., Hardin, C., Bhupatiraju, S., Hussenot, L.,, Mesnard, T., Shahri- ari, B., Ramé, A., et al. Gemma 2: Improving open language models a practical size. arXiv preprint arXiv:2408.000 000000001110 18, 2024.
Park, J. S., O'Brien, J., Cai, C. Morris, M. R., Liang, P., and Bernstein, M. S. Generative agents: Interactive simulacra of human behavior. In Proceedings of the 36th annual acm symposium on user interface software and technology, pp. 1-22, 2023.
Touvron, H., Lavril, T., Izacard, G., Martinet, X., Lachaux, M.-A., Lacroix, T., Roziere, B., Goyal, N., Hambro, E., Azhar, F., Rodriguez, A., Joulin, A., Grave, E.,, Lample, G., and Jegou, H. Llama: Open and efficient founda- tion language models. arXiv preprint arXiv:2302.13971, 2023.
Paul, I., Luo, J., Glavas, G., and Gurevych, I. Ircoder: Intermediate representations make language models ro- bust multilingual code generators. arXiv.org, 2024. doi: 10.488500/rxiv.2443.oo 234944934
Peng, D., Zheng, S., Li, Y., Ke, G., He, D., and Liu, T.- Y. How could neural networks understand programs? In International Conference on Machine Learning, pp- 8476-8486. PMLR, 2021.
Vaswani, A., Shazeer, N.. Parmar, N., Uszkoreit. J., Jones, L.. Gomez, A. N., Kaiser, E., and Polosukhin, I. At- tention is all you need. Advances in neural information processing systems, 30, 2017.
Pennington, J., Socher, R., and Manning, C. D. Glove: Global vectors for word representation. In Proceedings of the 2014 conference on empirical methods in natural language processing (EMNLP), pp. 1532-1543, 2014.
VenkataKeerthy, S., Aggarwal, R., Jain, S., Desarkar, M. S., Upadrasta, R., and Srikant, Y. Ir2vec: Llvm ir based scalable program embeddings. ACM Transactions Architecture and Code Optimization (TACO), 17(4):1-27, 2020.
Qiu, X., Sun, T., Xu, Y., Shao, Y., Dai, N., and Huang, X. Pre-trained models for natural language processing: A survey. Science China technological sciences, 63(10): 1872-1897, 2020.
Wan, Y., Bi, Z., He, Y., Zhang. J., Zhang, H.,, Sui, Y., Xu. G., Jin, H., and Yu, P. Deep learning for code intelligence: Survey, benchmark and toolkit. ACM Computing Surveys, 56(12):1-41 2024.
Radford, A.,
Wu, J., Child, R., Luan, D., Amodei, D., Sutskever, I., et al. Language models are unsupervised multitask learners. OpenAl blog, 1(8);9, 2019.
Wang. Y., Wang, W., Joty, S., and Hoi, S. C. Codet5: Identifier-aware unified pre-trained encoder-decoder mod- els for code understanding and generation. arXiv preprint 10tiv:22090008300 2021.
Reissmann, N., Meyer, J. C., Bahmann, H., and Själander, M. Rvsdg: An intermediate representation for optimizing compilers. ACM Transactions on Embedded Computing Systems (TECS), 19(6):1 2020.
Wang. Y., Le, H., Gotmare, A. D., Bui, N. D., Li, J., and Hoi, S. C. Codet5+: Open code large language models for code understanding and generation. arXiv preprint arXiv: 230.:::30000.2..... 2023.
Rozière, B. Gehring, J., Gloeckle, F,, Sootla, S., Gat, I., Tan, X. E., Adi, Y., Liu, J., Sauvestre, R., Remez, T., Rapin, J., Kozhevnikov, A., Evtimov, I., Bitton, J., Bhatt, M.. Ferrer, C. C., Grattafiori, A., Xiong, W., Défossez, A., Copet, J., Azhar, F,, Touvron, H., Martin, L.,, Usunier, N., Scialom, T., and Synnaeve, G. Code Ilama: Open foundation models for code, 2024. URL https:// arxiv.orgi abs/ 2200 12950.
Webb, B. J.. Utting. M., and Hayes, I. J. A formal se- mantics of the graalvm intermediate representation. In International Symposium Automated Technology for Verification and Analysis, pp. 111-126. Springer, 2021.
Wei, J., Wang. X., Schuurmans, D., Bosma, M., Xia, F.. Chi, E., Le, Q. V., Zhou, D., et al. Chain-of-thought prompting elicits reasoning in large language models. Advances in neural information processing systems, 35:24824-24888 2022.
Sbirlea, A., Shirako, J., Pouchet, L.-N., and Sarkar, V. Poly- hedral optimizations for data-flow graph language. In International Workshop on Languages and Compilers for Parallel Computing, pp. 57-72. Springer, 2015.
12

---

Can Large Language Models Understand Intermediate Representations in Compilers?
Wong, W. K., Wang, H., Li, Z., Liu, Z., Wang, S., Tang, Q. Nie, S., and Wu, S. Refining decompiled code with large language models. arXiv preprint a3X0v:2310.06530, 2023.
Transuctions Reliability, 2023. doi: 1099 3176922.
Zhao, W. X., Zhou, K., Li, J., Tang, T., Wang, X., Hou, Y., Min, Y., Zhang, B., Zhang, J., Dong, Z., et al. A survey of large language models. arXiv preprint arXiv:2303. 182233 2023.
Wu, Y. Wan, Y., Chu, Z., Zhao, W., Liu, Y., Zhang, H., Shi, X., and Yu, P. S. Can large language models serve as evaluators for code summarization? arXiv preprint arXiv:2412.01333. 2024.
Zheng. Q., Xia, X., Zou, X., Dong, Y., Wang, S., Xue, Y., Wang. Z., Shen, L.., Wang, A., Li, Y., Su, T., Yang. Z., and Tang. J. Codegeex: A pre-trained model for code gener- ation with multilingual benchmarking
Xie, Y., Jiang, B., Mallick, T., Bergerson, J. D., Hutchison, J. K., Verner, D. R., Branham, J., Alexander, M. R., Ross, R. B., Feng, Y., Levy, L., Su, W., and Taylor, C. J. A rag-based multi-agent Ilm system for natural hazard re- silience and adaptation. arXiv preprint arXiv:2504. 17200, April 2025. "As proof of concept, present Wild- fireGPT. ccontenfReferenceeoaiiiee! Jindexx11.
humaneval-x. In Proceedings of the 29th ACM SIGKDD Conference on Knowledge Discovery and Data Mining, pp. 5673-5684, 2023.
Zhou, M., Chen, Liu, Y., Ackah-Arthur, H., Chen, S., Zhang, Q., and Zeng, Z. A method for software vulner- ability detection based on improved control flow graph. Wuhan University Journal of Natural Sciences, 24(2): 149-160, 2019
Yu, Z.,, Zheng, W., Wang, J., Tang, Q. Nie, S., and Wu, S. Codecmr: Cross-modal retrieval for function-level binary source code matching. Advances in Neural Information Processing Systems, 33:3872-3888, 2020.
Zhu, C., Wei, R., Chen, L., Wu, T., Du, G., and Shi, G. Vulnerability localization based on intermediate code rep- resentation and feature fusion. The Computer Journal, 2024. doi: 10. 10.10///comma//xxa0441.
Yuan, D., Fang, S., Zhang, T., Xu, Z., and Luo, X. Java code clone detection by exploiting semantic and syntax information from intermediate code-based graph. IEEE
13

---

Can Large Language Models Understand Intermediate Representations in Compilers?
A. Comprehensive Related Work
LLMs for High-level Programming Languages Advancements in pre-trained LLMs for natural language processing (NLP) have significantly extended to code understanding, enabling models to comprehend high-level programming lan- guages such as Python, C++, and Java. General-purpose models like GPT (Brown et al., 2020), GPT-4 (OpenAI, 2023), Llama (Touvron et al., 2023), and Claude 3 (Anthropie, 2024) have shown strong performance in tasks such code genera- tion, translating natural language descriptions into executable code. Meanwhile, specialized models such as StarCoder (Li et al., 2023), Code Llama (Rozière et al., 2024), DeepSeek- Coder (Guo et al., 2024), and Code Gemma (Team, 2024) have further refined these capabilities, tackling more complex programming tasks and supporting a range of software engineering applications (Zhao et al., 2023).
Pre-trained LLMs have transformed NLP by learning generalizable language representations from large-scale corpora, which can be fine-tuned for diverse downstream tasks (Qiu al., 2020). Early embedding models such as Werd2 (Mikolov, 2013) and GloVe (Pennington et al., 2014) captured semantic similarity but lacked contextual sensitivity (Han et al., 2021). The emergence of transformer-based architectures, exemplified by GPT (Radford et al., 2019) and BERT (Devlin et al., 2018), enabled context-aware modeling, substantially improving performance on language understanding tasks (Vaswani et al., 2017).
These LLMs have been adapted to the programming domain, leading to specialized models such as CodeBERT (Feng et al., 2020), GraphCodeBERT (Guo et al., 2020), UnixCoder (Guo et al., 2022), and CodeT5 (Wang et al., 2021). These models integrate Transformer architectures encode code semanties effectively. Enhancements such as data flow integration in GraphCodeBERT (Guo et al., 2020), multi-mode learning in UnixCoder (Guo et al., 2022), and encoder-decoder frameworks in CodeT5 (Wang et al., 2021) enable improved code comprehension and generation.
IR Representation Learning IR representation learning incorporates structural and flow-based features such as to- ken sequences (Peng et al., 2021), CFGs (VenkataKeerthy et al., 2020; Yu et al., 2020), and control-data flow graphs (CDFGs) (Ben-Nun et al., 2018; Brauckmann et al., 2020; Cummins et al., 2021).
For model architectures, graph neural networks (GNNs) have been widely employed to encode CFG and CDFG structures through pressage-passing techniques (Brauckmann et al., 2020; Cummins et al., 2021; Yu et al., 2020). Other strategies include skip-gram embeddings, such as inst2vec (Ben-Nun et al., 2018), and relation-based embeddings, such as TransE (Bor- des et al.. 2013), trained on CDFGs and CFGs to generate instruction-leww embeddings. However, these models lack task-agnostic pre-trained embeddings, preventing them from capturing contextual information crucial for downstream tasks.
Approaches like IR2Vec (VenkataKeerthy et al., 2020) mitigate this issue by introducing hierarchical vector representations to improve semantic comprehension of IRs. Recent work, such as FAIR (Niu et al., 2024), further refines IR representations using graph Transformers reduce over-smoothing issues while incorporating pre-training tasks that explicitly capture IR token semantics and flow-type information. FAIR has achieved state-of-the the performance across multiple code-related tasks, highlighting the increasing importance of pre-trained IR models.
Meta's LLM Compiler (Cummins et al., 2024) aligns with these efforts, offering pre-trained models for code optimization tasks. While prior work has explored IR representation learning for code optimization and analysis, no studies have systematically examined how L.MSs comprehend IR syntax, CFG structures, execution behavior, and semantic relationships. Our study addresses this gap by providing the first empirical evaluation of LLMs' IR comprehension across these dimensions.
B. Discussion on Dataset Scope and Generalizability
Although the HumanEval dataset consists of relatively short source programs (averaging fewer than 10 lines of Python code), their corresponding LLVM IRs exhibit substantial structural and semantic complexity. We selected this dataset to investigate fundamental challenges in LLM- IR comprehension under controlled conditions. Our rationale is based on the following considerations:
Controlled Complexity. Despite the brevity of HumanEval source programs, their compiled LLVM IRs contain nontrivial structures- -inluurr---inluiiin nested loops, conditional branches, function calls, and memory operations- that typify core IR-level reasoning challenges. This controlled environment allows us to isolate and assess LLMs' ability to understand structural and semantic aspects of IRs.
Task Difficulty Beyond Code Size. The difficulty of IR reasoning does not scale linearly with source code length. Even short programs can produce intricate IR patterns that demand nuanced comprehension of control flow, data dependencies,
14

---

Can Large Language Models Understand Intermediate Representations in Compilers?
and low-level semanties. Our results show that LLMs often fail capture these subtleties, despite the apparent simplicity of the original code.
Evidence of Systemic Deficiencies. Across multiple tasks-includingg CFG reconstruction, DDG extraction, and symbolic execution- we observe consistent failure patterns across various LLMs. These failures suggest fundamental limitations in LLMs' ability to reason over IRs, reflecting broader architectural or training shortcomings rather than artifacts of dataset size.
While we agree that future work should expand to larger and more diverse datasets, our findings provide strong empirical evidence of systemic challenges in IR-level reasoning, even in simplified and controlled scenarios.
C. Prompts
We design our prompts to align with the requirements of each IR task. In particular, we adopt two main prompting strategies:
Task-Specific Expert Prompts. For each task, we assign a specialized role to the model (e.g.. "CFG analyzer for IRs") and clearly define the input, task objective, and expected output format. For example, in Task 1, the output must follow DOT format with explicit node and edge definitions.
Advanced Prompting Techniques. We employ zero-shot and few-shot prompting in Tasks 1-3 to provide context or examples where needed. For Task 4, we further incorporate Chain-of-Thhouge (Co')) prompting encourage step-hy-step execution reasoning.
C.1. Prompt for Structural Understanding: Inferring Control Flow from IRs (Task 1)
Prompt
You are control flow graph analyzer for Intermediate Representations (IRs). will provide you with LLVM Intermediate representation (IRs), a low-level, hatform-indepenn
representation of program.
Here is the IR code input: [IR]
Your task is to generate the control flow graph from the IR. The output format should be a DOT file, including nodes and edges. You do not need to list the content of cach basic block; show each node's title.
Here the IR code example to follow: [IR Example]
The output of the control flow graph should exactly match the following format: [CFG Example]
C.2. Prompt for Syntactic Comprehension: Decompiling IRs to High-Level Code (Task 2)
Prompt
You are expert in high-performance computation. I will provide you with LLVM IRs (Intermediate Representations), which is low-level,
representation of a program.
Here the IR code input: [IR]
Your task is to decompile this IR code into a pure C or C++ source code format that can be run directly.
**Do not add any extra comments, explanations, or characters, and do not use any markdown formatting like *** or "'cpp. **
C.3. Prompt for Semantic Comprehension: Generating Natural Language Deseriptions from IRs (Task 3)
Prompt
I will give you an IR code. Here the IR code input: [TR]
**tpptt Format**; There are [n] functions the code: function[I [I (I in n)] takes [m] inputs: inputl, input2. inputm. Function[[ (I in n)] is [doing semantical function] on inputl. input2.... ..., impuim. and outputs [output].
would like you to summarize the code according to the following specifications: 1.
2. **Tpp and Variable Name Mapping Rules**:- - '% "class. vectorr should be summarized as 'vector;float, numbers'. Other IR-specific types should mapped their equivalent C++ types, following this pattern where possible.
3. **Strict Adherence to Formatting**
The summary should strictly match the format provided below. No additional comments, explanations, or deviations from the format should be added. - Do not use any markdown formatting such as " or "'cpp. Here is the IR code example to follow: [IR Example]
The output should exactly match the following format: [Output Example]
** **Important Instructions*** The summary must not include any additional comments, explanations, or formatting. Ensure that variable names and types are directly transcribed as described in the example. No markdown formatting (e.g., no " or "'cpp) should be used. D*Do not deviate from the specified format under any circumstances, **
15

---

Can Large Language Models Understand Intermediate Representations in Compilers?
C.4. Prompt for Execution Reasoning: Inferring Program Behavior (Task 4)
Prompt
I need your help to analyze whether given assertion passes fails based the provided LLVM IR code for function.
LLVM Code: [
Assertion Statement: [Assertion]
Please write down your thinking process, and list the pass/Tail result of each assertion at the end. NOTE: Make sure the format the pass/fail result of each assertion at the end follow the example: [Output Example]
D. HumanEval Setups
Building prior work (Zheng al., 2023; Tan et al., 2024), we utilize widely recognized benchmarks, specifically HumanEval and its extended test case version, to evaluate the ability of five state-of-the-art LLMSs to comprehend semantics and compare them against various golden baselines. HvmanEval, introduced by OpenAI, is a benchmark designed to assess the multilingual capabilities of code-generative models. It comprises 164 carefully handwritten programming challenges, each featuring a function signature, natural language (NL) description, a function body, and set of unit tests, with an average of 7.7 assertion statements per challenge (Chen al., 2021). In our experiments, these 164 C++ programs serve as the source code for IR analysis.
The compilation experiments were conducted on Dell Workstation equipped with 32 Intel(R) Xeon(R) CPUs E5-2620 v4 @ 2.10GHz, running on an x86-64 architecture with 64-bit system. For these experiments, we used Clang adapted for LLVM 13 on Ubuntu 18.04. The C++ source code were compiled into IRs (.bc files) using the following command:
clang++ -0(Opt-leve] -emit-1lvm (benchmark).com - (benchmark)
We use LLVM's built-in passes generate golden CFGs with the -dotcffg option to ensure accurate and consistent representations of the code structure. The golden code summarization is meticulously performed to ensure that the generated graphs and code representations faithfully reflect the original source code's 's semantics. This process is critical for precise evaluations in subsequent tasks, such as static analysis, code summary analysis, and dynamic analysis. By employing these methodologies, we ensure that the golden datasets are the highest fidelity, providing a solid foundation for assessing the granular understanding and representation of IRs by LLMs.
E. Repetition Results for Stability Analysis
To evaluate the stability of model performance, we repeated each experiment three times (Run 1-Run 3). The results showed minimal variance across runs, demonstrating the robustness of our evaluation. The following tables summarize the performance of DeepSeek R1 across repetitions.
E.1. CFG Construction (Task 1)
Table 6 presents three runs of our CFG reconstruction task by DeepSeek R1 alongside GPT-4's performance. It demonstrates the consistency of our method across repeated trials and highlights performance relative to a state-of-the-art baseline.
Table CFG construction performance across three DeepSeek RI runs and GPT-4
Run
Completed
Node Accuracy
Full Accuracy
Partial Accuracy
Run
69
55
53
2
Run
77
64
57
6
Run
73
62
60
2
GPT-4
164
50
39
11
E.2. IR Decompilation (Task 2)
Table 7 compares decompilation task results over three DeepSeek RI runs, focusing on re-execution correctness. Llama 3.1 included for reference. The data shows slight variation across runs, consistent overall trends.
16

---

Can Large Language Models Understand Intermediate Representations in Compilers?
Table 7. IR decompilation performance three DeepSeck R1 and Llama 3.1.
Model
Completed
Re-execution Completed
Re-execution Success
Run
72
18
Run
77
17
Run
75
39
14
Llama 3.1
77
23
14
E.3. Code Summarization (Task 3)
Table 8 shows the summarization performance of three DeepSeek RI runs across BLEU, ROUGE, and METEOR metrics.
includes both high-quality summary counts (> 0.8) and average scores per run. Llama 3.1 listed for comparison.
Table 8, Code summarization results across three DeepSeek RI runs and Llama 3.1.
Model
Task Comp.
> 0.8 BLEU
ROUGE
METEOR
BLEU
ROUGE
METEOR
Run
49
5
8
0.413
0.637
0.699
Run
49
5
10
0.420
0.639
0.692
Run
49
4
10
0,433
0.640
0.705
Llama 3.1
81
5
11
0.390
0.610
0.670
E.4. Execution Reasoning (Task 4)
Table 9 presents execution reasoning results across three DeepSeek RI runs, covering both IR-based and source-code-based variants. It includes Pass/Partial Pass counts and final pass rates. Llama 3.1 is added for reference.
Table 9. Execution reasoning results IR and source code across three independent DeepSeek RI runs, compared with Llama 3.1.
Model
IR Comp.
SC
Comp.
IR Pass
SC Pass
IR Partial
SC Partial
IR Rate
SC Rate
Run 1
164
164
31
133
133
31
0.189
0.811
Run 2
164
164
30
146
134
18
0.183
0.817
Run 3
164
164
32
139
132
25
0.195
0.804
Llama 3.1
164
164
119
114
35
0.189
0.720
F. Cases of Findings
F.1. Example of CFG construction
Figure illustrates end-to-end example of DeepSeek's Chain-of- Thought (CoT) reasoning for control flow graph (CFG) generation from LLVM IR (with the IR code shown in Figure ??). The model receives LIVM IR code as input, produces an explicit multi-step reasoning trace to identify the control flow structure, and finally outputs the corresponding DOT file representation of the CFG.
F.2 Example of decompilation
Figure 5 shows a re-execution mismatch from GPT-4, The original code searches for a value in the key object and increments out when not found. In contrast, the decompiled version checks only for matching characters two strings, incrementing count instead. This discrepancy likely arises because GPT-4 loses context during loop reconstruction, leading to partial logic and a different outcome during re-execution.
17

---

Can Large Language Models Understand Intermediate Representations in Compilers?
DeepSeek Chain-of- -hhought Reasoning and Output
Step-by-step reasoning:
Okay, so I'm trying to create a control flow graph (CFG) for the given IR code. The code is in LLVM IR format, and the function is named 2155ruucats Sruncaae..uuberr..... Let me break this down step by step.
First, 1 need understand what function does. takes float input and returns a float. The function seems to involve some floating- point to integer conversion and back. Let's look at the instructions:
1. %2 fptosi float %0 132: This converts the input float %0 32-bit integer and stores it in %2.
2. %. sitofp 132 %2 float: Then, this integer %2 converted back float stored in %3.
3. %4 fsub float %0, %3: This subtracts %3 from 960, storing the result in %4.
4. ret float %4: Finally, the result %4 is returned.
Now, create the CFG, need identify the hasic blocks. basic block is sequence of instructions without any control flow changes. In this function, there are no branches or loops, so straight- line sequence instructions. That means there's 's only one basic block.
The node is the entry point of the function, which the beginning of the function where the arguments are prepared. Then, each instruction follows in sequence. The end node the return stalement where function exits.
So, the CFG will have three nodes: the start node, the basic block containing all the instructions, and the end node. The edges will show the flow from start the basic block and then the end.
need to represent this in DOT file format. Each node will be a label, and the edges will be directed arrows showing the flow from one node another.
Putting all together, DOT will have nodes labeled as start, the basic block, and end, with edges connecting them in order.
Generated DOT output:
digraph CFG node (shape-box, style=filled];
start [label="Start]]
basic block_ 1 [label="Basic Block"]; end (label="End");
start -> basic_ block_l 1; basic_block_1 -> end;
This graph shows the sequential execution from the start, through the single basic block, the end.
Figure 4. An example of DeepSeek's step-by-step reasoning and DOT output for CFG generation from LLVM IR.
Original source code
Code from decompilation
for (int 1=0; i<num. length fi++)
for (size_t 1=0; ++1)
iffiind (key. wegy.beginni (), key.end
if 1Lntmp.1ength() & str[i] tmp (il)
(), num != key.end)) out+=1;
count+t;
Figure 5. An example of re-execution mismatch from GPT-4.
18

---

Can Large Language Models Understand Intermediate Representations in Compilers?
F.3. Example of summarization
Figure illustrates the LLVM Intermediate Representation (IR) the truncate nnmbeer function, which extracts the decimal part of floating- point number.
Table 10 presents the summarization of the Example IR generated by various LLMs, showing how each model interprets the function's 's behavior.
LLVM IR Source Code
; ModulcID ..../..//uun. /Bumanumnnnna
httpaa/_wccccmaaakkk666////22222 cppppppppppppp
Source_filenane = "..//.//....... BaaiinnnnXBeccmmaasss_sss 2BAp:////cP_//PP____22
define dso_lecal float L_..1ttruncctteenumeerr (flooffffff (float 80) {
$2 fptosi float $0 to 132
sitofp 132 $2 float
fsub float 80, 83
ret Float 84
Figure 6. An example of LLVM IR.
Table 10. Summarization of example IR by different LLMS.
LLMs
Summarization of Example IR
Ground Truth
There function in the code. Function[1] takes input: a float number. truncates the float number to its integer part and outputs decimal part the floating-pott point number.
GPT-4
There function in the code. Function[1 takes input: float number. It truncates the float number to its integer part and outputs the fractional part float.
GPT-3
There is function in the code. Function[1 I] takes 1 input: float number. It truncates the decimal part of the input number and outputs the truncated value.
Gemma 2
There is 1 function in the code. Function[1 takes input: float number. It truncates the fractional part of the input number and outputs float.
Llama 3.1
There function in the code. Function[[] takes input: float number. truncates the decimal part and outputs the decimal part of the input number. There function in the code. Function[ I] takes inputs: a vector of floats and float threshold. checks if any elements in the vector are closer each other than the threshold and outputs boolean value indicating if such elements exist.
Code Llama
F.4. Example execution reasoning
Fig. 7 presents the source code for execution reasoning. The IR code shown the same as the one used for summarization the previous section.
19

---

Can Large Language Models Understand Intermediate Representations in Compilers?
Table 11. Execution reasoning result of the example from GPT-4.
assert (truncate number (3.5) 5) = 0.5);
Pass
Step 1: Understanding the Function
The function Cruncate_number
takes floating-pont pointtponttt number as input. It uses int () to truncate the decimal part, returning the largest integer smaller than or equal to the input. Then, subtracts this integer from the original number to get the decimal part.
Step Analyzing the Input
The assertion checks if Sruunaatennuber (3.5) equals 0.5. Let's evaluate it step-by-stepy
int (3.5) returns 3
3.5
equals 0.5
Step Comparing with the Assertion
The assertion compares the function output with expected value:
Since truncate _nubeer expects 0 5, the result true.
cnunber (3.5) returns 0 5, and the assertion
Step Conelusion
The assertion passes because the expected value (0. 5) matches the actual output of the function (0 5).
Assertion Result: Pass
Table 12. An example of exccution reasoning result of the C++ example from GPT-4,
assert (truncate number (3.5) 0.5);
Pass
Analysis of the Provided C/C++ Code
The function t truncate_number takes floating- -pintt number as input and returns the decimal part. It uses int () to truncate the number, effec- tively removing the fractional part. The decimal part then calculated as the original number minus the integer part.
Step-by-Step Breakdown
The input number passed to the function.
int (number) truncates the decimal part.
The truncated integer is subtracted from the original number.
The result (decimal part) is returned.
Analysis of the Assertion Statement
The assertion checks whether truncate.number (3.5) equals 0 5. Evaluation steps:
int (3.5) gives 3.
3.5
equals 0 5.
So, the function should return C 5.
Final Result
The actual output matches the expected result.
Assertion Result: Pass
20

---

Can Large Language Models Understand Intermediate Representations in Compilers?
Code from execution reasoning
#inelude <stdio... h> #include <math.h>
2
float truncate_number (float number)
return number (int) number;
Figure An example of source code C++.
F.4.1. MISUNDERSTANDING ALGORITHM LOGIC (45 CASES)
Example (CPP.69) 69);
assert (search ((5, 5, 5, 5, 11) = 1);
Expected behavior: The function should find the greatest integer with a frequency greater than or equal to its value. Example (CPP_12) as shown in Table 13:
assert (solutions) (15, 7, 11) 12)
Expected behavior:
Sum all odd elements at even positions.
LLM failure: Instead of following the logic step-by-step, the model guesses the sum based on pattern recognition from prior examples rather than executing the correct loop iteration. LLM failure: The model fails to track frequency accumulation and instead returns an incorrect value, misunderstanding how the search function should operate.
F.4.2. PATTERN BASEDD GUESSING (35 CASES)
Example (CPP 121) shown in Table 13:
assert (solutions ((5, 8, 7, 1)) == 12)
Sum all odd elements at even positions. LLM failure: Instead of following the logic step-by-step, the model guesses the sum based on pattern recognition from prior examples rather than executing the correct loop iteration.
Expected behavior:
F.4.3. CONTROL FLOw ISSUES (28 CASES)
Example (CPP.92)
assert (any_ime int (2.5, 2, 3) false)
Expected behavior: The function should correctly handle nested conditionals to determine if any sum of two numbers equals the third.
LLM failure: The model does not properly evaluate the if-else branching and incorrectly processes non non-integer values.
F.4.4. OVERRELIANCE ON FUNCTION NAMES (22 CASES)
Example (CPP_105)
assert (issame (by lbygthht 1, 1, 4, 5, 8, 2, 31) ("Eight", "Five", "Four", "Three", "Two", "Two", "One", "One")));
Sort numbers, reverse the list, and replace them with corresponding word representations. LLM failure: Instead of applying the transformation rules, the model simply assumes the function operates based on its name and produces incorrect output.
Expected behavior:
21

---

Can Large Language Models Understand Intermediate Representations in Compilers?
Table 13. An example of guessing result hased on patterns from GPT-4.
assert (solutions (5, B, 7, 1) == 12);
Fail
the function 'solutions' takes pointer to 'std :: vector < int >> its argument. the vector is expected to contain integers.
Step 1: function signature
the function uses "getelementptr" and 'load' instructions to access the elements of the vector. the vector accessed way that suggests iterating over its elements.
Step loading vector elements
the function performs several operations on the elements of the vector, including loading elements, performing shifts, and using the 'srem" (signed remainder) operation.
Step vector operations
the function also uses vector reduction operation add. vdd.v))))d which indicates that it sums up the elements of the vector in groups of four.
(
the function appears to summing the elements the vector, but it also applies some conditions based on the results the 'srem' opera- tions.
Step understanding the logic
the 'srem' operation checks if the elements are odd (i.c., remainder when divided by 2 is 1) and conditionally includes them in the sum.
the input vector '5, 8, 7, 1' contains the integers 5, 8, 7, and 1. the odd numbers in this vector are 5, and 1. the even number
Step 5: input values
the of all elements is '5 8 7 21
however, since the function seems apply filter based the "srem" operation, need to check which elements are included in the final sum.
Step 6: calculating the expected result
the 'srem' operation checks if the remainder is 1, which means only the odd numbers (5, 7, and 1) would included in the sum.
therefore, the sum of the odd numbers '5 + 7 13'.
the assertion checks if the result of 'solutions(5, 8, 7, 1) equals 12. based on our analysis, the function would return 13, not 12.
Step 7: final comparison
F.4.5. STRING PROCESSING ERRORS (18 CASES)
Example (CPP_1L) 112):
assert (issame (reverse_delete ("abode", "ae") ("bod", "False")))
Expected behavior:
Remove characters and check for palindromicity.
LLM failure: The model fails correctly apply character deletions and check the reverse consistency, leading to incorrect assertions.
F.5. Conclusion
These findings highlight common failure patterns in LLM-based code interpretation, including logical misinterpretation, heuristic-based assumptions, and issues with numerical computations. Further refinement in model training and debugging processes recommended.
22


Reviews:
[{"review_id": "P26LKR10V5", "reviewer": "Reviewer_k42A", "summary": "The authors explored the capabilities of large language models (LLMs) in understanding intermediate representations (IRs), primarily for applications such as code comprehension, optimization, and automated reasoning. Their findings indicate that while LLMs are proficient in understanding static IR features and basic control flow structures, they struggle with more complex representations, such as loop reasoning and execution simulation. Additionally, LLMs perform better at capturing semantic-level behavior rather than instruction-level details.\n\nTheir findings are highly significant for the broader research community in the field of LLMs, particularly in the domain of software analysis and handling IR-related tasks.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you very much for your detailed review and positive feedback on our work. We greatly appreciate your recognition of the novelty and significance of our study, as well as your thorough evaluation of our experimental setup. We would like to address your comments as follows:\n\n\n**W1:** The study does not consider fine-tuning on a benchmark-specific dataset, which limits its ability to reflect true benchmarking metrics.\n\n**AW1:** We appreciate the reviewer’s comment. Our study is primarily exploratory, positioned to assess the untuned performance of current LLMs (including zero/few-shot and chain-of-thought prompting) in understanding intermediate representations (IRs) without relying on benchmark-specific fine-tuning. This approach enables us to use tasks such as CFG reconstruction, IR decompilation, code summarization, and execution reasoning as benchmarks for evaluating LLMs' understanding of intermediate representations. Moreover, our findings reveal inherent limitations in how these models understand and process IRs.\nWe acknowledge that fine-tuning LLMs on specialized datasets could boost their performance on IR-related downstream tasks. However, by focusing on their untuned performance, our work establishes a clear baseline that exposes key challenges. Our findings pinpoint areas where current LLMs fall short—such as control flow comprehension (Finding 1), granular semantic understanding (Findings 4–5), and loop handling—while also offering guidance for targeted fine-tuning strategies. We are committed to pursuing fine-tuning as a central focus of our future research, leveraging our insights to significantly enhance models' abilities to tackle more complex IR-related tasks.\n\n**Typos and Formatting Issues**\nWe appreciate you highlighting some spelling and formatting errors. We have reviewed the manuscript carefully and will correct these issues in the final version.\n\nAgain, thank you for your constructive comments.", "claims_and_evidence": "The claims and research findings are highly interesting and novel, supported by an extensive experimental setup and validated with convincing results. The findings are realistic and impressive.", "methods_and_evaluation_criteria": "Yes, the proposed methods, target study tasks/categories, and corresponding evaluation criteria are well-defined and clearly presented. The research is thorough and well-structured, providing an in-depth validation of the authors' claims and findings. The authors benchmarked the performance of selected but widely used LLMs, including GPT-3/4, Gemma 2, LLaMA 3.1, and Code Llama, on key IR-related tasks: (a) CFG construction, (b) decompilation ability, (c) code summarization, and (d) execution reasoning. Their findings effectively highlight the relative strengths and limitations of these LLMs for the aforementioned tasks.", "theoretical_claims": "Yes, I have already elaborated on this in the \"Methods and Evaluation Criteria\" section. However, I also considered the research limitations highlighted by the authors. They acknowledged that their study is constrained by a limited benchmark dataset, as the HumanEval-derived IRs do not fully reflect data diversity. Additionally, the impact on model performance due to the lack of exploration into advanced prompting techniques and the omission of fine-tuning strategies—such as IR-specific dataset augmentation and fine-tuning—was not accounted for.", "experimental_designs_or_analyses": "I already mentioned and discussed in sections like \"Methods And Evaluation Criteria*\" and \"Theoretical Claims*\".", "relation_to_broader_scientific_literature": "Their findings are highly significant for the broader research community in the field of LLMs, particularly in the domain of software analysis and handling complex IR-related tasks.", "essential_references_not_discussed": "NA", "other_strengths_and_weaknesses": "Strength:\n1. The manuscript is well-written, thoroughly analyzed, and presents a detailed experimental setup to highlight the shortcomings of LLMs in reasoning and understanding complex IRs.\n2. It provides a comprehensive benchmarking of widely used LLMs, including GPT-3/4, Gemma 2, LLaMA 3.1, and Code Llama, on key IR-related tasks: (a) CFG construction, (b) decompilation ability, (c) code summarization, and (d) execution reasoning.\n\nWeakness:\n\nThe study does not consider fine-tuning on a benchmark-specific dataset, which limits its ability to reflect true benchmarking metrics. This omission is a drawback, as it does not fully showcase the potential capabilities of LLMs, particularly in the domain of software analysis.", "other_comments_or_suggestions": "several typos have been noticed,", "questions_for_authors": "Please see Weakness and other comments", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {"ethical_review_concerns": "NA"}}, {"review_id": "CCLlHyYVaF", "reviewer": "Reviewer_k99i", "summary": "The paper provides an empirical evaluation of current LLMs on IR understanding tasks, namely -- \n\n- CFG reconstruction\n- decompilation\n- code summarization, and \n- execution reasoning\n\nand find that models struggle with complex reasoning about IRs", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely thank the reviewer for the thoughtful and constructive feedback. We appreciate your recognition of the novelty of applying LLMs to compiler IRs and the value of our in-depth evaluation. Below, we address the key concerns:\n\n**Q1:** Can you provide more details on the level of prompting?\n\n**A1:** In our paper, we have adopted two prompting strategies: **(1) Task-Specific Expert Prompts.**  For each IR task, prompts assign a role (e.g., “CFG analyzer for IRs”) and clearly define the input, task objective, and required output format (e.g., DOT files with nodes and edges); **(2) Advanced Prompting Techniques.** We also applied the *Zero/Few-Shot Prompting* and *Chain-of-Thought (CoT) Prompting* in Tasks 1–3 and Task 4, respectively.\n\nNote that the main text summarizes our approach, and complete prompt templates are provided in Appendix B (extended examples omitted for brevity).\n\n---\n\n**Q2:** Can you provide the statistics of the IR humaneval programs?\n\n**A2:** Key statistics of 164 IR programs across optimization levels are as follows:\n| OPT Level | LoC | Tokens | Functions | Loops | Basic Blocks |\n|:-:|:-:|:-:|:-:|:-:|:-:|\n|   -O0     |    162,389    | 1,079,290 |   5,195   |  463  |   13,228     |\n|   -O1     |     66,598    |  454,954  |    393    |  577  |    7,708     |\n|   -O2     |     69,102    |  478,002  |    374    |  625  |    8,048     |\n|   -O3     |     75,519    |  526,395  |    367    |  827  |    8,917     |\n\nAt -O0, minimal optimizations yield verbose IR with duplicate functions to preserve debugging (‘linkonce_odr’). -O1 and -O2 remove redundant elements, reducing size but slightly increasing loop counts, while -O3 further simplifies the structure at the cost of more loops. These trends explain why decompilation performs best at -O1/O2. We will include a detailed description and a summary table in the revised manuscript, along with data-driven explanations to substantiate our findings.\n\n---\n\n**W1:** Choice of HumanEval problems used for constructing tasks likely challenges the generalizability.\n\n**AW1:** We acknowledge that the HumanEval dataset may limit generalizability. However, our results reveal “general limitations” in LLMs’ IR comprehension that extend beyond dataset scale or diversity from two aspects: **(1) Controlled Complexity:**  Although the programs are small, the extracted IRs involve common operations—including nested loops, conditional statements, and function calls—that pose significant challenges; **(2) Systemic Failures:** Consistent errors across tasks, such as CFG reconstruction and execution reasoning, indicate systemic deficiencies in IR understanding. \n\nThese issues represent fundamental “general challenges” that LLMs have yet to overcome for most programs. Once these preliminary obstacles are addressed, we plan to expand our evaluations to larger, more diverse datasets to continuously enhance generalizability. We will also discuss future directions for incorporating more diverse IR samples in the revised manuscript.\n\n\n---\n\n**W2:**  It is unclear how to calibrate the findings once we develop models explicitly optimized for IR programs\n\n**AW2:** We agree that fine-tuning on IR-specific datasets may improve performance. Our study is primarily exploratory, positioned to assess the untuned performance of current LLMs in understanding IRs without relying on benchmark-specific fine-tuning. We acknowledge that fine-tuning LLMs on specialized datasets could boost their performance on IR-related downstream tasks. However, by focusing on their untuned performance, our work establishes a clear baseline that exposes key challenges. Our findings pinpoint areas where current LLMs fall short—such as control flow comprehension (Finding 1), granular semantic understanding (Findings 4–5), and loop handling—while also offering guidance for targeted fine-tuning strategies. We are committed to pursuing fine-tuning as a central focus of our future research, leveraging our insights to significantly enhance models' abilities to tackle more complex IR-related tasks.\n\n---\n\n**W3**: The related work for intermediate/assembly language is well provided; however, a rich body of work about code understanding is not discussed.\n\n**AW3:** We appreciate the suggestion to contextualize our work further within the broader code understanding literature. Although our manuscript comprehensively covers IR-level work, we agree that expanding the discussion of static/dynamic analysis methods (e.g., CRUXEval [2] and CFG path analysis in [3]) would enrich our related work section. In the revised manuscript, we will incorporate these references to better position our study within the broader landscape of code understanding research.\n\n[1] Chris C., et al. LLM Compiler: Foundation Language Models for Compiler Optimization. CC’25\n\n[2] CRUXEval: Code Reasoning, Understanding, and Execution Evaluation\n\n[3] LLMs: Understanding Code Syntax and Semantics for Code Analysis", "claims_and_evidence": "- Pioneering empirical study to investigate the capabilities of LLMs -- first work to evaluate LLMs on IR related tasks\n- Empirical findings\n    - LLMs recognize syntax but struggle with control flow and semantics\n    - Loop handling remains a fundamental challenge\n\nThese are demonstrated with comprehensive evaluations on the 4 tasks.", "methods_and_evaluation_criteria": "They propose new benchmarks (derived from compiling humaneval problems to IRs) on four diverse tasks to evaluate IR capabilities of models. The tasks are not necessarily novel and have been studied for programs in high-level language, but are new in the context of LLVM code. \n\nThe benchmark is collected from HumanEval problems, which might not provide insights toward understanding real-world programs. Particularly, given a challenge with IR / Assembly programs is terseness of such programs, it is unclear if those effects are accounted for here.", "theoretical_claims": "none", "experimental_designs_or_analyses": "- The experimental design and evaluation are well presented. Each task is described clearly with appropriate metrics and prompts.\n\n- Given IR understanding can be seen as an OOD task, it would be useful to further enhance the description about the prompting effort that was applied across different models. \n\n- Reasoning models (such as O1 or R1) are not discussed. Given considerable performance gains witnessed on programming tasks, it deserves analysis in this paper.", "relation_to_broader_scientific_literature": "The paper contextualizes the research with relevant work in LLMs for intermediate/assembly language.", "essential_references_not_discussed": "The related work for intermediate/assembly language is well provided, however, a rich body of work about code understanding is not discussed. For example [1] introduced code execution evaluations, which has been followed up by many other works attempting to use different static and dynamic approaches to evaluate code understanding (including cfg path analysis in [2])\n\n[1] CRUXEval: Code Reasoning, Understanding, and Execution Evaluation\n[2] LLMs: Understanding Code Syntax and Semantics for Code Analysis", "other_strengths_and_weaknesses": "Strength.\n- The paper is well written and the findings are easy to understand. The recommendations for improving IR reasoning capabilities would also be useful for future work. \n\nWeakness.\n- Choice of HumanEval problems used for constructing tasks likely challenges generalizability of the findings.\n- Current LLMs are likely not heavily optimized for IR programs. It is unclear how to calibrate the findings once we develop models explicitly optimized for IR programs. It is possible that the findings would considerably change for such models.", "other_comments_or_suggestions": "none", "questions_for_authors": "- Can you provide more details on the level of prompting required for models to work with IR programs?\n\n- Can you provide the statistics of the IR humaneval programs?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "HYiYyLvhcM", "reviewer": "Reviewer_2sD2", "summary": "The paper experiments with applying LLMs to the control flow graph of programming code, identifying key challenges of control flow, semantic understanding, and loop handling. These challenges, as analyzed through 4 tasks, seem to permeate over a variety of language models including Coda Llama, Gemma 2, and GPT-4. The paper claims to be the first to analyze how LLMs perform on intermediate representations of the compiler.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for the thoughtful feedback and recognition of our novel application of LLMs to compiler IRs. We also appreciate the acknowledgement of  our focus on task-specific analysis. We address the key concerns below.\n\n**Q1 & W1:** Are there any non-LLM approaches that would be applicable to the introduced tasks, e.g., fine-tuning or training a classifier on top of IR2vec / Bert-style models and/or applying graph representation learning methods? Or is there any particular reason that those are not included/applicable\\\n&\\\n**W3:** The paper does not include non-LLM baselines.\n\n**A1:** We appreciate the reviewer’s point that established non-LLM methods can serve as valuable baselines. For instance, pre-trained models such as FAIR [1] have shown impressive results in semantic summarization. However, while non-LLM approaches (e.g., GNNs, fine-tuned BERT-style models) excel on specific tasks, they require extensive dataset-specific training and are typically applied to higher-level applications.\n\nIn contrast, our study focuses on the raw, untuned IR comprehension of general-purpose LLMs. By using tasks such as CFG reconstruction, IR decompilation, code summarization, and execution reasoning, we establish diagnostic benchmarks that evaluate the zero/few-shot and chain-of-thought (CoT) prompting capabilities of current LLMs without any task-specific fine-tuning. This approach not only reveals the inherent limitations of existing LLMs in understanding intermediate representations but also provides actionable insights for future improvements.\n\nMoreover, as detailed in **A2**, we will incorporate additional CoT prompting in the revised manuscript. In future work, we plan to explore non-LLM baselines and a broader range of prompting techniques to further enhance our analysis.\n\n\n\n\n---\n\n**W2 & W3:** Evaluation is limited to zero-shot experiments; few-shot and basic CoT prompting have not been explored.\n\n**A2:** Our evaluation is not strictly zero-shot. We use zero-shot prompting for CFG Construction and IR Decompilation, few-shot examples for Code Summarization, and chain-of-thought prompting for Execution Reasoning. Although these techniques improve performance, fundamental challenges in IR understanding—particularly in control flow and execution semantics—persist, with model rankings remaining unchanged.\nTo assess the benefits of chain-of-thought prompting, we conducted supplementary experiments using DeepSeek R1. We selected DeepSeek R1 because (1) it is among the state-of-the-art LLMs, and (2) it features an inherent chain-of-thought mechanism that decomposes complex IR tasks into intermediate reasoning steps, showing potential advantages for our IR tasks.\n\nWe ran DeepSeek R1 on all four tasks three times to ensure stability. Our preliminary results (one as an example) are as follows:\n\nT1: CFG Construction\n\n|       | Comp. | Node Acc. | Full Accu. | Partial Accu. |\n|:-:|:-:|:-:|:-:|:-:|\n| R2 |     77     |     **64**   |     57     |       6       |\n| GPT-4 |     164    |     50    |     39     |       11      |\n\nT2: IR Decompilation\n\n|       | Comp. | Re-exe. Comp. | Re-exe. Success |\n|:-:|:-:|:-:|:-:|\n| R1 |       72       |           36           |          18         |\n|LLaMA 3.1|    77      |          23            |          14         |\n\nT3: Code Summarization\n\n|           | Task Comp. | BLEU > 0.8 | METEOR > 0.8 | ROUGE > 0.8 |  Avg. BLEU |  Avg. ROUGE |  Avg. METEOR |\n|:-:|:-:|:-:|:-:|:-:|:-:|:-:|:-:|\n|   R1   |     49     |   1   |   5   |    8   | 0.413 | 0.637 |  0.699 |\n| LLaMA 3.1 |     81     |   1   |   5   |   11   |  0.39 |  0.61 |  0.67  |\n\nT4: Execution Reasoning\n\n|       | IR Completed |SC Completed | IR Pass | SC Pass | IR Partial Pass |SC Partial Pass | IR Pass rate | SC Pass Rate |\n|:-:|:-:|:-:|:----:|:-:|:-:|:-:|:-:|:-:|\n| R1 |       164      |       164      |  31  |  133 |      133     |      31      |   0.189   |   0.811   |\n|LLaMA 3.1 |       164      |       164      |  31  |  119 |      114     |      35      |   0.189   |    0.72   |\n\nDeepSeek R1 performs comparably to LLaMA 3.1 on T2–T4 but excels in T1 (CFG Construction) due to its integrated chain-of-thought mechanism that helps identify basic blocks and critical control flow instructions (e.g., \"br\" and \"jmp\"). \n\n**Revised Finding 1:** LLMs generally struggle with detecting basic blocks and constructing control flow edges; however, chain-of-thought prompting modestly enhances the recognition of key control flow instructions.\n\n**In the final version, we will include comprehensive analyses, additional experiments, and concrete examples to support these findings.** In future work, we will systematically explore additional prompting techniques to further validate and extend these findings.\n\n[1] Niu, Changan, et al. \"Fair: flow type-aware pre-training of compiler intermediate representations.\" ICSE’24.", "claims_and_evidence": "The main claim is negative, that LLMs struggle with CFG-based tasks. This negative claim receives support from the results, with some limitations (as described below) -- as common with negative claims -- where a comprehensive study of all possibilities is difficult to bring forward. It is difficult to put these results that the LLMs currently achieve into context without seeing some non-LLM baselines for these specific tasks.", "methods_and_evaluation_criteria": "The dataset is a processed version of a subset of MMLU, which is an established benchmark for evaluating LLMs. The tasks around the CFGs are newly designed and seem to make sense. There are technically no new proposed methods, but a new application of LLMs to these newly designed tasks.", "theoretical_claims": "Not applicable.", "experimental_designs_or_analyses": "The experimental design seems to be appropriate for the studied research question of how LLMs would perform on CFG-related tasks.", "relation_to_broader_scientific_literature": "The key contributions match with the general trend of applying LLMs to a variety of problems. The paper covers a particular topic within applying LLMs to code.", "essential_references_not_discussed": "None that I'm aware of, yet it is not well argued why methods from the related work were not considered as baselines.", "other_strengths_and_weaknesses": "## Strengths\n\n- The paper provides an in-depth analysis of applying LLMs to control flow graphs\n- The importance of analyzing control flow graphs is nicely motivated.\n- Specific findings are emphasized and justified by the results.\n- Applying LLM to the intermediate representations of a compiler is, to the best of my knowledge, a novel application of LLMs\n\n## Weaknesses\n\n- The paper doesn't consider any non-LLM baselines, e.g. graph neural nets (as those mentioned in the related work) that would be arguably are more natural fit for the task. Even if LLMs currently don't compete with GNNs and other graph representation learning methods, it would be interesting to see how big the margin is.\n- Evaluation is limited to zero-shot experiments. There are no few-shot experiments and no basic CoT prompting was tried. Both these are fairly established as standard techniques and could very well influence the conclusions.", "other_comments_or_suggestions": "- The paper could be strengthened by efforts to improve the basic LLM performance with well-established basic prompting techniques (e.g., chain-of-thought, few-shot prompting)\n- The paper could be improved by adding task-specific baselines.", "questions_for_authors": "Are there any non-LLM approaches that would be applicable to the introduced tasks, e.g., fine-tuning or training a classifier on top of IR2vec / Bert-style models and/or applying graph representation learning methods? Or is there any particular reason that those are not included/applicable?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "kkDfsU0XaI", "reviewer": "Reviewer_ua9h", "summary": "This paper presents an empirical study of the capability of LLMs to understand intermediate representations (IRs) of code. The LLMs are evaluated on 4 types of tasks of IR understanding: control-flow graph (CFG) reconstruction, IR decompilation, code summarization and execution reasoning. The results indicate that while LLMs are capable of parsing IR syntax and recognizing high-level structures in code tasks, they struggle with control flow reasoning, execution semantics, and loop handling.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely thank the reviewer for the thoughtful and constructive feedback. Below, we address the key concerns:\n\n**W1:** The paper only uses the HumanEval, which contains code functions with an average of fewer than 10 lines.\n\n**A1:** Though HumanEval consists of relatively short functions, our results reveal fundamental limitations in LLMs’ IR comprehension that extend beyond dataset scale from three aspects: **(1) Controlled Complexity:** Even small programs require precise reasoning about IR constructs; **(2) Task Difficulty Beyond Code Size:** Despite their brevity, small programs exhibit complex low-level semantics.; **(3) Systemic Failures:** Consistent errors across tasks—such as CFG Construction and Execution Reasoning—indicate systemic deficiencies in IR understanding.\n\n---\n\n**W2:** The paper evaluates GPT4, Gemma, and so on. More SOTA LLMs, such as GPT-4o, DeepSeek-V3 (or R1), should be included.\\\n&\\\n**W4:** It is not specified whether the results are obtained in multiple turns.\n\n**A2:** We agree that including SOTA models is essential. We supplemented our experiments with DeepSeek R1.\n\n**A4:** We fully agree that multi-turn evaluations using metrics like pass@k enhance robustness. We ran DeepSeek R1 on all four tasks three times (R1–R3) to ensure stability.\n\nT1: CFG Construction\n\n|       | Comp. | Node Acc. | Full Accu. | Partial Accu. |\n|:-:|:-:|:-:|:-:|:-:|\n| R1 |     69     |     **55**    |     53     |       2       |\n| R2 |     77     |     **64**    |     57     |       6       |\n| R3 |     73     |     **62**    |     60     |       2       |\n| GPT-4 |     164    |     50    |     39     |       11      |\n\nT2: IR Decompilation\n\n|       | Comp. | Re-exe. Comp. | Re-exe. Success |\n|:-:|:-:|:-:|:-:|\n| R1 |       72       |           36           |          18         |\n| R2 |       77       |           38           |          17         |\n| R3 |       75       |           39           |          14         |\n|LLaMA 3.1|    77      |          23            |          14         |\n\nT3: Code Summarization\n\n|           | Task Comp. | BLEU > 0.8 | METEOR > 0.8 | ROUGE > 0.8 |  Avg. BLEU |  Avg. ROUGE |  Avg. METEOR |\n|:-:|:-:|:-:|:-:|:-:|:-:|:-:|:-:|\n|   R1   |     49     |   1   |   5   |    8   | 0.413 | 0.637 |  0.699 |\n|   R2   |     49     |   1   |   5   |   10   | 0.420 | 0.639 |  0.692 |\n|   R3   |     49     |   1   |   4   |   10   | 0.433 | 0.640 |  0.705 |\n| LLaMA 3.1 |     81     |   1   |   5   |   11   |  0.39 |  0.61 |  0.67  |\n\nT4: Execution Reasoning\n\n|       | IR Com. |SC Com. | IR Pass | SC Pass | IR Partial Pass |SC Partial Pass | IR Pass % | SC Pass % |\n|:-:|:-:|:-:|:----:|:-:|:-:|:-:|:-:|:-:|\n| R1 |       164      |       164      |  31  |  133 |      133     |      31      |   18.9   |   81.1   |\n| R2 |       164      |       164      |  30  |  146 |      134     |      18      |   18.3   |   81.7   |\n| R3 |       164      |       164      |  32  |  139 |      132     |      25      |   19.5   |   80.4   |\n|LLaMA 3.1 |       164      |       164      |  31  |  119 |      114     |      35      |   18.9   |    72.0   |\n\nDeepSeek R1 performs comparably to LLaMA 3.1 on T2–T4 but excels in T1 (CFG Construction) due to its integrated chain-of-thought mechanism that helps identify basic blocks and critical control flow instructions (e.g., \"br\" and \"jmp\"). \n\n**Revised Finding 1:** LLMs generally struggle with detecting basic blocks and constructing control flow edges; however, chain-of-thought prompting modestly enhances the recognition of key control flow instructions.\n\n**A4:** Notably, the three runs yielded consistent results, demonstrating the reliability of DeepSeek R1’s performance. We will include these multi-turn results in the revised manuscript.\n\n**In the final version, we will include comprehensive analyses, additional experiments, and concrete examples to support these findings.** In future work, we will systematically explore additional prompting techniques to further validate and extend these findings.\n\n---\n\n**W3:** There exist non-neural developed tools for IR tasks in common software testing applications that could help highlight the dis/advantages of LLM-based methods.\n\n**A3:** We appreciate the reviewer’s point that established non-neural methods serve as valuable baselines. Tools such as Ghidra, various IDE tools, and symbolic execution engines like KLEE excel at CFG construction, decompilation, and execution reasoning through extensive domain-specific training. In contrast, our study uses these tasks solely as diagnostic benchmarks to evaluate the raw, untuned IR comprehension of LLMs. Our goal is not to outperform specialized tools but to reveal intrinsic LLM limitations—such as in control flow inference, granular semantic understanding, and loop handling—that can guide future optimizations. In the final manuscript, we will include a detailed discussion comparing our results with these non-neural baselines.\n\n---\n\nW4 & A4 are included in A2.", "claims_and_evidence": "The paper provides experiments to support the claims, by using HumanEval dataset and five LLMs.", "methods_and_evaluation_criteria": "The method is only evaluated on HumanEval dataset. This dataset is at the code function level with the average code lines of less than 10. \n\nThe paper investigates GPT-4, GPT-3, Gemma 2, LLaMA 3.1, and Code Llama, in understanding IRs. Since the capabilities of LLMs are evolving rapidly, SOTA LLMs should be used for evaluation.", "theoretical_claims": "This paper is an empirical study. There is no proof of theoretical claims.", "experimental_designs_or_analyses": "The designs and analysis of experiments make sense for all 4 types of tasks, demonstrating the LLMs’ capability of understanding IRs.\n\nThe evaluation of LLMs is usually conducted in multiple turns, and the results are often presented with statistical metrics such as pass@k, while whether the results in this paper are obtained in multiple turns is not specified.", "relation_to_broader_scientific_literature": "This paper lists multiple relevant papers and analysis their differences. For example, Meta’s LLM Compiler offers pre-trained models for code optimization tasks. While prior work has explored IR representation learning for code optimization and analysis, no studies have systematically examined how LLMs comprehend IR syntax, CFG structures, execution behavior, and semantic relationships. This paper addresses this gap by providing the first empirical evaluation of LLMs’ IR comprehension across these dimensions.", "essential_references_not_discussed": "No.", "other_strengths_and_weaknesses": "Strengths:\n1. This paper presents a empirical study to investigate the capabilities of LLMs, including GPT-4, GPT-3, Gemma 2, LLaMA 3.1, and Code Llama, in understanding IRs.\n2. The IR comprehension of LLMs is analyzed across four tasks: Control Flow Graph (CFG) reconstruction, decompilation, code summarization, and execution reasoning.\n\nWeaknesses:\n1. The paper only uses HumanEval dataset, which is at code functional levels with fewer average lines.\n2. The paper evaluates the capabilities of LLMs, including GPT-4, GPT-3, Gemma 2, LLaMA 3.1, and Code Llama. More SOTA LLMs should be included, such as GPT-4o, DeepSeek-V3 or R1, Qwen.\n3. There are non-neural developed tools for these tasks, many of them integrated in commonly-used IDEs or software testing applications. If the performances of these tools can be listed together, they could show the advantages and disadvantages of LLM-based methods.\n4. The evaluation of LLMs is usually conducted in multiple turns, and the results are often presented with statistical metrics such as pass@k, while whether the results in this paper are obtained in multiple turns is not specified.", "other_comments_or_suggestions": "No.", "questions_for_authors": "Please refer to the weaknesses.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: This paper presents an empirical study to investigate how well LLMs can understand Intermediate Representations (IRs) by evaluating them on tasks such as control flow graph reconstruction, decompilation, code summarization, and execution reasoning. The main results show that the evaluated LLMs can parse IRs and recognize high level structures, but struggle with more complex instruction level reasoning especially for branch and loop handling, as well as control flow reasoning and execution semantics. All reviewers found the study to be interesting and well executed, and important contribution to the research community. There were some concerns around limited number of SOTA LLMs (including reasoning ones) being evaluated as well as only using HumanEval problems potentially limiting the applicability to more real-world coding usecases. The response with additional experiments with R1 helped with some of these concerns. It would be great to include the discussions and experiments from rebuttal response to the final version of the paper.
Decision: Accept (poster)