PERFOGRAPH: A Numerical Aware Program Graph Representation for Performance Optimization and Program Analysis
Authors: Ali TehraniJamsaz
Quazi Ishtiaque Mahmud
Le Chen
Nesreen K. Ahmed
Ali Jannesari
Venue: NeurIPS 2023 poster (2023)
Keywords: program representation
graph representation
program analysis
graph neural networks
performance optimization

Abstract: The remarkable growth and significant success of machine learning have expanded its applications into programming languages and program analysis. However, a key challenge in adopting the latest machine learning methods is the representation of programming languages which has a direct impact on the ability of machine learning methods to reason about programs. The absence of numerical awareness, aggregate data structure information, and improper way of presenting variables in previous representation works have limited their performances.  To overcome the limitations and challenges of current program representations, we propose a novel graph-based program representation called PERFOGRAPH. PERFOGRAPH can capture numerical information and the aggregate data structure by introducing new nodes and edges. Furthermore, we propose an adapted embedding method to incorporate numerical awareness.
These enhancements make PERFOGRAPH a highly flexible and scalable representation that can effectively capture programs' intricate dependencies and semantics. Consequently, it serves as a powerful tool for various applications such as program analysis, performance optimization, and parallelism discovery. Our experimental results demonstrate that PERFOGRAPH outperforms existing representations and sets new state-of-the-art results by reducing the error rate by 7.4% (AMD dataset) and 10% (NVIDIA dataset) in the well-known Device Mapping challenge. It also sets new state-of-the-art results in various performance optimization tasks like Parallelism Discovery and Numa and Prefetchers Configuration prediction.

PERFOGRAPH: A Numerical Aware Program Graph Representation for Performance Optimization and Program Analysis
Ali TehraniJamsazz" Quazi Ishtiaque Mahmud Le Chen', Nesreen K. Ahmed?, Ali Jannesari Iowa State University, Ames, Iowa, USA (tehrani, mahmud, lechen, jannesari @iastate.edu 2 Intel Labs, Santa Clara, CA, USA @ intel.com
Abstract
The remarkable growth and significant success of machine learning have expanded its applications into programming languages and program analysis. However, a key challenge in adopting the latest machine learning methods is the representa- tion of programming languages, which directly impacts the ability of machine learning methods to reason about programs. The absence of numerical awareness, aggregate data structure information, and improper way of presenting variables in previous representation works have limited their performances. To overcome propose the limitations and challenges of current program representations, a graph-basee based program representation called PERFOGRAPH. PERFOGRAPH can capture numerical information and the aggregate data structure by introducing new nodes and edges. Furthermore, we propose an adapted embedding method incorporate numerical awareness. These enhancements make PERFOGRAPH a highly flexible and scalable representation that effectively captures programs' intricate dependencies and semanties. Consequently, it serves powerful tool for various applications such as program analysis, performance optimization, and parallelism discovery. Our experimental results demonstrate that PERFOGRAPHI outperforms existing representations and sets new state-of-the-art results by re- ducing the error rate by 7.4% (AMD dataset) and 10% (NVIDIA dataset) in the well-known Device Mapping challenge. It also sets new state-of saaee-of-teeaaee the-hth-rtt results in various performance optimization tasks like Parallelism Discovery and NUMA and Prefetchers Configuration prediction.
Introduction
recent years, the remarkable success of machine learning has led to transformative advancements across numerous fields, including compiler optimization and program analysis. The applications include compiler heuristics prediction, optimization decisions, parallelism detection, etc. [4, [5].
The training process generally involves feeding program data input and transforming it into representation suitable for machine learning models. The selection of program representation is erucial, it can significantly impact the performance of the machine learning model [12]. With the development of graph neural networks (GNNs), an increasing number of graph representations of programs have been incorporated into GNN models for program analysis 3 28 [0] One of the pioneering efforts in developing a comprehensive graph representation for programs is PRO- GRAMI [14. PROGRAML incorporates control, data, and call dependencies integral components of a program's 's representation. In contrast to prior sequential learning systems for code, PROGRAMI.
Equal contribution.
37th Conference on Neural Information Processing Systems (NcurIPS 2023).

---

closely resembles the intermediate representations used by compilers, and the propagation of infor- mation through these graphs mimics the behavior of typical iterative data-flow analyses. Despite the success that PROGRAM has achieved, there are shortcomings in this current state-of-the-art program representation, especially for performance-oriente downstream tasks. These limitations stem from neglecting numerical values available at compile time and the inadequate representation of aggregate data types.
In this paper, we present PERFOGRAPII to address the limitations of the current state-of-the-art program representation. Additionally, we propose a novel way to embed numbers in programs in an elegant way so that our DL model will not face unknown numbers during inference time. Our experiments demonstrate that PERFOGRAPH sets new state-of-the-an results in numerous downstream tasks. For example, in the Device Mapping downstream task, PERFOGRAPH yield error rates as low as 6% and 10% depending on the target hardware. Moreover, PERFOGRAPE even outperforms the tools and models specially designed for specific tasks such as parallelism discovery.
Overall, the main contributions of this paper are:
An enhanced compiler and language-agnostic program representation based PROGRAML that represents programs as graphs.
The proposed representation supports aggregate data types and provides numerical awareness, making it highly effective for performance optimization tasks.
Evaluation of the proposed representation on common downstream tasks and exceeding the performance of PROGRAML.
Quantification of the proposed approach on new set of downstream tasks such as parallelism discovery and configuration of NUMA systems.
The rest of the paper is structured as follows: Section
2 presents the related works. In section 3 we provide motivational example, showing the limitations of PRoGRAMI, the state-of-the-ard program representation. This section is followed by section 4 where we present our proposed representation PERFOGRAPH along with the novel way of embedding numerical values. In section S experimental results on downstream tasks are provided, and finally, Section concludes the paper and discusses some future works.
Related Works
Machine learning has brought significant advancements in many fields, and program analysis and software engineering are no exceptions. However, Machine Learning (ML) and Deep Learning (DL) models can not directly process raw source code reason about programs. Therefore, researchers have explored different approaches to represent applications in a format suitable for DL models. Generally, there are three types of commonly used program presentations: sequence of tokens, Abstract Syntax Tree (AST), and Intermediate Representation (IR).
The initial attempts [6 21 27 represented source code sequence of tokens, such as identifiers, variable names, or operators. This approach intuitively treats programming languages similarly to natural languages. It allows for the utilization of advanced natural language process (NLP) techniques, such as large language models [19 22 23 However, this token-based representation overlooks the inherent dependency information within the program's 's structure. It fails to capture the unique relationships and dependencies between different elements of the code, which can limit its effectiveness in tasks such compiler optimization and code optimization.
Sequence of tokens:
AST: An AST represents the structure is constructed based on the syntactic rules of the programming language and provides a high-level abstraction of the code. Previous works have leveraged ASTs as inputs to tree-based models for various code analysis tasks like software defect prediction [17] and code semantic study [9]. Moreover, there have been efforts to augment ASTs into graphs that incorporate program analysis flows such as control flow and data flow. These AST-based graph representations capture more comprehensive code dependency information and have shown superior results compared to traditional approaches in previous works 3
program by capturing its hierarchical organization.
IR: IR is an intermediate step between the source code and the machine code generated by a compiler. Previous work [39 has utilized IR to train an encoding infrastructure for representing programs as a distributed embedding in continuous space. augments the Symbolic encodings with the flow of information to capture the syntax well as the semantics of the input programs. However, it

---

generates embedding at the program or function level and also requires data-flow analysis type for generating the embedding. contrast, our approach derives embedding from the representation and works at the more fine-grained instruction level. More recent works 66 have leveraged IR-based graph representation to better capture essential program information, such as control flow, data flow, and dependencies. However, despite their success, IR- graph representations have certain limitations. For example, these representations may not be numeric-aware or may lack the ability adequately represent aggregate data types. In this work, we propose PERFOGRAPII, graph representation based on IR, to address these limitations.
14 g
3 Motivation
As stated in the related work section, program representations based the intermediate representation are very effective in enabling DL models to automate the process of various optimizations, One such representation is PROGRAML, whose performance surpasses other code representations, making it state-of-the art in various optimizations and downstream tasks. However, despite its potential, it suffers from several limitations. To name a few: it incapable of properly carrying out information regarding read and write operations to the memory location, has no support for aggregate data types, and discards numerical values. Listing I shows code snippet where 3-dimensional array is defined. Figure T shows the PROGRAMIL representation of this code snippet. For illustration purposes, instruction nodes and control flow edges are shown blue, whereas nodes, and data flow edges. Green edges show the call graph. As it can be seen, PROGRAME fails to represent some critical information. For instance, code float arr [2] [3] [4] is converted to LLVM IR [2 [3 x [4 float]]]*, which is used to construct node in PROGRAML. It eliminates the aggregate data structure information, like the array's dimension. Leaving up to the DL model infer the meaning behind the numbers in [2 [3 [4 float]]] Moreover, in this representation, only the type of numbers (c.g.. int8, float) are considered, and the actual values of the numbers of PROGRAMM in applications where numerical values play an important role. A numerically aware representation help understand and optimize operations involving numeric data types, constants, and expressions. There are also some anomalies in the way temporary variables are depicted. For example, in [
represents variable, constant
given attention. The absence of numerical awareness limits the performance
the fourth alloca node allocates memory for variable, and store instructions are applied on two separate nodes representing the variable. Thus, the information about the first store instruction
carried out properly when the second store instruction happens. In the following section, we will see how PERFOGRAPE APII effectively addresses many limitations in the current state-of- the-the-art program representation. PERFOGRAPH uses PROGRAML representation as its initial graph and reconstructs the graphs by addressing the aforementioned limitations.
1 #include <stdio
2
3
int main (int arge char sargy (I)
4 int arr arr_state 0;
5 float arr [2][3][[]
Lf [[100:.1.0...OF.....00
6
=
7
(2.0f.2.0f.20ff.2.000
8
[3.0f.3.0f.3.0ff.33
9
[44.0f.4.0f.4.0f.44
11
(6.0f.6.0f.6.0f.66o001
12
arr state
return 0;
Listing 1: C++ code example.
Figure 1: ProGraML representation of Listing
4 PERFOGRAPHE A fined-grained numerical aware graph representation
PERFOGRAPH is graph representation based on LLVM IR. It is built top of PROGRA MLI; however, does not suffer from the limitations that the PROGRAME has, helping DL models to reason over the complex structure of the programs and enabling them to make more accurate optimization decisions,

---

especially in terms of performance optimization. Figure 2 shows how various enhancements and improvements are applied to construct more precise representation. Consider simple code example of defining a variable and increasing it by one (int i = 0; i++;). Figure 2a shows the PROGRAMI representation of this code example. It has two store nodes, as one responsible for storing O and the other one storing the incremented value of i. In the following subsection, we will explain how PERFOGRAPH constructed by addressing the limitations shown in Figure 2a
wllncw
Atare
Toai
D01
A32
Store
store
store
(a) Initial graph
(b) Local variables
(c) store nodes
(d) Numbers
Figure 2: PERFOGRAPIC addresses the existing limitations program representation.
4.1
Representing Local Identifiers and store instruction
Local identifiers' names are preceded by % LLVM Intermediate representation. Memory allocation the stack is done by alloca instruction. One of the limitations of the current state-of-the-art program representation, PROGRAML, is that it is unable to carry out information regarding the operations that happen memory location. For instance, in Figure 2a the two store nodes represent storing values of O and 1 to variable i. However, as shown, each store instruction node is connected network to reason over the operations that happen to a memory location. For the embedding vector of the second store node in 2a to represent the fact that some information regarding the variable i has changed, one has to increase the number of GNN layers to 3 to support up to 3 hops when propagating the messages GNN. This can potentially limit the ability of the GNN model if there are greater number of hops between the two store nodes shown in Figure 2a To address this limitation, instead of having more than one variable node (oval-shape nodes) per identifier, PERFOGRAP! APH only considers one variable node in its graph representation. Any load or store instruction will refer to the same variable node. These changes are shown in Figure 2b We see that the store nodes in Figure access the same memory location, thus representing the fact that those store instructions are modifying the same memory location.
Local Identifiers:
separate variable node, making it difficult for the graph neural
Store instruction: LLVM uses store instruction to write into memory. store instruction has two arguments: value store and the address to which it will store the value. PROGRAML differentiates between these two arguments by adding position feature to the edges shown in Figure 2a However, since the store instruction modifies the contents at the corresponding memory address, we posit that it is better to reflect the fact the content of the identifier has changed. To present this information, PERFOGRA APH adds an extra edge from the store node to node representing the identifier whose value is modified by the store instruction. Figure 2c shows these changes in the graph constructed by PERFOGRAPES
Numbers: Numbers can be a significant factor in optimization decisions. For example, they can show the loop bound, and different optimizations can be considered depending on the loop bound. PERFOGRAPH, unlike PROGRAMM. not only considers the type of numbers such as 132, 164, float but also the actual values of the numbers. As illustrated in Figure 2d numerical constant nodes have the actual value of the number in their feature set in addition to the type of the number. Even though numerical constant nodes have the value of the number as one of their features, there a need to embed the numbers in way that unknown numbers will not be seen in the inference. Unlike other tokens, numbers are harder embed as an infinite amount of numbers exists, and to handle all ranges of numbers, need to have very large vocabulary set.

---

41.25
4.2 Numerical Awareness
However. with a very large vocabulary size, the DL mod- els may still encounter numbers in the inference phase that they have not seen the training phase. We propose novel way of embedding numbers called Digit Embedding, Figuref]showss our approach. To embed number, we first break down the number to its digits: then, we consider a position for each one of the digits. The goal is let DI. models realize the place value of each digit. Then, each digit and its corresponding position are embedded and summed together. Therefore, will have embedding representing the information about the digits and their po- sitions. For instance, in Figure 3 we embed each digit and its corresponding position with an output dimension of 3. Since the number has four digits, the results would be vector/tensor of size 4 x 3. To make sure the Digit Em- bedding of numbers has the same length across numbers with varying sizes of digits, we apply an aggregation function over the embedding dimension. Since the output embedding dimension is three in this example, we would have vector of length three representing the number after aggregation. The aggregation function can be of any type (Max, Mean, etc.).
Digit::
4
1
5
Poetioon::
2
-2
Dight omb:
Posiiions emb:
Digi Postioon
Aggregate
emb:
Figure 3: Overview of the digit embed- ding.
4.3 Aggregate Data Types
Aggregate data types, such arrays and vectors, are an essential part of applications. They play an impor- tant role in many applications, such as matrix multipli- cations. Thus, presenting these data types helps the DL models better understand programs. Current LLVM IR- based program representations fail to present aggregate data types appropriately. For example, consider three- dimensional integer array, In LLVM IR, this array is shown 3 [2 x [3 1 As can be seen, the length of the arrays and their data types are infer- able. However, without proper representation, the DL model's capacities will be spent on learning these de-
allocs
1Bx
call
(a) support for aggregate data types.
terministic facts (i.e., the length of the arrays and their type). PERFOGRAPH considers aggregate data types a new node type in its representations. Figure 4b shows how aggregate data types supported by PERFOGRAPH. Unlike other LLVM IR-bd representations, PERFO- GRAPH supports multi-dimensional arrays and vectors. PERFOGRAPH creates a chain of nodes to present the different dimensions of the arrays. In Figure 4a see there is a node representing the ftree-dimensional array [3 [2 x [3 132]]]* PERFOGRAPHI breaks down the corresponding node into three (since is a three- dimensional array) white nodes as shown in Figure 4b Then, each node has a context representing that specific dimension of the array. For example, the context for the third dimension is [3 x i32], whereas for the second dimension, the context [2 [3 x 132]]. For each aggregate type node, in addition to the context of the node, we specifically add the length of the array and its type as additional features. For aggregate data types whose lengths are not known during compile time, we follow the LLVM conventions by considering the length of those data types as vscale. These enhancements will help the DL models to reason over the dimensions and types of aggregate data types. As result, PERFOGRAPEE will ultimately enable the DI models to have more accurate predictions for applications that deal with arrays and vectors.
X1274
12
call
(b) Break-down of aggregate data types.
Figure PERFOGRAPH supports aggre- gate data types.
5

---

Experimental Results and Downstream Tasks
In this section, we evaluate PERFOGRAS APH on six downstream tasks. For each downstream task, we will explain the task itself, the dataset, and the baselines.
5.1 Experimental Setup
In our experiments, we use DGL's 40 RGCN 37 implementation for PERFOGRAPH representation. The graphs from PERFOGRAPH are treated as heterogeneous and managed via the HeteroGraphCon. module. We use hardware setup of two 18-core Intel Skylake 6140 CPUs and two NVIDIA Tesla V100-32GB GPUs. The embedding space for numbers is generated by extracting digits and positions from numeric token of IR statement, then passed and position embeddings. These are combined for the final numeric token embedding. Non-numerie tokens directly go through the PyTorch embedding layer. Each PERFOGRAPI heterogeneous node converts to 120-dimensional vector via this embedding. We use the Adam 266 Optimizer, relu [ activation function, a learning rate of 0.01, and hidden_di __dim parameter of 60. Mean aggregation applied to combine different node-type results before linear classification layer, which outputs probability distribution for each class. The class with the highest probability the prediction.
PyTorch [31] embedding layer for digit
5.2 Device Mapping
Problem Definition: We apply PERFOGRAPH to the challenging heterogeneous device map- ping 13 problem. In this task, there are number of OpenCL kernels that we need to predict which accelerator (CPU or GPU) yields higher performance. We compare PER- FOGRAPA against DeepTune [3), Inst2Vec (6), and PROGRAML [14.. The results of the base- lines quoted from [1]]
DresTunc
ProGroML
PERFOGRAPH
201
Dataset: For this task, we use the dataset pub-
Figure 5: Performance comparison the device mapping task with state-of-the-ar models [lower is better].
lished in 13 In this dataset, there are 256 OpenCL kernels available, and 680 LLVM instances are extracted from them. There are
two types of GPUs: AMD and NVIDIA. For each of the GPUs, the runtimes the kernels are recorded in the dataset. For AMD, 276 kernels show better performance in GPU, while 395 kernels show better performance in CPU. Whereas for NVIDIA, 385 kernels have better runtimes with GPU, and 286 kernels have better runtimes with CPU. We consider this as a binary CPU GPU classification problem.
Results: As the dataset is small, we use the same 10-fold validation (with 80% training, 10% validation, and 10% testing) like PROGRAML [4] and chose the model with the highest validation accuracy. The hand-crafted features of 20 are also used graph-level features in our model to enhance the performance following the approach [14. Table I and 2 show the final precision, call, fl-score, and accuracy for AMD and NVIDIA devices. Figure [ PERFOGRAPH with wate-of-the-ar models on the Device Mapping dataset. We can see that PERFOGRA APH sets new state-of-the-art results by achieving the lowest error rate among the baselines both for AMD and NVIDIA, indicating the effectiveness of PERFOGRAPH.
Table 1: PERFOGRAPE results for AMD devices.
Table 2: PERFOGR APH results for NVIDIA devices.
Precision
Recall Fi-score Accuracy
Precision
Recall
F1-score
Accuracy
CPU
0.94
0.94
0.94
CPU GPU
0.87
0,90
0.89
0.94
0.90
GPU
0.94
0.94
0.94
0.92
0.89
0.90
5.3
Parallelism Discovery
Problem Definition: In this problem, given sequential loop, we try to predict whether a loop can be executed in parallel. We treat this problem as binary classification problem with two classes: Parallel and Mor-Pallll

---

The OMP_ Serral dataset [ is used for this task. It contains around 6k compilable source files with Parallel and Non-Paralles loops. The training dataset contains around 30k IR files. The @MP_Serria dataset has three test subsets to compare the performance with three traditional parallelism assistant tools: Pluto (4032 IR files), AutoPar (3356 files), and DiscoPoP (1226 IR files).
Dataset:
Results: We evaluate PERFOGRAPH on all three subsets and compare with traditional rule-based tools: Pluto [7]. AutoPar 33 DiscoPoP [29]. and also Deep Learning based tools: Graph2Par [T] PROGRAML. Table 3 shows the results. The results of Pluto and Graph2par are reported from [ As PROGRAML does not have this downstream task in their paper, we used the PROGRAML representation in our pipeline to generate the results. Results show that traditional rule-based tools have the highest precision but the lowest accuracy because those tools are overly conservative while predicting parallel loops. So, they miss out on a lot of parallelism opportunities. PERFOGR APH achieves considerably good precision scores across all the test subsets. In terms of accuracy, PERFO- GRAPH surpasses the current state-of-the-art approaches by 2% in the Pluto and AutoPar subset. In the DiscoPoP subset, it achieves an impressive 99% accuracy and surpasses PROGRAML by 9%.
Table 3: Performance comparison of PERFOGRAPHI on the DMP_Seriae dataset.
Subset
Approach
Precision
Recall
Fi-score
Accuracy
Pluto
0.39
0.56
0.39
Graph2par
0.88
0.93
0.91
0.86
Pluto
PROGRAML
0.88
0.88
0.87
0.89
PERFOGRAPH
0.91
0.90
0.89
0.91
AutoPar
0.14
0.25
0.38
Graph2par
0.90
0.79
0.84
0.80
autoPar
PROGRAML
0.92
0.69
0.67
0.84
PERFOGRAPH
0,85
0.91
0.85
0.86
DiscoPoP
0.54
0.70
0.63
Graph2par
0.90
0.79
0.84
0.81
DiscoPoP
PROGRAML
0.92
0.94
0.92
0.91
PERFOGRAPH
0.99
0.99
0.99
5.4 Parallel Pattern Detection
Problem Definition: Parallel loops often follow some specific patterns. Identifying parallel patterns important because it helps developers understand how to parallelize specific program since each parallel pattern needs to be treated differently. As result, we apply PERFOGRAPH to identify potential parallel patterns in sequentially written programs. Only the three most common parallel patterns are considered: Do-all (Private), Reduction, and Stencil 34). Given loop, the task is to predict the pattern.
For this experiment, we also use the OMP Serrill dataset 11 This dataset contains source codes of different parallel patterns. These programs are collected from well-known benchmarks like NAS Parallel Benchmark (24), PolyBench (32), BOTS benchmark [18] and the Starbench benchmark 5.. Then, template programming packages like Jinja (35 are used to create synthetic programs from the templates collected from the mentioned benchmarks. The dataset contains 200 Do-all (Private), 200 Reduction, and 300 Stencil loops.
Dataset:
We used 80% of the dataset for training and 20% for testing. Table represents our findings. The results of Pragformer (25) and Graph2par [ are reported from [T We compare with these two approaches as they are specifically developed for solving this problem. For generating the results with PROGRAML, we used the PROGRAML representation in our pipeline. We can see PERFOGRAPE APH achieves an impressive 99% accuracy on the -Mria Parallel Pattern dataset. It surpasses the state-of-the-a to capture the syntactic and structural patterns embedded into source programs. From Table 4 we can also see PERFOGRAPH has high precision for all three patterns and achieves a high precision score for Do-all and Steneil patterns while maintaining very good accuracy.
Results:
PROGRAML. model by 3%. This indicates the strength of PERFOGR APH
7

---

Table 4: Performance comparison for the parallel pattern detection task with PERFOGRAPH on the
OMP Serial Dataset.
Approach
Pattern
Precision
Recall
Fl-score
Accuracy
Do-all
0.86
0.85
0.86
Pragformer
Reduction
0.89
0.87
0.87
0.86
Stencil
N/A
N/A
N/A
Do-all
0.88
0.87
0.87
Graph2Par
Reduction
0.9
0.89
0.91
0.9
Stencil
N/A
N/A
N/A
Do-all
0.92
0,90
0.91
PROGRAML
Reduction
0.92
0.92
0.92
0.96
Stencil
0.98
1
0.99
Do-all
0.97
0.99
PERFOGRAPH
Reduction Stencil
0.97
1
0.99
0.99
1
5.5 NUMA and Prefetchers Configuration Prediction
An appropriate configuration of Non-Uniform Memory Access (NUMA) and hardware prefetchers significantly impacts program performance. In this experiment, we define the task of NUMA and prefetcher selection as predicting the right configuration within given tuning parameter search space. We evaluate the performance of both PROGRAML and PERFOGRAPII for this task by converting each program in the dataset to PROGRAML and PERFOGRAPH graphs following the approach in (38].
Problem Definition:
ProGraML
PERFOGRAPH1
ProGraML
PERFOGRAP1
0.3
0.
0.0
29g
2
3
2V3
Fold
Fold
(a) Error distribution for Sandy Bridge architecture.
(b) Error distribution for Skylake architecture.
Figure Breakdown of the NUMA and prefetchers configuration prediction per fold [lower better).
Dataset: We use the dataset in 38 which includes a diverse set of intermediate representation files coupled with the optimal configuration [6). The dataset incorporates various LLVM compiler optimization flags to produce different forms of the same program. There are 57 unique kernels (IR files) in this dataset, and around 1000 optimization flags are applied, resulting in 57000 IR files in total. Each IR file within the dataset is accompanied by its runtime two architectures, Sandy Bridge and Skylake, across thirteen different NUMA and prefetcher configurations.
Results: Following the approach in the study of TehraniJamsaz et al., we partition the dataset into folds for cross-validation. Figure 6a and 6b illustrate the performance results in terms of error rates. On average, PERFOGRAPH outperforms PROGRAML by achieving 3.5% and 1.8% better error rates on average for the Sandy Bridge and Skylake architecture, respectively. These improvements demonstrate the effectiveness of PERFOGRA APH compared to the state-of-the-art PROGRAMI.
5.6 Thread Coarsening Factor (TCF) Prediction
Problem Definition: Thread coarsening an optimization technique for parallel programs by fusing the operation of two more threads together. The number of threads that can be fused together known as the Thread Coarsening Factor (TCF). For given program, the task to predict the coarsening factor value (1, 2, 4, 8. 16, 32) that leads to the best runtime. The run- ning time with coarsening factor used as the baseline for calculating speedups. For this task, we compare PERFOGRAPII against DeepTune (13), Inst2Vec [6] and PROGRAML [14]. The results of the baselines are quoted from [6]. Since PROGRAML has not been evaluated on this task in the past, apply PROGRAML representation in our setup for comparison.

---

Kernels with correct TCF predicted
We use the dataset of Ben-Nun et al. [3]. The dataset contains only 17 OpenCL ker- nels. For each kernel, the dataset has the runtime information on four different GPUs for the dif- ferent thread coarsening factor values. Hence, for each kernel, we have the runtime correspond- ing to each thread coarsening factor value on specific GPU device.
Dataset:
Radwwn HD 5500
Talell 7970
GTX 460
Tests K20e
Results: we design the problem as a multi-class classification problem where, given kernel, we try to predict which thread coarsening factor provides the highest performance. As the dataset very small, we apply a 17-fold eross-van voss-validation approach. In each fold, we train our model on 16 data points, and the model is tested on the one unseen data point that is left out of the training set. 7 the comparison of kernels with the correct Thread Coarsening Factor (TCF) found by PROGRAML and PERFOGRAPIL Across the four platforms in total TERFOGRAPIL PROGRAML is able to find only 9 cases. In two of the platforms (AMD Radeon HD 5900 and NVIDIA GTX 480) where PROGRAML failed to find any kernel with the correct TCF, PERFOGRAPH find three kernels in both of the platforms with the correct TCF value. As shown in even though PERFOGRAPE outperforms PROGRAML on most computing platforms, it falls behind inst2vec. We posit the reason is that inst2vec has pretraining phase where it is trained using kip-gram. On the other hand, 17 kernels are very small. Therefore, a DI.-based model is not able to generalize enough. that even smaller dataset, PERFOGRAPH achieved comparable speedups with respect to the current mate-of-the-aas models.
Figure 7: Correct TCF found by PROGRAML PERFOGRAPII [higher is better].
able to correctly predict the TCF for 17 cases, whereas
However,
Table 5: Speedups achieved by coarsening threads
Computing Platform Deep'Tune inst2vec PROGraML PERFOGRAPH
AMD Radeon HD 5900
1.1
1.37
1.15
1.19
AMD Tahiti 7970
1.05
1.1
1.00
1.14
NVIDIA GTX 480
1.1
1.07
0.98
1.03
NVIDIA Tesla K20c
0.99
1.06
1.01
5.7 Algorithm Classification
Problem Definition:
Previous downstream tasks showed that in most of the cases, PERFOGRAPH We go further by applying
outperforms the baselines. Those tasks were mostly performance-orient PERFOGRAPH on a different downstream task, which is algorithm classification. The task involves classifying a source code into of 104 classes. In this task, we compare the results of PERFOGRAPHI to those of inst2vec, PROGRAML. The results of the baselines are quoted from [4].
Dataset: We the POJ- 104 dataset (30 in similar setup
14 that contains around 240k IR
files for training and 10k files for testing.
Results: For this task, inst2vec has error rate of 5.17, whereas PROGRAML has eπror rate of 3.38. PERFOGRAPH APH yields an error rate of 5.00, which better than inst2vec and slightly behind
PROGRAML. One of the reasons is that PROGRAML already has very small error rate in this task, leaving very small gap for improvement; however still PERFOGRAPH'S result is very close to that of PROGRAML. We could applied PROGR A ML in our setup, the error rate of PROGRAME was 6.00. Moreover, we posit that for algorithm classification, numbers are not significant factor. Therefore, numerical awareness can confuse the models a little bit. However, this experiment shows that PERFOGRAPH very close PROGRAML's performance this task and shows the applicability of PERFOGRAPH to wider range of downstream tasks.
reproduce the results in PROGRAML paper in our setup. When
5.8 Ablation Study
We further analyzed how each one of the enhancements in PERFOGRAPE affects the end results. We performed an ablation study on the Device Mapping task and trained our GNN models on variations of PERFOGRAPH.

---

Results without aggregate data type nodes: First, aggregate data type nodes are removed from PERFOGRAPH representation. Please note that in this setup, the Digit Embedding still applied. 7nd shows the results. It can be seen that when the representation does not support aggregate data type, the error rate increases 13% in AMD and 15% in the NVIDIA dataset. This clearly indicates that having aggregate-type nodes in the representation helped the model to learn the code features more accurately.
Results without Digit Embedding: Then Digit Embedding is removed from the pipeline for the second experiment and the aggregate data type nodes are kept. Table and can see that unlike having aggregate data type nodes, removing digit embedding does not hurt the error rate that much for the task of device mapping. However, we can still see a small increase (1.1%) in the error rate for the AMD dataset. For the NVIDIA dataset, the error rate increases from 10.0 to 10.6%.
and 7 shows the results. We
Table 6: Summarizing PERFOGRAPH results for AMD device.
Approach
Error (56)
DeepTune 13
28.1 1
inst2vee
6
19.7
PROGRAMI.
14
13.4
PERFOGRAPH (without aggregate data type nodes)
13.0
PERFOGRAPH (without digit embedding)
7.1
PERFOGRAPH (aggregate data type nodes digit embedding)
6.0
Table 7: Summarizing PERFOGRAPHI results for NVIDIA device.
Approach
Error (%)
Duep B3
39.0
inst2vec G
21.5
PROGRAML
14
20.0
PERFOGRAPH (without aggregate data type nodes)
15.0
PERFOGRAPH (without digit embedding)
10.6
PERFOGRAPH (aggregate data type nodes + digit embedding)
10.0
Finally, we can conclude that both components in our representation helped the model to learn the code features better to some extent. However, aggregate data type nodes in the embedding helped our model more than Digit Embedding for the task of device mapping. The reason can be that there are not many numbers in the dataset. However, in tasks where there many numbers, Digit Embedding can play a significant role.
6 Conclusion and Future Work
In this paper, presented PERFOGRAPH, LLVM IR-based graph representation of programs that supports aggregate data types such as arrays and vectors and is numerical aware. Moreover, addresses several limitations of the previous IR-based graph representations. PERFOGRAPH evaluated on various downstream tasks, and experimental results indicate that PERFOGRAPII is indeed effective, outperforming state-of-the-art in most of the downstream tasks. PERFOGRAPII numerical awareness capability is limited to the numerical values that are available at the compile time. For future work, we intend augment our representation by adding support for dynamic information and checking the possibility of integrating hardware performance counters with our representation. Moreover, we plan develop pre-trained embedding model using our representation. Having pre-trained model will help to solve the problem of limited training samples in some downstream tasks.
Acknowledgement
This project funded by NSF (#2211982) and Intel Labs. We would like to thank them for their generous support. Additionally, we extend our gratitude to the Research IT tean of lowa
"ntpss//researchit..k las iastate.edu/
10

---

State University for their continuous support in providing access to HPC clusters for conducting the experiments of this research project.
References
[I] A. F. Agarap, Deep learning using rectified linear units (relu), arXiv preprint 1rXiv:1833.0S3 2018.
[2] Allamanis. Graph neural networks in program analysis. Gruph Neural Networks: Foundations, Frontiers, and Applications, pages 483-497, 2022.
[3] Allamanis, M. Brockschmidt, and Khademi. Learning to represent programs with graphs. arXiv preprint xrXiv:1111000400, 2017.
[4] Allamanis, E, Barr, Devanbu, C. Sutton. A survey of machine learning big code and naturalness. ACM Computing Surveys (CSUR), (-3) 2018.
[5] Andersch, Juurlink, and Chi, henchmark suite evaluating parallel programming models, In Proceedings of Workshop on Parallel Systems and Algorithms (PARS), volume 28, pages 1-6, 2013.
[6] T. Ben-Nun, A. Jakobovits, and T. Hoefler. Neural code comprchension: A learnable representation of code semantics. Advances in Neural Information Processing Systems, 31, 2018.
[7] U. Bondhugula, A. Hartono, J. Ramanujam, and P. Sadayappan. practical automatic polyhedral parallelizer and locality optimizer. SIGPLAN Not., 43(6))101-113 jun 2008. ISSN 0362-1340. doi: 10. (0.1145/1379022.153 137559. httpss////do
1145/1379022 1375595
[8] A. Brauckmann, A. Goens, S. Ertel, and J. Castrillon. Compiler- Doasiler-based graph representations for deep learning models of code. In Proceedings of the 29th International Conference on Compiler Construction. pages 201-211, 1, 2020.
[9] L. Chen, W. Ye, and S. Zhang. Capturing source code semantics via tree- rre-bbsedd convolution over api- enhanced ast. In Proceedings of the 16th ACM International Conference on Computing Frontiers, pages 174-182, 2019.
[10] L. Chen, Q. Mahmud, and A. Jannesari. Multi-view learning parallelism discovery sequential programs. In 2022 IEEE International Parallel and Distributed Processing Symposium Workshops (IPDPSW), pages 295-303, IEEE, 2022.
[11] L. Chen, Q. L. Mahmud, H. Phan, N. Ahmed, and A. Jannesari. Learning to parallelize with openmp by augmented heterogeneous ast representation. Proceedings of Machine Learning and Systems, 5, 2023.
[12] Z. Chen and M. Monperrus. arXiv: I (004.0306,
literature study of embeddings on source code. arXiv preprint
[13] C. Cummins, P. Petoumenos, Z. Wang, and H. Leather. End-to-end deep learning of optimization heuristies. In 2017 26th International Conference on Parallel Architectures and Compilation Techniques (PACT). pages 219-232. IEEE, 2017.
[14] C. Cummins, Z. V. Fisches, T. Ben-Nun, T. Hoefler, and H. Leather. Programl: Graph-based deep learning for program optimization and analysis. arXiv preprint arXiv:2003. 10536, 2020.
[15] C. Cummins, B. Wasti, J. Guo, B. Cui, J. Ansel, S. Gomez, S. Jain, J. Liu, O. Teytaud, B. Steiner, et al. Compilergym: Robust, performant compiler optimization environments for research. In 2022 IEEE/ACM International Symposium on Code Generation and Optimization (CGO), pages 92-105. IEEE, 2022.
[16] H. Dam, T. Tran, and T. Pham. deep language model for software code. arxiv: 02715, 2016.
arXiv preprint
[17] H. K. Dam, Pham, S. Ng. T. Tran, J. Grundy, A. Ghose, T. Kim, and C.-J. Kim. Lessons learned from using a deep tree-based model for software defect prediction in practice. In 2019 IEEE/ACM 16th International Conference on Mining Software Repositories (MSR), pages 46-57. IEEE, 2019.
[18] A. Duran, X. Teruel, R. Ferrer, X. Martorell, and E. Ayguade. Barcelona openmp tasks suite: A set of benchmarks targeting the exploitation of task parallelism in openmp. In 2009 international conference on parallel processing, pages 124-131. IEEE, 2009.
[19] Z. Feng, D. Guo, D. Tang, N. Duan, X. Feng. M. Gong. L. Shou, B. Qin. T. Liu, D. Jiang, et al. Codebert: A pre-trained model for programming and natural languages. arXiv preprint arXiv:2002.0811 2020.
11

---

[20] D. Grewe, Z. Wang, and M. F. O'Boyle. Portable mapping of data parallel programs to opencl for heterogeneous systems. In Proceedings of the 2013 IEEE/ACM International Symposium Code Generation and Optimization (CGO). pages 1-10. IEEE, 2013.
[21] X. Gu, H. Zhang, D. Zhang, and S. Kim. Deep api learning. In Proceedings of the 2016 24th ACM SIGSOFT international symposium on foundations of software engineering, pages 631-642, 2016.
[22] D. Guo, S. Ren, S. Lu, Z. Feng, D. Tang, S. Liu, L. Zhou, N. Duan, A. Svyatkovskiy, Fu, et al. Graphcodebert: Pre-training code representations with data flow. arXiv preprint arXiv:2009.08366 2020.
[23] D. Guo, S. Lu, N. Duan, Y. Wang, M. Zhou, and J. Yin, Uniccoder: Unified cross-modal pre-training for code representation. arxiv preprint arXiv:2203 03850, 2022.
[24] H.QQ. Jin, M. Frumkin, and J. Yan. The openmp implementation of nas parallel benchmarks and its performance. 1999.
[25] T. Kadosh, N. Hasabnis. T. Mattson, Y. Pinter, G. Oren, et al. Pragformer: Bata-driven parallel source code classification with transformers. 2023.
[26] D. Kingma and Ba. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6988 2014.
[27] D. Levy and L. Wolf. Learning to align the source code to the compiled object code. In International Conference on Machine Learning, pages
PMLR, 2017.
[28] Y. Li, C. Gu, T. Dullien, O. Vinyals, and Kohli. Graph matching networks for learning the similarity of graph structured objects. In International conference on machine learning, pages 3835-3845. PMLR, 2019.
[29] Z. Li, R. Atre, Z. U. Huda, A. Jannesari, and F. Wolf. Unveiling parallelization opportunities in sequential programs, Journal of Systems and Software, 1 17::222-295, July 2016. doi: 10.1016/j..ss.200 2016.00
[30] L. Mou, G. Li, L. Zhang, T. Wang, and Z. Jin. Convolutional neural networks over tree structures for programming language processing. In Proceedings of the AAAI conference artificial intelligence, volume 30, 2016.
[31] A. Paszke, S. Gross, F. Massa, A. Lerer, J. Bradbury, G. Chanan, T. Killeen, Z. Lim, N. Gimelshein, L. Antiga, A. Desmaison, A. Kopf, E. Yang. Z. DeVito, M. Raison. A. Tejani, S. Chil- amkurthy. B. Steiner, L. Fang, J. Bai, and S. Chintala. Pytorch: An imperative style, high- performance deep learning library. In Advances in Neural Information Processing Systems 32, pages 8024-8035, Curran Associates, Inc., 2019, URL hutt:://paper:: waappers.neuree wtt.pppp..ppppppp/e/eeee 9015 fall--yytocc--- an- imperative sepperative-stlle.. style- high beech--prfrrmaanc..m deep- learning Seree-iibrrrrrree
[32] L.-N. Pouchet and T. Yuki. Polybench: The polyhedral benchmark suite (version 4.2), 2017.
[33] D. Quinlan and C. Liao. The ROSE source-to-source compiler infrastructure. In Cetus users and compiler infrustructure workshop, conjunction with PACT, volume 2011, page Citeseer, 2011.
[34] J. Reinders, B. Ashbaugh, J. Brodman, M. Kinsner, J. Pennycook, X. Tian, J. Reinders, B. Ashbaugh, J. Brodman, M. Kinsner, et al. Common parallel patterns. Data Parallel C++ Mastering DPC++ for Programming of Heterogeneous Systems using C++ and SYCL, pages 323-352, 2021.
[35] A. Ronacher. Jinja2 documentation. Welcome to fiia2- Documentation (2.8-dev), 2008
[36] I. Sánchez Barrera, D. and optimizing numa effects and prefetching with machine learning. In Proceedings of the 34th ACM International Conference on Supercomputing, pages 1-13, 2020.
M. Casas, M. Moretó, A. Stupnikova, and M. Popov. Modeling
[37] M. Schlichtkrull, T. N. Kipf, P. Bloem, R. data with graph convolutional networks. In The Semantic Web: 15th International Conference, ESWC 2018, Heraklion, Crete, Greece, June 3-7, 2018, Proceedings 15, pages 593-607. Springer, 2018.
Van Den Berg, I. Titov, and M. Welling. Modeling relational
[38] A. Tehranilamsaz, M. Popov, A. Dutta, E. Saillard, and A. Jannesari. Learning intermediate representations using graph neural networks for numa and prefetchers optimization. In 2022 IEFF International Parallel and Distributed Processing Symposium (IPDPS), pages 1206-1216. IEFF, 2022.
[39] S. VenkataKeerthyy, R. Aggarwal, S. Jain, M. S. Desarkar, R. Upadrasta, and Y. Srikant. Ir2vec: Llvm in based scalable program embeddings. ACM Transactions on Architecture and Code Optimization (TACO), 17(4):1-27 2020.
[40] M. Y. Wang. Deep graph library: Towards efficient and scalable deep learning graphs. In ICLR workshop representation learning on graphs and manifolds, 2019.
12


Reviews:
[{"review_id": "wNUxim2u6c", "reviewer": "Reviewer_f7Gk", "summary": "The research identifies limitations in the current state-of-the-art program representation PROGRAML, in capturing features of numerical values and composite data types. \nTo address these limitations, this work introduces an enhanced GNN-based program representation for LLMV-IR with modifications to the nodes and edges of the program graphs to capture features of numerical values and composite data structures. It also presents a digit embedding approach to enhance numerical awareness within the representation. \nThe evaluation shows that PERFOGRAPH reduced the prediction error rate by 7.4% (AMD dataset) and 10% (NVIDIA dataset) in the Device Mapping challenge compared to PROGRAML. Additionally, it outperforms PROGRAML and traditional rule-based methods in tasks such as determining if a loop is parallelizable, classifying the parallel patterns, and configuring NUMA and prefetchers selections.\n", "questions": "1. It would be helpful to visualize the digit embedding values of different numerical values to facilitate better understanding.\n2. It would be beneficial to compare the cascaded context approach for representing multi-node representation with alternative representations (e.g. storing only the size of one rank) to assess the effectiveness in encoding array information.\n3. To provide a more comprehensive analysis, I recommend conducting an ablation study for each major representation enhancement to understand the specific benefits of each component.\n4. It would be valuable to measure and compare the inference time of PERFOGRAPH with PROGRAML and other rule-based analysis methods to evaluate the computational efficiency.", "limitations": "N/A", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "1. PERFOGRAPH employs effective graph modifications to enhance the learnability of the LLVM-IR graph for GNNs. These methods are well-reasoned and provide valuable domain information for improving program graph abstractions in learning.\nFor instance, PERFOGRAPH unifies the graph nodes for the same local identifier variable in the program graph. It also modifies the LLVM IR graph to add edges from the store instruction to the variables it modified. These modifications all simplify the complex structure of the LLVM-IR graph.\n\n2. PERFOGRAPH also distinguishes between composite data structures and regular data types, which prior approaches did not adequately address. This distinction is crucial because operations involving composite data structures can differ significantly from those involving regular data types.\n\n3. The evaluation of PERFOGRAPH is comprehensive, considering six performance-oriented tasks. The results demonstrate that PERFOGRAPH outperforms PROGRAML in terms of accuracy on these tasks. More importantly, in parallelism discovery tasks, PERFOGRAPH achieves higher accuracy compared to traditional rule-based approaches.\n", "weaknesses": "1. It is unclear to me how the digit embedding ensures that unknown numbers are not encountered during the test phase and what specific information is encoded through the addition of digit and position embeddings.\n2. The reasons behind representing multi-dimensional arrays as multiple nodes are not adequately explained. It remains unclear how this representation enhances expressiveness or facilitates reasoning by GNNs.\n3. An ablation study that isolates the individual benefits of LLVM-IR graph modification, digit embedding, and support for composite types is absent. Such a study would provide valuable insights into understanding the specific advantages offered by each component.\n4. PERFOGRAPH may have higher computational requirements compared to PROGRAML, potentially posing challenges for its practical deployment.\n\n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "- **(W1) Unknown numbers during the testing phase:**\\\nHere, by unknown numbers, we mean numeric tokens not encountered by the model during the training phase but present in the testing phase. Digit Embedding allows us to generate embedding for those unknown numbers. Because it breaks the numbers into digits, which are from 0 to 9 only. Then we get the numeric position of each of the digits. Then embedding is generated by combining each digit with its numeric position. This way, we can generate embedding for unknown numeric tokens. The most important feature of Digit Embedding is that it uses digits and their positions to generate embedding rather than the number itself, and as we know, digits are from 0 to 9. \n \n- **(W2, W3, Q1, Q2, Q3) Ablation Study, the Benefits of representing composite data types and visualization of Digit Embedding values:**\\\nWe have conducted an ablation study to have more insights on the contribution of each one of the enhancements that *PerfoGraph* brings; for example, in terms of composite data types, removing them increases the error rate of the AMD dataset by %13 for the device mapping dataset. Please refer to the global response and supplementary material file for more details. We have also included the visualizations of Digit Embedding values in the supplementary material file.\n\n- **(W4, Q4) The computation cost of PerfoGraph:**\\\nThank you for pointing this out. As mentioned in the global response, our observations indicate that *PerfoGraph* does not impose significant overhead when training DL models. In fact, we observed a slightly lower overhead with *PerfoGraph*. Please refer to the global response for more details regarding the computation time experiments.\n"}, {"review_id": "HDgIvUdIpB", "reviewer": "Reviewer_inFa", "summary": "This work proposes Perfograph, a program graph representation based on LLVM-IR and an extension to ProGraML. This graph representation is designed for the purpose of performance optimization and program analysis applications based on graph neural networks (GNN). This work made three contributions to the existing representation. First, it tracks reused local identifiers and memory locations. Second, it uses decimal-based encoding to embed numerical constants into the graph. Last, it breaks down array and vector types into multiple nodes. Perfograph is tested on 6 downstream tasks: device mapping, parallelism discovery, parallel pattern detection, NUMA and prefetchers configuration prediction, thread coarsening factor (TCF) prediction, and algorithm classification. In the device mapping challenge, the new model outperforms ProGraML by 7.4% on the AMD dataset and 10% on the NVIDIA dataset.", "questions": "1. It claims that “Perfograph is built on top of ProGraML” in Section 4, but also claims “A **new** compiler and language agnostic program representation based on LLVM-IR”. Is this work an extension of ProGraML, or is it a new system? This is somewhat unclear.\n2. Section 4.1 claims that this work combines multiple writes to the same variable into one node. However, LLVM deliberately separates multiple writes for the ease of certain aliasing analyses, the opportunities of reordering, and optimizing for parallelism. It would be helpful to provide more evidence that this combined representation outperforms separated representation in all (or most) downstream tasks.\n3. Section 4.2 mentions that decimal-based encoding is used for numerical constant embedding. This seems counterintuitive since computers natively speak binary.\n4. Section 4.3 claims that Perfograph supports composite data types (but only covers arrays and vectors). It would be helpful to further explain how Perfograph deals with programs that use structs (and maybe unions).", "limitations": "N/A", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "1. This paper is well organized.\n2. This work clearly listed all 3 of its design decisions.\n3. This work contains detailed evaluations on multiple targets.\n4. Each evaluation contains a carefully described problem definition, dataset, and results.", "weaknesses": "1. The improvement of this work over ProGraML seems limited. For example, the 3 main design contributions are not fundamentally new. They are more like minor tweaks on the existing ProGraML system.\n2. The evaluation section does not show how each design choice contributes to the overall improvement.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "- **(W1) Differences and Contributions that differentiate PerfoGraph from ProGraML:**\\\nWhile we acknowledge the similarities between *PerfoGraph* and *ProGraML*, as both represent programs as graphs using LLVM Intermediate Representations, *PerfoGraph* differs itself from *ProGraML* by:\\\nA) A more precise representation of local variables\\\nB) Incorporating and embedding numbers\\\nC) Supporting aggregate data types.\\\nMoreover, PerfoGraphs enhancements are not dependent on ProGraML. For instance, numerical embedding and our aggregate data types representation can easily be incorporated into other graph representations.\n\n- **(W2) Contribution of each design choice:**\\\nThank you for mentioning this. To provide insights into the contribution of each design choice, we have conducted an ablation study. For instance, for the Device Mapping dataset removing the composite data types increased the error rate by 13% for the AMD dataset. To see all the details on the ablation study, please refer to the global response and the supplementary materials file. Thank you.\n\n- **(Q1) Is this work an extension of ProGraML, or is it a new system?**\\\nWe drew inspiration from *ProGraML* but observed certain limitations in its design. Specifically, *ProGraML* sometimes fails to capture some essential information in its representation, notably numbers, composite data type nodes, and local identifiers. In *PerfoGraph*, we prioritized refining the representation to address these shortcomings. Given that ProGraML integrates foundational graphs like CFG and DFG when using IR as the basic element for graph construction, it is challenging to bypass it entirely. Our approach in *PerfoGraph* sets itself apart by offering novel digital embedding for numbers and tailored representations for data type nodes and local identifiers.\n\n- **(Q2) Combining multiple writes to the same variable:**\\\nYou are indeed right that LLVM separates the multiple writes to the same variable for the sake of simplifying optimization choices. In our representation, we do not combine multiple writes to the same node. Sorry for the confusion. We make sure to clarify this in the paper. For the example in Figure 2a in the paper, there are two `store` instructions. The first `store` instruction corresponds to assigning 0 to the variable *i*, and the second `store` instruction corresponds to increasing the value of *i* by 1. In LLVM IR, this is shown as follows:\n    ```\nstore i32 0, i32* %2, align 4\nstore i32 %4, i32* %2, align 4\n    ```\nAs we can see, the two `store` instructions are writing to the same local identifier, which is `%2`. However, ProGraML creates a separate node to present `%2` in each `store` instruction, as we can see in Figure 2a. This will make it difficult for DL models to understand that those `store` instructions are writing to the same local variable. In contrast, *PerfoGraph* considers only one node to represent `%2`.  For each temporary variable (`%2`, `%3`, …) that LLVM creates, *PerfoGraph* will also create separate nodes. The issue with ProGraML was that it was creating several nodes for the same temporary variable (Figure 2a).\n\n\n- **(Q3) Numerical Embedding is counterintuitive since computers speak binary:**\\\nWe proposed Digit Embedding to enable DL/ML models to be aware of numerical values. That is, if, for example, the loop bound (which can be a number) is known at compile time, we want the DL model to understand the scale and value of the number without losing its generalizability, that is, facing unknown numbers at test time. Digit embedding will essentially help to generate embedding for numeric tokens, which are present in the nodes of the graph representation. This will help DL models to make better predictions. Moreover, we intend to make the DL model aware of the numerical values at compile time. Therefore, we assume at the LLVM IR phase, we are not dealing with binary code.\n\n- **(Q4) Support for struct:**\\\nCurrently, *PerfoGraph* does not handle structs in a specific way different from *ProGraML*. In fact, *PerfoGraph* shows struct through a series of nodes. For example, a simple struct such as\n    ```\nstruct {\n    int myNum;\n    int myAge;\n  } myInfo;\n  myInfo.myNum = 815;\n  myInfo.myAge = 23;\n    ```\nIn LLVM IR is:\n    ```\n%2 = alloca %struct.anon, align 4\n%3 = getelementptr inbounds %struct.anon, %struct.anon* %2, i32 0, i32 0\nstore i32 815, i32* %3, align 4\n%4 = getelementptr inbounds %struct.anon, %struct.anon* %2, i32 0, i32 1\nstore i32 23, i32* %4, align 4\n    ```\n*PerfoGraph* will present these instructions by having\\\nControl nodes as: `alloca`, `getelementptr`, `store`, `getelementptr`, `store`\\\nData nodes as: `%struct`, `i32 0`, `i32 1`, `i32 23`, `i32 815`\\\nWhich is basically the same way that *ProGraML* handles struct. We will update the paper and use the term **aggregate data types** to avoid confusion, as suggested by another reviewer. \nThank you for pointing this out."}, {"review_id": "1DkZ8vJYVr", "reviewer": "Reviewer_vHZH", "summary": "The paper presents a graph-based representation for LLVM IR programs for processing with graph neural networks. The representation is based on ProGraML, an established technique for such graph representations, but adds several features to the graph: collapsing nodes that refer to the same variable, adding additional edges between memory stores and the variables they can affect, explicitly representing constants in the graph, and representing compositional data types (e.g., arrays of arrays) as multiple distinct nodes rather than single nodes. The authors evaluate across a range of LLVM IR learning tasks, and show that Perfograph outperforms ProGraML (and all other baselines in most contexts).", "questions": "* Please see the questions in \"clarity\" in the discussion of weaknesses above (and other questions throuhgout).\n* Does Perfograph use more compute or have more capacity than ProGraML? Does it still outperform ProGraML when given equal power?\n* What steps have the authors taken to ensure that the implementation of ProGraML compared against in Sections 5.3-5.7 is correct (since as noted in Line 372, in one context the authors \"could not reproduce the results in PROGRAML paper\")?\n* Do all other baselines use the exact same training/test set split?\n* How were hyperparameters tuned for Perfograph?\n\nI would be willing to raise my score given evidence that for the same compute budget, Perfograph still outperforms ProGraML (or evidence that my assumption here, that Perfograph uses more compute than ProGraML, is incorrect).\n", "limitations": "The paper does not explicitly discuss limitations of the technique.", "rating": 7, "confidence": 4, "soundness": 2, "presentation": 2, "contribution": 3, "strengths": "* Finding and understanding good program representations is an important problem in the field of learning for code\n* The proposed approach identifies a novel set of features which qualitatively seem important, and which are validated to be useful in the ablation study\n* Assuming that the evaluation is indeed fair (see weaknesses below), the proposed approach significantly outperforms ProGraML, achieving a large increase in accuracy (seemingly solving many of these tasks near perfectly).\n", "weaknesses": "* Fairness of comparison against baselines:\n  * Perfograph is a version of ProGraML with some added and removed nodes and edges. Each added node/edge adds additional power for a graph neural network. For a fair comparison against baselines (ProGraML in particular), the authors must show that for a given compute budget (i.e., FLOPs or wall clock time), Perfograph outperforms ProGraML; this would involve either scaling up/down ProGraML or Perfograph.\n  * Are the datasets used exactly the same as in prior work? For example, Section 5.4 reports using an 80/20 train/test split, which I cannot find discussed in [10]\n  * Line 372 mentions that the authors \"could not reproduce the results in PROGRAML paper\" for the algorithm classification task. Does this lack of reproducibility also affect the evaluated benchmarks that were not in the ProGraML paper (seemingly, everything other than 5.2)? If so, this is a major point that must be discussed at a high level in the paper.\n  * Given the advent of large-context Transformers, I would be curious how well such a model performs when applied to the raw IR (i.e., without an explicitly featurized graph representation). However, since this paper is explicitly building on the prior work of ProGraML, I do not consider this to be a condition for acceptance.\n* Overall, the clarity of the submission is moderate.\n  * Section 4.1: without a deep understanding of the ProGraML representation, it is very hard to understand these graphs (and therefore the issues that the proposed approach fixes in them). Perhaps highlighting or otherwise identifying the specific nodes discussed could help.\n  * Line 323: is this an arithmetic or geometric average?\n  * Line 368: what are the units of these errors?\n* Several claims about novelty or related work are misleading:\n  * Lines 72-73: \"this token-based representation... fails to capture...\" This claim is misleading. Sequence processing models are entirely capable of learning such graph-structured relationships; such complex relationships are also not unique to code (e.g., natural language has pronouns, nested structures, etc.)\n  * Line 176: the proposed \"Digit Embedding\" is very similar to some digit representations approaches used in Transformers (for example, Geva et al. 2020) which embed each digit, add/concatenate with a positional encoding, and aggregate with attention.\n* I believe the paper would be significantly stronger with the ablation studies moved to the main body, but this is a stylistic choice and is not a factor in my score\n* If I'm understanding Appendix 2 correctly, the Perfograph GNN is evaluated with only 2 layers; ProGraML is evaluated with 6 (Cummins et al., Section 6.2). How was this parameter chosen? What is the average diameter of the evaluated graphs (for both Perfograph's graphs, and ProGraML's graphs)?\n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "- **(W1.a, Q2) The computation cost PerfoGraph:**\\\nWe conducted an experiment in terms of the time it takes to train the GNN model using *PerfoGraph* versus *ProGraML*. We found out that *PerfoGraph* takes less training time than *ProGraML*, and *PerfoGraph* also has better performance than *ProGraML*. For more details on this experiment, please refer to the global response. Thank you.\n\n- **(W1.c, Q3) Lack of reproducibility of ProGraML for algorithm classification:**\\\nApologies for any ambiguity. As detailed in Section 5.7, we assessed the algorithm classification performance using both *ProGraML* and *PerfoGraph*. However, our replication did not mirror the error rate mentioned in the *ProGraML* paper. We observed an error rate of 6%, whereas the *ProGraML* paper cited 3.38%. This minor discrepancy in error rate could potentially be rectified with further fine-tuning of the GNN model tailored for *ProGraML*. Nonetheless, the *ProGraML* repository did not provide any checkpoints. For this paper, we initialized the GNN models with identical settings and gauged the performance of *ProGraML*. In the interest of fairness, we have presented both the error rate from the *ProGraML* paper and our results.\\\nFor the Device Mapping task (section 5.2), we were able to reproduce the same results as reported in the *ProGraML* paper. So, we compared *PerfoGraph* with the accuracy mentioned in the *ProGraML* paper.\\\nOther downstream tasks (sections 5.3, 5.4, 5.5, 5.6) are not described in the *ProGraML* paper. For a fair comparison of *PerfoGraph* with *ProGraML* for these downstream tasks, we keep everything the same in our pipeline except for the program representation.\n\n- **(W2.a) The difference with ProGraML in Figure 2:**\\\nWe will make sure to highlight those nodes and edges in Figure 2 for better visibility. Some enhancements of PerfoGraph are shown in the global response pdf file.\n\n- **(W2.b) Type of average on line 323 (Arithmetic of Geo Mean?):**\\\nThis is an arithmetic average. We ensure to clarify this in the paper.\n\n- **(W2.c) Unit of errors on line 368:**\\\nWe represent the error rates as percentages. For example, inst2vec has an error rate of 5.17 means inst2vec has an accuracy of 94.83%. We will include the percentage sign in the revised submission to avoid confusion.\n\n- **(W3.a) The claim of token-based representation fails to capture relations is misleading:**\\\nSorry for the confusion. While there have been studies that incorporate relations in token-based representations, these often necessitate specialized models, tailored representations, or additional training data and time to effectively capture relation information. In contrast, previous works (Zhang, Jian, et al. 2019, and Chen, Le, et al. 2023) have shown that graph representations of code inherently encapsulate critical relational data more naturally than their token-based counterparts. Using compiler-generated IR, *PerfoGraph* ensures that relational information is both accurate and precise, making it an efficient representation for models to capture code features. We will adjust our description of token-based representation to ensure clarity.\n\n-  **(W3.b) The Digit Embedding is similar Geva et al:**\\\nThank you for pointing this out. The idea of digit embedding was actually inspired by input and position embeddings in Transformers. We applied the position encoding to each one of the digits of a number. The idea can be considered similar to character encoding with assigning a position to each character. However, to the best of our knowledge, we did not find any tool or model that can do this type of embedding for numbers. Hence we proposed Digit Embedding. \n\n- **(W4) Moving the ablation to the main paper:**\\\nThank you for mentioning this point. As mentioned in the global response, we will make sure to include the ablation study in the paper.\n\n- **(W5) What is the diameter of the graphs:**\\\nWe measured the average diameter of the 30k training IR files in the Parallelism Discovery task as an example. Below are the results:\\\nAvg diameter of PerfoGraph:  28.3\\\nAvg diameter of ProGraML: 31.54\n\n\n- **(W1.b, Q4) Is Train/Test Split in other baselines the same?**\\\nYes, the datasets are used exactly the same as in prior works.\\\nSec 5.2: Same as *ProGraML* paper. We do 80% training, 10% validation, and 10% testing\\\nSec 5.3: Same as *Graph2Par* paper. Around 30k for training and testing on three subsets *Pluto* (4032 IR files), *autoPar* (3356 IR files), and *DiscoPoP* (1226 IR files)\\\nSec 5.4: *Graph2Par* paper reproduced the *Pragformer* work to show the comparisons, and in the *Pragformer* repository, they used 80% and 20% split. So we used the same 80% training and 20% testing split ratio\\\nSec 5.5: Same as in the study of TehraniJamsaz et al., 10-fold cross-validation\\\nSec 5.6: Same as in the *inst2vec* paper, leave one out cross-validation\\\nSec 5.7: Same as *ProGraML* paper. 240k IR for training and 10k IR file for testing.\n\n- **(W5,Q5) How are hyperparameters tuned?**\\\nWe have tuned the hyperparameters of the model experimentally. We did experiment with a higher number of layers; however, it did not bring any benefits, despite the higher computation cost. Below we show the comparison of accuracies among different layers of our GNN-based model for the task of Device Mapping.\n\n    | Device | 2 layers accuracy  | 3 layers accuracy | 4 layers accuracy | 5 layers accuracy  | 6 layers accuracy|\n    |--------|------|------|------|------|------|\n    | AMD    | 0.94 | 0.88 | 0.91 | 0.87 | 0.90 |\n    | NVIDIA | 0.90 | 0.85 | 0.90 | 0.90 | 0.88 |\n\n    We use Adam optimizer as it is the default configuration for DGL-based RGCN and is widely used (e.g., in the ProGraML paper). We use Relu activation (default of DGL-RGCN) as it is also widely used. We experimented with commonly used hidden layer sizes (32, 48, and 64) and learning rates (0.1, 0.01, and 0.001) and observed best results are obtained with hidden layer size 64 and learning rate 0.01. "}, {"review_id": "xoVHolcnw9", "reviewer": "Reviewer_ubPp", "summary": "The paper proposed a novel graph-based program representation, PerfoGraph, which is based on the current state-of-the-art method PrograML and aims to address its limitation by providing numerical awareness, introducing new tactics for handling local variables, and supporting aggregate data types. By conducting experiments on various performance-optimizing oriented downstream tasks, it is shown that the proposed PerfoGraph representation is more effective compared to the existing methods.", "questions": "1. The authors claimed that the proposed PerfoGraph representation could better capture **composite** data structure information. However, the paper only introduced a novel method for incorporating list-like data structures, such as **arrays and vectors**, into the graph representation of the program. The reviewer's question is how PerfoGraph could support more general composite data types such as struct. From the current presentation, the reviewer's opinion is that the term **aggregate data types** suit better.\n2. How does the choice of aggregation function impact the performance of PerfoGraph? Is it possible that the information compression introduced in the aggregation function confuses the deep learning model? \n3. Do the authors consider moving the part of the ablation study from Supplementary Materials to the main text to better clarify the importance of numerical awareness in program representations?\n4. Possible error in Figure 2a: In section 4.1, the authors used Figure 2a as an example of the PrograML representation to point out that PrograML assigns a position feature to the address edge of the store node. However, there is no corresponding label near the first store node. ", "limitations": "The authors have addressed the limitations of their work in Section 6.", "rating": 7, "confidence": 5, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "- The paper was clearly presented overall.\n- The paper showed great originality in proposing a novel program representation method, PerfoGraph, that enhances previous results by providing numerical awareness, introducing new tactics for handling local variables, and supporting aggregate data types.\n- Regarding numerical awareness, the authors introduced a novel idea to split the numbers into digits and positions before encoding, which could effectively represent numbers without a huge vocabulary.\n- Regarding local variables, the author took good consideration of GNN's mechanism and adjusted the graph representation accordingly.\n\n", "weaknesses": "- Unconvincing experiment setup in comparison with PrograML.\nIn the original PrograML paper, MPNN was introduced as an encoder. However, the proposed PerfoGraph chose RGCN for the same role. The reviewer believes that it is possible to also use RGCN as PrograML's encoder. Thus, it is unclear whether the performance improvements in the comparisons with PorgraML stem from the novel graph representation or from a more suitable encoder.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "- **(W1) Unconvincing experiment setup in comparison with PrograML:**\\\nIn the downstream tasks of sections 5.3, 5.4, 5.5, and 5.6, we used the RGCN encoder with ProGraML and compared it with PerfoGraph. For each of these downstream tasks, we used an RGCN model with the same architecture as described in Table 1 of Supplementary Materials both for PerfoGraph and ProGraML so that we can be sure that the differences in results are only because of changing the program representation. And as mentioned in the paper, results show that *PerfoGraph* representation performs better than *ProGraML* in almost all these tasks.\n\n- **(Q1) Aggregate data types:**\\\nYou are right. *PerfoGraph*, at the moment, only supports aggregate data types. Therefore, we will update the paper and will mention aggregate data types instead of composite data types.\nAs explained to the question of inFa reviewer, *PerfoGraph* does not handle structs in a specific way different from *ProGraML*. In fact, *PerfoGraph* shows struct through a series of nodes. For example, a simple struct such as\n    ```\nstruct {\n    int myNum;\n    int myAge;\n  } myInfo;\n  myInfo.myNum = 815;\n  myInfo.myAge = 23;\n    ```\nIn LLVM IR is:\n    ```\n%2 = alloca %struct.anon, align 4\n%3 = getelementptr inbounds %struct.anon, %struct.anon* %2, i32 0, i32 0\nstore i32 815, i32* %3, align 4\n%4 = getelementptr inbounds %struct.anon, %struct.anon* %2, i32 0, i32 1\nstore i32 23, i32* %4, align 4\n    ```\n*PerfoGraph* will present these instructions by having\\\nControl nodes as: `alloca`, `getelementptr`, `store`, `getelementptr`, `store`\\\nData nodes as: `%struct`, `i32 0`, `i32 1`, `i32 23`, `i32 815`\\\nWhich is basically the same way that *ProGraML* handles struct. We will update the paper and use the term **aggregate data types** to avoid confusion.\n\n\n- **(Q2) Does the choice of aggregation function impact the performance of PerfoGraph?**\\\nWe do observe some impact of aggregation function in the `GraphConv` layer of our GNN pipeline. We found `sum` aggregation provides the best performance overall. We did an experiment with the Device Mapping task using different aggregation functions.\n\n    | Device | Sum agr. accuracy  | Max agr. accuracy | Mean agr. accuracy | Min agr. accuracy  |\n    |--------|------|------|------|------|\n    | AMD    | 0.94 | 0.90 | 0.93 | 0.79 |\n    | NVIDIA | 0.90 | 0.85 | 0.90 | 0.84 |\n\n\n- **(Q3) Moving Ablations study to the main paper:**\\\nAs discussed in global repose, the ablation study was moved to the supplementary material file due to the lack of space in the paper. But for the final version of the paper, we will make sure to include the ablation study as requested. Thank you for pointing this out.\n \n- **(Q4) Possible error in Figure 2a:**\\\nThanks for pointing this out. The label is there but it is overlapping with the edge itself that's why it is difficult to see. We will fix it in the revised version of the paper."}]

Author rebuttal:
We would like to thank all reviewers for their constructive feedback and comments.
Hereby, we address some of the common concerns.
- **Moving the Ablation Study to the main paper:**\
Due to the lack of space in the paper, we currently have the ablation study results in the Supplementary file. We will make sure to move and include the ablation study in the main paper. Thank you.
- **Ablation Study Results:**\
We have conducted experiments on the contribution of each design choice and presented the results as Ablation Study in the Supplementary Materials file.
For example, for the device mapping problem, removing composite data type nodes increases the error rate to %13 for the AMD dataset and 15% for the NVIDIA dataset. The supplementary material file contains more information about the Digit Embedding, Architecture of the models, Ablations study, and dataset. 
- **Computation cost of PerfoGraph:**\
Thank you for pointing this out.
We measured the computation cost of *PerfoGraph* against *ProGraML*. For this experiment, we took the *OMP_Serial* dataset of the Parallelism Discovery task, which contains 30k IR files for training. The hardware configuration is the same as reported in section 5.1. We train our GNN model for 100 epochs with both *ProGraML* and *PerfoGraph* five times and report the average training time as follows:\
*PerfoGraph: 14 minutes and 59 seconds.*\
*ProGraML: 15 minutes and 56 seconds.*\
Then we measured the testing time with both *ProGraML* and *PerfoGraph* representation for the *Pluto* subset of *OMP_Serial* dataset as it is the largest subset containing 4032 test IR files. For measuring testing time, we again measured the testing time five times and reported the average testing time.\
The average testing time results for  the *Pluto* subset are as follows:\
*PerfoGraph: 881 milliseconds*\
*ProGraML: 1231 milliseconds*\
And as already shown in the paper, for the *Pluto* subset, *PerfoGraph* has higher accuracy **(91%)** than *ProGraML* **(89%)**.
Moreover, we measured the average diameter of *PerfoGraph* and *ProGraML* for the 30k IR files in the Parallelism Discovery training dataset:\
*Average diameter of PerfoGraph:  28.3*\
*Average diameter of ProGraML: 31.54*\
Based on these observations, we believe it is fair to say that the computational overhead of *PerfoGraph* is slightly less than *ProGraML*, while *PerfoGraph* has shown better performance, as discussed in the paper.


Meta-review: This is a primarily empirical submission, presenting three modifications to the existing ProGraML approach to representing LLVM IR as graphs and illustrates their usefulness with experiments on a range of a performance-oriented tasks from the literature, showing that Perfograph autperforms baseline methods.

The proposed modifications are small and not grounded in theory, but instead stem from observations of shortcomings in ProGraML (as highlighted by reviewer inFa). This implies a high quality bar for the experimental section of the paper. As noted above, these experiments are wide in nature, covering six different tasks. However, the considered baselines are fairly dated (apart from Graph2Par and Pragformer used in Sect. 5.3 and 5.4, all were published in 2020 or earlier) and do not reflect that in the wider ML4Code community, LLMs have fully replaced graph-based models (see e.g. https://openreview.net/pdf?id=B1lnbRNtwr for a paper from 2020 on this). This was raised by reviewer vHZH, and the authors chose not to respond to this in their rebuttal.

Indeed, nearly-concurrent work by https://arxiv.org/pdf/2306.14979.pdf by Chen et al (the authors of reference [10] in the paper, from which the tasks in Sect. 5.3 and 5.4 are drawn) illustrates the importance of a comparison to LLMs: gpt-3.5-turbo with retrieval-augmented prompts outperforms all considered methods on parallelism detection (Sect. 5.3 in the submission), including the Graph2Par baseline compared to in this submission. This is in line with other recent works, which show that LLMs succeed on harder, more free-form performance optimization tasks (e.g. https://arxiv.org/pdf/2206.13619.pdf and the very recent https://arxiv.org/pdf/2306.17281.pdf).

Finally, the paper in its submitted form does not include substantial ablation studies. This point was raised by reviewers f7Gk, inFa; reviewers vHZH and ubPp found some unreferenced experiments in the supplement and asked for them to be moved into the main paper. While the authors have promised to include them in the main text, these ablations remain incomplete: they do not consider the effect of one of the proposed changes (Sect. 4.1) at all.

To summarise, this submission proposes small improvements to the existing framework ProGraML framework and illustrates their utility on six benchmarks from the literature. However, it does not does not fully ablate the proposed changes and does not engage with the recent dominant trend in the research community (to be clear, I am not saying that every paper needs to follow this trend, but you have to acknowledge such things and compare performance).

As AC, I found this submission to be by far the hardest decision to make, and I spent significant time on investigating the paper and ended up doing a full review (see below for detailed comments). In the end, and given the limited space at a very competitive conference such as NeurIPS, I believe that the submission is not ready to be accepted: to fully illustrate the usefulness of the proposed techniques, wider experiments taking LLMs into account are required. LM4HPC (https://arxiv.org/pdf/2306.14979.pdf) illustrates how this could be done.

In final discussions with the SAC and PC, the issue of fairness was raised: the authors had no opportunity to rebut the weaknesses highlighted in this meta-review, and so the decision was made to accept the paper, even though there are open issues that should be addressed. The authors are encouraged to improve the final version of the paper in response to these comments.

Detailed comments from an in-depth review:
* l35-38: "[ProGraML's] limitations stem from neglecting numerical values available at compile time and the inadequate representation of composite data types." - this statement is just foreshadowing what the paper proposes, but is not actually supported by evidence that these are (all of) the most important limitations.
* l47-48 (contribution 1): "A new compiler and language agnostic program representation" - as noted by reviewer inFa, this is overclaiming a bit, given that the paper is largely based on ProGraML (and even defers to it for exact definitions).
* l73-75: "[token-based representation] fails to capture the unique relationships and dependencies between different elements of the code, which can limit its effectiveness in tasks such as compiler optimization and code optimization" - no evidence provided. The cited papers (e.g., [22]) already show by-now smallish LMs (<500M parameters) being on par with graph-based approaches, and follow-ups such as https://arxiv.org/pdf/2305.07922.pdf show pure LMs outperform graph-based approaches. See the discussion above for why this is problematic.
* Sect. 4.1: I found the writing in this section highly confusing - it would be very helpful if the LLVM IR that forms the basis of the graph would be presented explicitly, as it is repeatedly referred to. (I ended up using clang to generate it myself, which should not be expected of readers)
* Sect. 4.1: I'll also note that the use of edge features to indicate argument positions is a commonly used trick (though usually not highlighted). The earliest occurrence I could find is https://aclanthology.org/P17-1105.pdf.
* Sect. 4.2: The proposed "digit embedding" is a variation of standard schemes known from NLP (where embedding numbers by digit is common, and a natural consequence of BPE approaches). The core novelty here is the positional encoding of floating point numbers. However, positional embeddings are commonly used in combination with attention mechanisms. The aggregation functions proposed here (min/max/mean) seem to be unable to make use of this information, as they are order-invariant. In particular, when using the mean, the "positional" part of the embedding is entirely dependent on the length of the number (as `avg([emb(digit1) + emb(pos1), ..., emb(digitN) + emb(posN)])` can be rewritten as `avg([emb(digit1), ..., emb(digitN)]) + avg([emb(pos1), ..., emb(posN)])`).
Consequently, pairs of numbers such as `91` and `19` or `999.111` and `111.999` would yield the same representation.
Note that the paper does not actually provide any details on which aggregation function is used for the experiments, but in Fig. 1 of the supplementary material, exactly this effect shows up (e.g. for 34/43, 35/53). I'll note that the plot is surprising in that some numbers seem to be missing entirely (12, 13, 14, ...), as this would have made this issue very visible.

* Sect. 4.3: Again, the description here is somewhat confusing. The paper does not precisely define node representations, and it's unclear how the "additional features" are used (array length & data type). We have to assume that this is addition to the ProGraML node features (simple embedding lookup of the full textual node representation). I found the choice of using several chained nodes surprising (given that the authors raise the issue of the number of hops required to "understand" a construct in Sect. 4.1) and would appreciate more discussion of alternative representations here (e.g., you could imagine a similar scheme as in Sect. 4.2, where an array of shape [d1, ..., dN] and type T is represented by a embedding [d1, ..., dN] with additional positional information and concating that with an embedding of T).

* Sect. 4.2/4.3: Both problems tackled here (a naive representation of a potentially large set of node labels) would likely be solved by a more advanced approach of representing node labels, as discussed by http://proceedings.mlr.press/v97/cvitkovic19b/cvitkovic19b.pdf.
Decision: Accept (poster)