Title: PERFOGRAPH: A Numerical Aware Program Graph Representation for Performance Optimization and Program Analysis
Venue: NeurIPS 2023
Accepted: yes
Keywords: program representation

graph representation

program analysis

graph neural networks

performance optimization
Abstract: The remarkable growth and significant success of machine learning have expanded its applications into programming languages and program analysis. However, a key challenge in adopting the latest machine learning methods is the representation of programming languages which has a direct impact on the ability of machine learning methods to reason about programs. The absence of numerical awareness, aggregate data structure information, and improper way of presenting variables in previous representation works have limited their performances.  To overcome the limitations and challenges of current program representations, we propose a novel graph-based program representation called PERFOGRAPH. PERFOGRAPH can capture numerical information and the aggregate data structure by introducing new nodes and edges. Furthermore, we propose an adapted embedding method to incorporate numerical awareness.
These enhancements make PERFOGRAPH a highly flexible and scalable representation that can effectively capture programs' intricate dependencies and semantics. Consequently, it serves as a powerful tool for various applications such as program analysis, performance optimization, and parallelism discovery. Our experimental results demonstrate that PERFOGRAPH outperforms existing representations and sets new state-of-the-art results by reducing the error rate by 7.4% (AMD dataset) and 10% (NVIDIA dataset) in the well-known Device Mapping challenge. It also sets new state-of-the-art results in various performance optimization tasks like Parallelism Discovery and Numa and Prefetchers Configuration prediction.

PERFOGRAPH: A Numerical Aware Program Graph Representation for Performance Optimization and Program Analysis

Abstract

The remarkable growth and significant success of machine learning have expanded its applications into programming languages and program analysis. However, a key challenge in adopting the latest machine learning methods is the representation of programming languages, which directly impacts the ability of machine learning methods to reason about programs. The absence of numerical awareness, aggregate data structure information, and improper way of presenting variables in previous representation works have limited their performances. To overcome the limitations and challenges of current program representations, we propose a graph-based program representation called PERFOGRAPH. PERFOGRAPH can capture numerical information and the aggregate data structure by introducing new nodes and edges. Furthermore, we propose an adapted embedding method to incorporate numerical awareness. These enhancements make PERFOGRAPH a highly flexible and scalable representation that effectively captures programs' intricate dependencies and semantics. Consequently, it serves as a powerful tool for various applications such as program analysis, performance optimization, and parallelism discovery. Our experimental results demonstrate that PERFOGRAPH outperforms existing representations and sets new state-of-the-art results by reducing the error rate by 7.4% (AMD dataset) and 10% (NVIDIA dataset) in the well-known Device Mapping challenge. It also sets new state-of-the-art results in various performance optimization tasks like Parallelism Discovery and NUMA and Prefetchers Configuration prediction.

1 Introduction

In recent years, the remarkable success of machine learning has led to transformative advancements across numerous fields, including compiler optimization and program analysis. The applications include compiler heuristics prediction, optimization decisions, parallelism detection, etc. [4, 15]. The training process generally involves feeding program data as input and transforming it into a representation suitable for machine learning models. The selection of program representation is crucial, as it can significantly impact the performance of the machine learning model [12]. With the development of graph neural networks (GNNs), an increasing number of graph representations of programs have been incorporated into GNN models for program analysis [3, 28, 10]. One of the pioneering efforts in developing a comprehensive graph representation for programs is PROGRAML [14]. PROGRAML incorporates control, data, and call dependencies as integral components of a program's representation. In contrast to prior sequential learning systems for code, PROGRAML

closely resembles the intermediate representations used by compilers, and the propagation of information through these graphs mimics the behavior of typical iterative data-flow analyses. Despite the success that PROGRAML has achieved, there are shortcomings in this current state-of-the-art program representation, especially for performance-oriented downstream tasks. These limitations stem from neglecting numerical values available at compile time and the inadequate representation of aggregate data types.

In this paper, we present PERFOGRAPH to address the limitations of the current state-of-the-art program representation. Additionally, we propose a novel way to embed numbers in programs in an elegant way so that our DL model will not face unknown numbers during inference time. Our experiments demonstrate that PERFOGRAPH sets new state-of-the-art results in numerous downstream tasks. For example, in the Device Mapping downstream task, PERFOGRAPH yield error rates as low as 6% and 10% depending on the target hardware. Moreover, PERFOGRAPH even outperforms the tools and models specially designed for specific tasks such as parallelism discovery.

Overall, the main contributions of this paper are:

An enhanced compiler and language-agnostic program representation based on PROGRAML that represents programs as graphs.

The proposed representation supports aggregate data types and provides numerical awareness, making it highly effective for performance optimization tasks.

Evaluation of the proposed representation on common downstream tasks and exceeding the performance of PROGRAML.

Quantification of the proposed approach on a new set of downstream tasks such as parallelism discovery and configuration of NUMA systems.

The rest of the paper is structured as follows: Section 2 presents the related works. In section 3, we provide a motivational example, showing the limitations of PROGRAML, the state-of-the-art program representation. This section is followed by section 4 where we present our proposed representation PERFOGRAPH along with the novel way of embedding numerical values. In section 5, experimental results on downstream tasks are provided, and finally, Section 6 concludes the paper and discusses some future works.

2 Related Works

Machine learning has brought significant advancements in many fields, and program analysis and software engineering are no exceptions. However, Machine Learning (ML) and Deep Learning (DL) models can not directly process raw source code to reason about programs. Therefore, researchers have explored different approaches to represent applications in a format suitable for DL models. Generally, there are three types of commonly used program presentations: sequence of tokens, Abstract Syntax Tree (AST), and Intermediate Representation (IR).

Sequence of tokens: The initial attempts [16, 21, 27] represented source code as a sequence of tokens, such as identifiers, variable names, or operators. This approach intuitively treats programming languages similarly to natural languages. It allows for the utilization of advanced natural language process (NLP) techniques, such as large language models [19, 22, 23]. However, this token-based representation overlooks the inherent dependency information within the program's structure. It fails to capture the unique relationships and dependencies between different elements of the code, which can limit its effectiveness in tasks such as compiler optimization and code optimization.

AST: An AST represents the structure of a program by capturing its hierarchical organization. It is constructed based on the syntactic rules of the programming language and provides a high-level abstraction of the code. Previous works have leveraged ASTs as inputs to tree-based models for various code analysis tasks like software defect prediction [17] and code semantic study [9]. Moreover, there have been efforts to augment ASTs into graphs that incorporate program analysis flows such as control flow and data flow. These AST-based graph representations capture more comprehensive code dependency information and have shown superior results compared to traditional approaches in previous works [2, 3].

IR: IR is an intermediate step between the source code and the machine code generated by a compiler. Previous work [39] has utilized IR to train an encoding infrastructure for representing programs as a distributed embedding in continuous space. It augments the Symbolic encodings with the flow of information to capture the syntax as well as the semantics of the input programs. However, it

generates embedding at the program or function level and also requires a data-flow analysis type for generating the embedding. In contrast, our approach derives embedding from the representation and works at the more fine-grained instruction level. More recent works [6, 14, 8] have leveraged IR-based graph representation to better capture essential program information, such as control flow, data flow, and dependencies. However, despite their success, IR-based graph representations have certain limitations. For example, these representations may not be numeric-aware or may lack the ability to adequately represent aggregate data types. In this work, we propose PERFOGRAPH, a graph representation based on IR, to address these limitations.

3 Motivation

As stated in the related work section, program representations based on the intermediate representation are very effective in enabling DL models to automate the process of various optimizations. One such representation is PROGRAML, whose performance surpasses other code representations, making it state-of-the-art in various optimizations and downstream tasks. However, despite its potential, it suffers from several limitations. To name a few: it is incapable of properly carrying out information regarding read and write operations to the memory location, has no support for aggregate data types, and discards numerical values. Listing 1 shows a code snippet where a 3-dimensional array is defined. Figure 1 shows the PROGRAML representation of this code snippet. For illustration purposes, instruction nodes and control flow edges are shown in blue, whereas red represents variable, constant nodes, and data flow edges. Green edges show the call graph. As it can be seen, PROGRAML fails to represent some critical information. For instance, code float arr[2][3][4] is converted to LLVM IR [2 x [3 x [4 x float]]]* , which is used to construct a node in PROGRAML. It eliminates the aggregate data structure information, like the array's dimension. Leaving it up to the DL model to infer the meaning behind the numbers in [2 x [3 x [4 x float]]]* . Moreover, in this representation, only the type of numbers (e.g., int8 , float ) are considered, and the actual values of the numbers are not given attention. The absence of numerical awareness limits the performance of PROGRAML in applications where numerical values play an important role. A numerically aware representation can help understand and optimize operations involving numeric data types, constants, and expressions. There are also some anomalies in the way temporary variables are depicted. For example, in 1, we see the fourth alloca node allocates memory for a variable, and two store instructions are applied on two separate nodes representing the variable. Thus, the information about the first store instruction is not carried out properly when the second store instruction happens. In the following section, we will see how PERFOGRAPH effectively addresses many limitations in the current state-of-the-art program representation. PERFOGRAPH uses PROGRAML representation as its initial graph and reconstructs the graphs by addressing the aforementioned limitations.

[external]

alloca

i32

store

store

ret

i8*

call

1

i64

2

i32

alloca

alloca

alloca

alloca

store

i1

3

i32*

i32*

i8***

i32*

i32*

[2

x

[3

x

[4

x

float]]]*

store

1

store

1

1

bitcast

1

i8*

1

i8**

i32

4 PERFOGRAPH: A fined-grained numerical aware graph representation

PERFOGRAPH is graph representation based on LLVM IR. It is built on top of PROGRAML; however, it does not suffer from the limitations that the PROGRAML has, helping DL models to reason over the complex structure of the programs and enabling them to make more accurate optimization decisions,

especially in terms of performance optimization. Figure 2 shows how various enhancements and improvements are applied to construct a more precise representation. Consider a simple code example of defining a variable and increasing it by one { int i = 0; i++; }. Figure 2a shows the PROGRAML representation of this code example. It has two store nodes, as one is responsible for storing 0 and the other one storing the incremented value of i . In the following subsection, we will explain how PERFOGRAPH is constructed by addressing the limitations shown in Figure 2a.

alloca

store

i32*

i32*

i32*

load

1

store

1

add

i32

i32

i32

i32

1

(a) Initial graph

alloca

store

i32*

load

1

store

1

add

i32

i32

i32

i32

1

(b) Local variables

alloca

store

i32*

load

1

store

1

add

i32

i32

i32

i32

1

(c)

store

nodes

alloca

store

i32*

load

1

store

1

add

i32

i32

[i32, 0]

[i32, 1]

1

(d) Numbers

4.1 Representing Local Identifiers and store instruction

Local Identifiers: Local identifiers' names are preceded by % in LLVM Intermediate representation. Memory allocation on the stack is done by alloca instruction. One of the limitations of the current state-of-the-art program representation, PROGRAML, is that it is unable to carry out information regarding the operations that happen to a memory location. For instance, in Figure 2a, the two store nodes represent storing values of 0 and 1 to variable i . However, as shown, each store instruction node is connected to a separate variable node, making it difficult for the graph neural network to reason over the operations that happen to a memory location. For the embedding vector of the second store node in 2a to represent the fact that some information regarding the variable i has changed, one has to increase the number of GNN layers to 3 to support up to 3 hops when propagating the messages in GNN. This can potentially limit the ability of the GNN model if there are a greater number of hops between the two store nodes shown in Figure 2a. To address this limitation, instead of having more than one variable node (oval-shape nodes) per identifier, PERFOGRAPH only considers one variable node in its graph representation. Any load or store instruction will refer to the same variable node. These changes are shown in Figure 2b. We see that the store nodes in Figure 2b access the same memory location, thus representing the fact that those store instructions are modifying the same memory location.

Store instruction: LLVM uses store instruction to write into memory. store instruction has two arguments: a value to store and the address to which it will store the value. PROGRAML differentiates between these two arguments by adding a position feature to the edges as shown in Figure 2a. However, since the store instruction modifies the contents at the corresponding memory address, we posit that it is better to reflect the fact the content of the identifier has changed. To present this information, PERFOGRAPH adds an extra edge from the store node to node representing the identifier whose value is modified by the store instruction. Figure 2c shows these changes in the graph constructed by PERFOGRAPH.

Numbers: Numbers can be a significant factor in optimization decisions. For example, they can show the loop bound, and different optimizations can be considered depending on the loop bound. PERFOGRAPH, unlike PROGRAML, not only considers the type of numbers such as i32 , i64 , float but also the actual values of the numbers. As illustrated in Figure 2d, numerical constant nodes have the actual value of the number in their feature set in addition to the type of the number. Even though numerical constant nodes have the value of the number as one of their features, there is a need to embed the numbers in a way that unknown numbers will not be seen in the inference. Unlike other tokens, numbers are harder to embed as an infinite amount of numbers exists, and to handle all ranges of numbers, we need to have a very large vocabulary set.

4.2 Numerical Awareness

Number:

Positions: Digit emb: Position emb: Final emb: Digit + position emb: However, with a very large vocabulary size, the DL models may still encounter numbers in the inference phase that they have not seen in the training phase. We propose a novel way of embedding numbers called Digit Embedding. Figure 3 shows our approach. To embed a number, we first break down the number to its digits; then, we consider a position for each one of the digits. The goal is to let DL models realize the place value of each digit. Then, each digit and its corresponding position are embedded and summed together. Therefore, we will have an embedding representing the information about the digits and their positions. For instance, in Figure 3, we embed each digit and its corresponding position with an output dimension of 3. Since the number has four digits, the results would be a vector/tensor of size 4 × 3 . To make sure the Digit Embedding of numbers has the same length across numbers

41.25

4

1

2

5

2

1

-1

-2

Positions:

Aggregate

+

Digits:

Digits emb:

Positions emb:

Digit + Position

emb:

Final emb:

with varying sizes of digits, we apply an aggregation function over the embedding dimension. Since the output embedding dimension is three in this example, we would have one vector of length three representing the number after aggregation. The aggregation function can be of any type (Max, Mean, etc.).

4.3 Aggregate Data Types

Aggregate data types, such as arrays and vectors, are an essential part of applications. They play an important role in many applications, such as matrix multiplications. Thus, presenting these data types helps the DL models better understand programs. Current LLVM IRbased program representations fail to present aggregate data types appropriately. For example, consider a threedimensional integer array. In LLVM IR, this array is shown as 3 x [2 x [3 x i32]]]* . As can be seen, the length of the arrays and their data types are inferable. However, without proper representation, the DL model's capacities will be spent on learning these deterministic facts (i.e., the length of the arrays and their type). PERFOGRAPH considers aggregate data types as a new node type in its representations. Figure 4b shows how aggregate data types are supported by PERFOGRAPH. Unlike other LLVM IR-based representations, PERFOGRAPH supports multi-dimensional arrays and vectors. PERFOGRAPH creates a chain of nodes to present the different dimensions of the arrays. In Figure 4a, we see there is a node representing the three-dimensional array [3 x [2 x [3 x i32]]]* . PERFOGRAPH breaks down the corresponding node into three (since it is a threedimensional array) white nodes as shown in Figure 4b. Then, each node has a context representing that specific dimension of the array. For example, the context for the third dimension is [3 x i32] , whereas for the second dimension, the context is [2 x [3 x i32]] . For each aggregate type node, in addition to the context of the node,

alloca

alloca

i32*

store

[3 x [2 x [3 x i32]]]*

1

bitcast

call

i8*

i1

3

i8*

1

i32

i64

2

i32

(a) No support for aggregate data types.

alloca

alloca

i32*

store

[3 x i32]

1

bitcast

[2 x [3 x i32]

call

i8*

[3 x [2 x [3 x i32]]]*

i1

3

i8*

1

i32

i64

2

i32

we specifically add the length of the array and its type as additional features. For aggregate data types whose lengths are not known during compile time, we follow the LLVM conventions by considering the length of those data types as vscale . These enhancements will help the DL models to reason over the dimensions and types of aggregate data types. As a result, PERFOGRAPH will ultimately enable the DL models to have more accurate predictions for applications that deal with arrays and vectors.

5 Experimental Results and Downstream Tasks

In this section, we evaluate PERFOGRAPH on six downstream tasks. For each downstream task, we will explain the task itself, the dataset, and the baselines.

5.1 Experimental Setup

In our experiments, we use DGL's [40] RGCN [37] implementation for PERFOGRAPH representation. The graphs from PERFOGRAPH are treated as heterogeneous and managed via the HeteroGraphConv module. We use a hardware setup of two 18-core Intel Skylake 6140 CPUs and two NVIDIA Tesla V100-32GB GPUs. The embedding space for numbers is generated by extracting digits and positions from a numeric token of an IR statement, then passed to a PyTorch [31] embedding layer for digit and position embeddings. These are combined for the final numeric token embedding. Non-numeric tokens directly go through the PyTorch embedding layer. Each PERFOGRAPH heterogeneous node converts to a 120-dimensional vector via this embedding. We use the Adam [26] Optimizer, relu [1] activation function, a learning rate of 0 . 01 , and hidden_dim parameter of 60 . Mean aggregation is applied to combine different node-type results before a linear classification layer, which outputs a probability distribution for each class. The class with the highest probability is the prediction.

5.2 Device Mapping

Problem Definition: We apply PERFOGRAPH to the challenging heterogeneous device mapping [13] problem. In this task, there are a number of OpenCL kernels that we need to predict which accelerator (CPU or GPU) yields higher performance. We compare PERFOGRAPH against DeepTune [13], Inst2Vec [6], and PROGRAML [14]. The results of the baselines are quoted from [14].

Dataset: For this task, we use the dataset published in [13]. In this dataset, there are 256 OpenCL kernels available, and 680 LLVM IR instances are extracted from them. There are

% Error

0

10

20

30

40

AMD

NVIDIA

DeepTune

Inst2Vec

ProGraML

PERFOGRAPH

two types of GPUs: AMD and NVIDIA. For each of the GPUs, the runtimes of the kernels are recorded in the dataset. For AMD, 276 kernels show better performance in GPU, while 395 kernels show better performance in CPU. Whereas for NVIDIA, 385 kernels have better runtimes with GPU, and 286 kernels have better runtimes with CPU. We consider this as a binary CPU or GPU classification problem.

Results: As the dataset is small, we use the same 10-fold validation (with 80% training, 10% validation, and 10% testing) like PROGRAML [14] and chose the model with the highest validation accuracy. The hand-crafted features of [20] are also used as graph-level features in our model to enhance the performance following the approach in [14]. Table 1 and 2 show the final precision, call, f1-score, and accuracy for AMD and NVIDIA devices. Figure 5 compares PERFOGRAPH with state-of-the-art models on the Device Mapping dataset. We can see that PERFOGRAPH sets new state-of-the-art results by achieving the lowest error rate among the baselines both for AMD and NVIDIA, indicating the effectiveness of PERFOGRAPH.

5.3 Parallelism Discovery

Problem Definition: In this problem, given a sequential loop, we try to predict whether a loop can be executed in parallel. We treat this problem as a binary classification problem with two classes: Parallel and Non-Parallel.

Dataset: The OMP_Serial dataset [11] is used for this task. It contains around 6k compilable source C files with Parallel and Non-Parallel loops. The training dataset contains around 30k IR files. The OMP_Serial dataset has three test subsets to compare the performance with three traditional parallelism assistant tools: Pluto (4032 IR files), AutoPar (3356 IR files), and DiscoPoP (1226 IR files).

Results: We evaluate PERFOGRAPH on all three subsets and compare it with traditional rule-based tools: Pluto [7], AutoPar [33], DiscoPoP [29], and also Deep Learning based tools: Graph2Par [11], PROGRAML. Table 3 shows the results. The results of Pluto and Graph2par are reported from [11]. As PROGRAML does not have this downstream task in their paper, we used the PROGRAML representation in our pipeline to generate the results. Results show that traditional rule-based tools have the highest precision but the lowest accuracy because those tools are overly conservative while predicting parallel loops. So, they miss out on a lot of parallelism opportunities. PERFOGRAPH achieves considerably good precision scores across all the test subsets. In terms of accuracy, PERFOGRAPH surpasses the current state-of-the-art approaches by 2% in the Pluto and AutoPar subset. In the DiscoPoP subset, it achieves an impressive 99% accuracy and surpasses PROGRAML by 9%.

5.4 Parallel Pattern Detection

Problem Definition: Parallel loops often follow some specific patterns. Identifying parallel patterns is important because it helps developers understand how to parallelize a specific program since each parallel pattern needs to be treated differently. As a result, we apply PERFOGRAPH to identify potential parallel patterns in sequentially written programs. Only the three most common parallel patterns are considered: Do-all (Private), Reduction, and Stencil [34]. Given a loop, the task is to predict the pattern.

Dataset: For this experiment, we also use the OMP_Serial dataset [11]. This dataset contains source codes of different parallel patterns. These programs are collected from well-known benchmarks like NAS Parallel Benchmark [24], PolyBench [32], BOTS benchmark [18], and the Starbench benchmark [5]. Then, template programming packages like Jinja [35] are used to create synthetic programs from the templates collected from the mentioned benchmarks. The dataset contains 200 Do-all (Private), 200 Reduction, and 300 Stencil loops.

Results: We used 80% of the dataset for training and 20% for testing. Table 4 represents our findings. The results of Pragformer [25] and Graph2par [11] are reported from [11]. We compare with these two approaches as they are specifically developed for solving this problem. For generating the results with PROGRAML, we used the PROGRAML representation in our pipeline. We can see PERFOGRAPH achieves an impressive 99% accuracy on the OMP_Serial Parallel Pattern dataset. It surpasses the state-of-the-art PROGRAML model by 3%. This indicates the strength of PERFOGRAPH to capture the syntactic and structural patterns embedded into source programs. From Table 4, we can also see PERFOGRAPH has high precision for all three patterns and achieves a high precision score for Do-all and Stencil patterns while maintaining very good accuracy.

5.5 NUMAand Prefetchers Configuration Prediction

Problem Definition: An appropriate configuration of Non-Uniform Memory Access (NUMA) and hardware prefetchers significantly impacts program performance. In this experiment, we define the task of NUMA and prefetcher selection as predicting the right configuration within a given tuning parameter search space. We evaluate the performance of both PROGRAML and PERFOGRAPH for this task by converting each program in the dataset to PROGRAML and PERFOGRAPH graphs following the approach in [38].

Fold #

% Error

0.0

0.1

0.2

0.3

0

1

2

3

4

5

6

7

8

9

avg

ProGraML

PERFOGRAPH

(a) Error distribution for Sandy Bridge architecture.

Fold #

% Error

0.0

0.1

0.2

0.3

0.4

0

1

2

3

4

5

6

7

8

9

avg

ProGraML

PERFOGRAPH

(b) Error distribution for Skylake architecture.

Dataset: We use the dataset in [38], which includes a diverse set of intermediate representation files coupled with the optimal configuration [36]. The dataset incorporates various LLVM compiler optimization flags to produce different forms of the same program. There are 57 unique kernels (IR files) in this dataset, and around 1000 optimization flags are applied, resulting in 57000 IR files in total. Each IR file within the dataset is accompanied by its runtime on two architectures, Sandy Bridge and Skylake, across thirteen different NUMA and prefetcher configurations.

Results: Following the approach in the study of TehraniJamsaz et al. , we partition the dataset into ten folds for cross-validation. Figure 6a and 6b illustrate the performance results in terms of error rates. On average, PERFOGRAPH outperforms PROGRAML by achieving 3.5% and 1.8% better error rates on average for the Sandy Bridge and Skylake architecture, respectively. These improvements demonstrate the effectiveness of PERFOGRAPH compared to the state-of-the-art PROGRAML.

5.6 Thread Coarsening Factor (TCF) Prediction

Problem Definition: Thread coarsening is an optimization technique for parallel programs by fusing the operation of two or more threads together. The number of threads that can be fused together is known as the Thread Coarsening Factor (TCF). For a given program, the task is to predict the coarsening factor value (1, 2, 4, 8, 16, 32) that leads to the best runtime. The running time with coarsening factor 1 is used as the baseline for calculating speedups. For this task, we compare PERFOGRAPH against DeepTune [13], Inst2Vec [6] and PROGRAML [14]. The results of the baselines are quoted from [6]. Since PROGRAML has not been evaluated on this task in the past, we apply PROGRAML representation in our setup for comparison.

Dataset: We use the dataset of Ben-Nun et al. [13]. The dataset contains only 17 OpenCL kernels. For each kernel, the dataset has the runtime information on four different GPUs for the different thread coarsening factor values. Hence, for each kernel, we have the runtime corresponding to each thread coarsening factor value on a specific GPU device.

Results: we design the problem as a multi-class classification problem where, given a kernel, we try to predict which thread coarsening factor

Kernels with correct TCF predicted

0

2

4

6

Radeon HD 5900

Tahiti 7970

Tesla K20c

ProGraML

PERFOGRAPH

provides the highest performance. As the dataset is very small, we apply a 17-fold cross-validation approach. In each fold, we train our model on 16 data points, and the model is tested on the one unseen data point that is left out of the training set. Figure 7 shows the comparison of kernels with the correct Thread Coarsening Factor (TCF) found by PROGRAML and PERFOGRAPH. Across the four platforms in total PERFOGRAPH is able to correctly predict the TCF for 17 cases, whereas PROGRAML is able to find only 9 cases. In two of the platforms (AMD Radeon HD 5900 and NVIDIA GTX 480) where PROGRAML failed to find any kernel with the correct TCF, PERFOGRAPH can find three kernels in both of the platforms with the correct TCF value. As shown in 5, even though PERFOGRAPH outperforms PROGRAML on most computing platforms, it falls behind inst2vec. We posit the reason is that inst2vec has a pretraining phase where it is trained using skip-gram. On the other hand, 17 kernels are very small. Therefore, a DL-based model is not able to generalize enough. However, we can see that even on a smaller dataset, PERFOGRAPH achieved comparable speedups with respect to the current state-of-the-art models.

5.7 Algorithm Classification

Problem Definition: Previous downstream tasks showed that in most of the cases, PERFOGRAPH outperforms the baselines. Those tasks were mostly performance-oriented. We go further by applying PERFOGRAPH on a different downstream task, which is algorithm classification. The task involves classifying a source code into 1 of 104 classes. In this task, we compare the results of PERFOGRAPH to those of inst2vec, PROGRAML. The results of the baselines are quoted from [14].

Dataset: We use the POJ-104 dataset [30] in a similar setup as [14] that contains around 240k IR files for training and 10k files for testing.

Results: For this task, inst2vec has error rate of 5.17, whereas PROGRAML has error rate of 3.38. PERFOGRAPH yields an error rate of 5.00, which is better than inst2vec and slightly behind PROGRAML. One of the reasons is that PROGRAML already has a very small error rate in this task, leaving a very small gap for improvement; however still PERFOGRAPH's result is very close to that of PROGRAML. We could not reproduce the results in PROGRAML paper in our setup. When we applied PROGRAML in our setup, the error rate of PROGRAML was 6.00. Moreover, we posit that for algorithm classification, numbers are not a significant factor. Therefore, numerical awareness can confuse the models a little bit. However, this experiment shows that PERFOGRAPH is very close to PROGRAML's performance in this task and shows the applicability of PERFOGRAPH to a wider range of downstream tasks.

5.8 Ablation Study

We further analyzed how each one of the enhancements in PERFOGRAPH affects the end results. We performed an ablation study on the Device Mapping task and trained our GNN models on variations of PERFOGRAPH.

Results without aggregate data type nodes: First, aggregate data type nodes are removed from PERFOGRAPH representation. Please note that in this setup, the Digit Embedding is still applied. Table 6 and 7 shows the results. It can be seen that when the representation does not support aggregate data type, the error rate increases to 13% in AMD and 15% in the NVIDIA dataset. This clearly indicates that having aggregate-type nodes in the representation helped the model to learn the code features more accurately.

Results without Digit Embedding: Then Digit Embedding is removed from the pipeline for the second experiment and the aggregate data type nodes are kept. Table 6 and 7 shows the results. We can see that unlike having aggregate data type nodes, removing digit embedding does not hurt the error rate that much for the task of device mapping. However, we can still see a small increase (1.1%) in the error rate for the AMD dataset. For the NVIDIA dataset, the error rate increases from 10.0 to 10.6%.

Finally, we can conclude that both components in our representation helped the model to learn the code features better to some extent. However, aggregate data type nodes in the embedding helped our model more than Digit Embedding for the task of device mapping. The reason can be that there are not many numbers in the dataset. However, in tasks where there are many numbers, Digit Embedding can play a significant role.

6 Conclusion and Future Work

In this paper, we presented PERFOGRAPH, an LLVM IR-based graph representation of programs that supports aggregate data types such as arrays and vectors and is numerical aware. Moreover, it addresses several limitations of the previous IR-based graph representations. PERFOGRAPH is evaluated on various downstream tasks, and experimental results indicate that PERFOGRAPH is indeed effective, outperforming state-of-the-art in most of the downstream tasks. PERFOGRAPH numerical awareness capability is limited to the numerical values that are available at the compile time. For future work, we intend to augment our representation by adding support for dynamic information and checking the possibility of integrating hardware performance counters with our representation. Moreover, we plan to develop a pre-trained embedding model using our representation. Having a pre-trained model will help to solve the problem of limited training samples in some downstream tasks.

References

A. F. Agarap. Deep learning using rectified linear units (relu). arXiv preprint arXiv:1803.08375 , 2018.

M. Allamanis. Graph neural networks in program analysis. Graph Neural Networks: Foundations, Frontiers, and Applications , pages 483-497, 2022.

M. Allamanis, M. Brockschmidt, and M. Khademi. Learning to represent programs with graphs. arXiv preprint arXiv:1711.00740 , 2017.

M. Allamanis, E. T. Barr, P. Devanbu, and C. Sutton. A survey of machine learning for big code and naturalness. ACM Computing Surveys (CSUR) , 51(4):1-37, 2018.

M. Andersch, B. Juurlink, and C. Chi. A benchmark suite for evaluating parallel programming models. In Proceedings of Workshop on Parallel Systems and Algorithms (PARS) , volume 28, pages 1-6, 2013.

T. Ben-Nun, A. S. Jakobovits, and T. Hoefler. Neural code comprehension: A learnable representation of code semantics. Advances in Neural Information Processing Systems , 31, 2018.

U. Bondhugula, A. Hartono, J. Ramanujam, and P. Sadayappan. A practical automatic polyhedral parallelizer and locality optimizer. SIGPLAN Not. , 43(6):101-113, jun 2008. ISSN 0362-1340. doi: 10.1145/1379022.1375595. URL https://doi.org/10.1145/1379022.1375595 .

A. Brauckmann, A. Goens, S. Ertel, and J. Castrillon. Compiler-based graph representations for deep learning models of code. In Proceedings of the 29th International Conference on Compiler Construction , pages 201-211, 2020.

L. Chen, W. Ye, and S. Zhang. Capturing source code semantics via tree-based convolution over apienhanced ast. In Proceedings of the 16th ACM International Conference on Computing Frontiers , pages 174-182, 2019.

L. Chen, Q. I. Mahmud, and A. Jannesari. Multi-view learning for parallelism discovery of sequential programs. In 2022 IEEE International Parallel and Distributed Processing Symposium Workshops (IPDPSW) , pages 295-303. IEEE, 2022.

L. Chen, Q. I. Mahmud, H. Phan, N. Ahmed, and A. Jannesari. Learning to parallelize with openmp by augmented heterogeneous ast representation. Proceedings of Machine Learning and Systems , 5, 2023.

Z. Chen and M. Monperrus. A literature study of embeddings on source code. arXiv preprint arXiv:1904.03061 , 2019.

C. Cummins, P. Petoumenos, Z. Wang, and H. Leather. End-to-end deep learning of optimization heuristics. In 2017 26th International Conference on Parallel Architectures and Compilation Techniques (PACT) , pages 219-232. IEEE, 2017.

C. Cummins, Z. V. Fisches, T. Ben-Nun, T. Hoefler, and H. Leather. Programl: Graph-based deep learning for program optimization and analysis. arXiv preprint arXiv:2003.10536 , 2020.

C. Cummins, B. Wasti, J. Guo, B. Cui, J. Ansel, S. Gomez, S. Jain, J. Liu, O. Teytaud, B. Steiner, et al. Compilergym: Robust, performant compiler optimization environments for ai research. In 2022 IEEE/ACM International Symposium on Code Generation and Optimization (CGO) , pages 92-105. IEEE, 2022.

H. K. Dam, T. Tran, and T. Pham. A deep language model for software code. arXiv preprint arXiv:1608.02715 , 2016.

H. K. Dam, T. Pham, S. W. Ng, T. Tran, J. Grundy, A. Ghose, T. Kim, and C.-J. Kim. Lessons learned from using a deep tree-based model for software defect prediction in practice. In 2019 IEEE/ACM 16th International Conference on Mining Software Repositories (MSR) , pages 46-57. IEEE, 2019.

A. Duran, X. Teruel, R. Ferrer, X. Martorell, and E. Ayguade. Barcelona openmp tasks suite: A set of benchmarks targeting the exploitation of task parallelism in openmp. In 2009 international conference on parallel processing , pages 124-131. IEEE, 2009.

Z. Feng, D. Guo, D. Tang, N. Duan, X. Feng, M. Gong, L. Shou, B. Qin, T. Liu, D. Jiang, et al. Codebert: A pre-trained model for programming and natural languages. arXiv preprint arXiv:2002.08155 , 2020.

D. Grewe, Z. Wang, and M. F. O'Boyle. Portable mapping of data parallel programs to opencl for heterogeneous systems. In Proceedings of the 2013 IEEE/ACM International Symposium on Code Generation and Optimization (CGO) , pages 1-10. IEEE, 2013.

X. Gu, H. Zhang, D. Zhang, and S. Kim. Deep api learning. In Proceedings of the 2016 24th ACM SIGSOFT international symposium on foundations of software engineering , pages 631-642, 2016.

D. Guo, S. Ren, S. Lu, Z. Feng, D. Tang, S. Liu, L. Zhou, N. Duan, A. Svyatkovskiy, S. Fu, et al. Graphcodebert: Pre-training code representations with data flow. arXiv preprint arXiv:2009.08366 , 2020.

D. Guo, S. Lu, N. Duan, Y. Wang, M. Zhou, and J. Yin. Unixcoder: Unified cross-modal pre-training for code representation. arXiv preprint arXiv:2203.03850 , 2022.

H.-Q. Jin, M. Frumkin, and J. Yan. The openmp implementation of nas parallel benchmarks and its performance. 1999.

T. Kadosh, N. Hasabnis, T. Mattson, Y. Pinter, G. Oren, et al. Pragformer: Data-driven parallel source code classification with transformers. 2023.

D. P. Kingma and J. Ba. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980 , 2014.

D. Levy and L. Wolf. Learning to align the source code to the compiled object code. In International Conference on Machine Learning , pages 2043-2051. PMLR, 2017.

Y. Li, C. Gu, T. Dullien, O. Vinyals, and P. Kohli. Graph matching networks for learning the similarity of graph structured objects. In International conference on machine learning , pages 3835-3845. PMLR, 2019.

Z. Li, R. Atre, Z. U. Huda, A. Jannesari, and F. Wolf. Unveiling parallelization opportunities in sequential programs. Journal of Systems and Software , 117:282-295, July 2016. doi: 10.1016/j.jss.2016.03.045.

L. Mou, G. Li, L. Zhang, T. Wang, and Z. Jin. Convolutional neural networks over tree structures for programming language processing. In Proceedings of the AAAI conference on artificial intelligence , volume 30, 2016.

A. Paszke, S. Gross, F. Massa, A. Lerer, J. Bradbury, G. Chanan, T. Killeen, Z. Lin, N. Gimelshein, L. Antiga, A. Desmaison, A. Kopf, E. Yang, Z. DeVito, M. Raison, A. Tejani, S. Chilamkurthy, B. Steiner, L. Fang, J. Bai, and S. Chintala. Pytorch: An imperative style, highperformance deep learning library. In Advances in Neural Information Processing Systems 32 , pages 8024-8035. Curran Associates, Inc., 2019. URL http://papers.neurips.cc/paper/ 9015-pytorch-an-imperative-style-high-performance-deep-learning-library.pdf .

L.-N. Pouchet and T. Yuki. Polybench: The polyhedral benchmark suite (version 4.2), 2017.

D. Quinlan and C. Liao. The ROSE source-to-source compiler infrastructure. In Cetus users and compiler infrastructure workshop, in conjunction with PACT , volume 2011, page 1. Citeseer, 2011.

J. Reinders, B. Ashbaugh, J. Brodman, M. Kinsner, J. Pennycook, X. Tian, J. Reinders, B. Ashbaugh, J. Brodman, M. Kinsner, et al. Common parallel patterns. Data Parallel C++ Mastering DPC++ for Programming of Heterogeneous Systems using C++ and SYCL , pages 323-352, 2021.

A. Ronacher. Jinja2 documentation. Welcome to Jinja2-Jinja2 Documentation (2.8-dev) , 2008.

I. Sánchez Barrera, D. Black-Schaffer, M. Casas, M. Moretó, A. Stupnikova, and M. Popov. Modeling and optimizing numa effects and prefetching with machine learning. In Proceedings of the 34th ACM International Conference on Supercomputing , pages 1-13, 2020.

M. Schlichtkrull, T. N. Kipf, P. Bloem, R. Van Den Berg, I. Titov, and M. Welling. Modeling relational data with graph convolutional networks. In The Semantic Web: 15th International Conference, ESWC 2018, Heraklion, Crete, Greece, June 3-7, 2018, Proceedings 15 , pages 593-607. Springer, 2018.

A. TehraniJamsaz, M. Popov, A. Dutta, E. Saillard, and A. Jannesari. Learning intermediate representations using graph neural networks for numa and prefetchers optimization. In 2022 IEEE International Parallel and Distributed Processing Symposium (IPDPS) , pages 1206-1216. IEEE, 2022.

S. VenkataKeerthy, R. Aggarwal, S. Jain, M. S. Desarkar, R. Upadrasta, and Y. Srikant. Ir2vec: Llvm ir based scalable program embeddings. ACM Transactions on Architecture and Code Optimization (TACO) , 17(4):1-27, 2020.

M. Y. Wang. Deep graph library: Towards efficient and scalable deep learning on graphs. In ICLR workshop on representation learning on graphs and manifolds , 2019.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper proposed a novel graph-based program representation, PerfoGraph, which is based on the current state-of-the-art method PrograML and aims to address its limitation by providing numerical awareness, introducing new tactics for handling local variables, and supporting aggregate data types. By conducting experiments on various performance-optimizing oriented downstream tasks, it is shown that the proposed PerfoGraph representation is more effective compared to the existing methods.\n\nSTRENGTHS:\n- The paper was clearly presented overall.\n- The paper showed great originality in proposing a novel program representation method, PerfoGraph, that enhances previous results by providing numerical awareness, introducing new tactics for handling local variables, and supporting aggregate data types.\n- Regarding numerical awareness, the authors introduced a novel idea to split the numbers into digits and positions before encoding, which could effectively represent numbers without a huge vocabulary.\n- Regarding local variables, the author took good consideration of GNN's mechanism and adjusted the graph representation accordingly.\n\nWEAKNESSES:\n- Unconvincing experiment setup in comparison with PrograML.\nIn the original PrograML paper, MPNN was introduced as an encoder. However, the proposed PerfoGraph chose RGCN for the same role. The reviewer believes that it is possible to also use RGCN as PrograML's encoder. Thus, it is unclear whether the performance improvements in the comparisons with PorgraML stem from the novel graph representation or from a more suitable encoder.\n\nQUESTIONS:\n1. The authors claimed that the proposed PerfoGraph representation could better capture **composite** data structure information. However, the paper only introduced a novel method for incorporating list-like data structures, such as **arrays and vectors**, into the graph representation of the program. The reviewer's question is how PerfoGraph could support more general composite data types such as struct. From the current presentation, the reviewer's opinion is that the term **aggregate data types** suit better.\n2. How does the choice of aggregation function impact the performance of PerfoGraph? Is it possible that the information compression introduced in the aggregation function confuses the deep learning model? \n3. Do the authors consider moving the part of the ablation study from Supplementary Materials to the main text to better clarify the importance of numerical awareness in program representations?\n4. Possible error in Figure 2a: In section 4.1, the authors used Figure 2a as an example of the PrograML representation to point out that PrograML assigns a position feature to the address edge of the store node. However, there is no corresponding label near the first store node.\n\nLIMITATIONS:\nThe authors have addressed the limitations of their work in Section 6.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 5, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-06-28"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper presents a graph-based representation for LLVM IR programs for processing with graph neural networks. The representation is based on ProGraML, an established technique for such graph representations, but adds several features to the graph: collapsing nodes that refer to the same variable, adding additional edges between memory stores and the variables they can affect, explicitly representing constants in the graph, and representing compositional data types (e.g., arrays of arrays) as multiple distinct nodes rather than single nodes. The authors evaluate across a range of LLVM IR learning tasks, and show that Perfograph outperforms ProGraML (and all other baselines in most contexts).\n\nSTRENGTHS:\n* Finding and understanding good program representations is an important problem in the field of learning for code\n* The proposed approach identifies a novel set of features which qualitatively seem important, and which are validated to be useful in the ablation study\n* Assuming that the evaluation is indeed fair (see weaknesses below), the proposed approach significantly outperforms ProGraML, achieving a large increase in accuracy (seemingly solving many of these tasks near perfectly).\n\nWEAKNESSES:\n* Fairness of comparison against baselines:\n  * Perfograph is a version of ProGraML with some added and removed nodes and edges. Each added node/edge adds additional power for a graph neural network. For a fair comparison against baselines (ProGraML in particular), the authors must show that for a given compute budget (i.e., FLOPs or wall clock time), Perfograph outperforms ProGraML; this would involve either scaling up/down ProGraML or Perfograph.\n  * Are the datasets used exactly the same as in prior work? For example, Section 5.4 reports using an 80/20 train/test split, which I cannot find discussed in [10]\n  * Line 372 mentions that the authors \"could not reproduce the results in PROGRAML paper\" for the algorithm classification task. Does this lack of reproducibility also affect the evaluated benchmarks that were not in the ProGraML paper (seemingly, everything other than 5.2)? If so, this is a major point that must be discussed at a high level in the paper.\n  * Given the advent of large-context Transformers, I would be curious how well such a model performs when applied to the raw IR (i.e., without an explicitly featurized graph representation). However, since this paper is explicitly building on the prior work of ProGraML, I do not consider this to be a condition for acceptance.\n* Overall, the clarity of the submission is moderate.\n  * Section 4.1: without a deep understanding of the ProGraML representation, it is very hard to understand these graphs (and therefore the issues that the proposed approach fixes in them). Perhaps highlighting or otherwise identifying the specific nodes discussed could help.\n  * Line 323: is this an arithmetic or geometric average?\n  * Line 368: what are the units of these errors?\n* Several claims about novelty or related work are misleading:\n  * Lines 72-73: \"this token-based representation... fails to capture...\" This claim is misleading. Sequence processing models are entirely capable of learning such graph-structured relationships; such complex relationships are also not unique to code (e.g., natural language has pronouns, nested structures, etc.)\n  * Line 176: the proposed \"Digit Embedding\" is very similar to some digit representations approaches used in Transformers (for example, Geva et al. 2020) which embed each digit, add/concatenate with a positional encoding, and aggregate with attention.\n* I believe the paper would be significantly stronger with the ablation studies moved to the main body, but this is a stylistic choice and is not a factor in my score\n* If I'm understanding Appendix 2 correctly, the Perfograph GNN is evaluated with only 2 layers; ProGraML is evaluated with 6 (Cummins et al., Section 6.2). How was this parameter chosen? What is the average diameter of the evaluated graphs (for both Perfograph's graphs, and ProGraML's graphs)?\n\nQUESTIONS:\n* Please see the questions in \"clarity\" in the discussion of weaknesses above (and other questions throuhgout).\n* Does Perfograph use more compute or have more capacity than ProGraML? Does it still outperform ProGraML when given equal power?\n* What steps have the authors taken to ensure that the implementation of ProGraML compared against in Sections 5.3-5.7 is correct (since as noted in Line 372, in one context the authors \"could not reproduce the results in PROGRAML paper\")?\n* Do all other baselines use the exact same training/test set split?\n* How were hyperparameters tuned for Perfograph?\n\nI would be willing to raise my score given evidence that for the same compute budget, Perfograph still outperforms ProGraML (or evidence that my assumption here, that Perfograph uses more compute than ProGraML, is incorrect).\n\nLIMITATIONS:\nThe paper does not explicitly discuss limitations of the technique.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2023-07-02"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis work proposes Perfograph, a program graph representation based on LLVM-IR and an extension to ProGraML. This graph representation is designed for the purpose of performance optimization and program analysis applications based on graph neural networks (GNN). This work made three contributions to the existing representation. First, it tracks reused local identifiers and memory locations. Second, it uses decimal-based encoding to embed numerical constants into the graph. Last, it breaks down array and vector types into multiple nodes. Perfograph is tested on 6 downstream tasks: device mapping, parallelism discovery, parallel pattern detection, NUMA and prefetchers configuration prediction, thread coarsening factor (TCF) prediction, and algorithm classification. In the device mapping challenge, the new model outperforms ProGraML by 7.4% on the AMD dataset and 10% on the NVIDIA dataset.\n\nSTRENGTHS:\n1. This paper is well organized.\n2. This work clearly listed all 3 of its design decisions.\n3. This work contains detailed evaluations on multiple targets.\n4. Each evaluation contains a carefully described problem definition, dataset, and results.\n\nWEAKNESSES:\n1. The improvement of this work over ProGraML seems limited. For example, the 3 main design contributions are not fundamentally new. They are more like minor tweaks on the existing ProGraML system.\n2. The evaluation section does not show how each design choice contributes to the overall improvement.\n\nQUESTIONS:\n1. It claims that “Perfograph is built on top of ProGraML” in Section 4, but also claims “A **new** compiler and language agnostic program representation based on LLVM-IR”. Is this work an extension of ProGraML, or is it a new system? This is somewhat unclear.\n2. Section 4.1 claims that this work combines multiple writes to the same variable into one node. However, LLVM deliberately separates multiple writes for the ease of certain aliasing analyses, the opportunities of reordering, and optimizing for parallelism. It would be helpful to provide more evidence that this combined representation outperforms separated representation in all (or most) downstream tasks.\n3. Section 4.2 mentions that decimal-based encoding is used for numerical constant embedding. This seems counterintuitive since computers natively speak binary.\n4. Section 4.3 claims that Perfograph supports composite data types (but only covers arrays and vectors). It would be helpful to further explain how Perfograph deals with programs that use structs (and maybe unions).\n\nLIMITATIONS:\nN/A", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-07"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe research identifies limitations in the current state-of-the-art program representation PROGRAML, in capturing features of numerical values and composite data types. \nTo address these limitations, this work introduces an enhanced GNN-based program representation for LLMV-IR with modifications to the nodes and edges of the program graphs to capture features of numerical values and composite data structures. It also presents a digit embedding approach to enhance numerical awareness within the representation. \nThe evaluation shows that PERFOGRAPH reduced the prediction error rate by 7.4% (AMD dataset) and 10% (NVIDIA dataset) in the Device Mapping challenge compared to PROGRAML. Additionally, it outperforms PROGRAML and traditional rule-based methods in tasks such as determining if a loop is parallelizable, classifying the parallel patterns, and configuring NUMA and prefetchers selections.\n\nSTRENGTHS:\n1. PERFOGRAPH employs effective graph modifications to enhance the learnability of the LLVM-IR graph for GNNs. These methods are well-reasoned and provide valuable domain information for improving program graph abstractions in learning.\nFor instance, PERFOGRAPH unifies the graph nodes for the same local identifier variable in the program graph. It also modifies the LLVM IR graph to add edges from the store instruction to the variables it modified. These modifications all simplify the complex structure of the LLVM-IR graph.\n\n2. PERFOGRAPH also distinguishes between composite data structures and regular data types, which prior approaches did not adequately address. This distinction is crucial because operations involving composite data structures can differ significantly from those involving regular data types.\n\n3. The evaluation of PERFOGRAPH is comprehensive, considering six performance-oriented tasks. The results demonstrate that PERFOGRAPH outperforms PROGRAML in terms of accuracy on these tasks. More importantly, in parallelism discovery tasks, PERFOGRAPH achieves higher accuracy compared to traditional rule-based approaches.\n\nWEAKNESSES:\n1. It is unclear to me how the digit embedding ensures that unknown numbers are not encountered during the test phase and what specific information is encoded through the addition of digit and position embeddings.\n2. The reasons behind representing multi-dimensional arrays as multiple nodes are not adequately explained. It remains unclear how this representation enhances expressiveness or facilitates reasoning by GNNs.\n3. An ablation study that isolates the individual benefits of LLVM-IR graph modification, digit embedding, and support for composite types is absent. Such a study would provide valuable insights into understanding the specific advantages offered by each component.\n4. PERFOGRAPH may have higher computational requirements compared to PROGRAML, potentially posing challenges for its practical deployment.\n\nQUESTIONS:\n1. It would be helpful to visualize the digit embedding values of different numerical values to facilitate better understanding.\n2. It would be beneficial to compare the cascaded context approach for representing multi-node representation with alternative representations (e.g. storing only the size of one rank) to assess the effectiveness in encoding array information.\n3. To provide a more comprehensive analysis, I recommend conducting an ablation study for each major representation enhancement to understand the specific benefits of each component.\n4. It would be valuable to measure and compare the inference time of PERFOGRAPH with PROGRAML and other rule-based analysis methods to evaluate the computational efficiency.\n\nLIMITATIONS:\nN/A", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-07"}