SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Problem Definition: Parallel loops often follow some specific patterns. Identifying parallel patterns is important because it helps developers understand how to parallelize a specific program since each parallel pattern needs to be treated differently. As a result, we apply PERFOGRAPH to identify potential parallel patterns in sequentially written programs. Only the three most common parallel patterns are considered: Do-all (Private), Reduction, and Stencil [34]. Given a loop, the task is to predict the pattern.
Dataset: For this experiment, we also use the OMP_Serial dataset [11]. This dataset contains source codes of different parallel patterns. These programs are collected from well-known benchmarks like NAS Parallel Benchmark [24], PolyBench [32], BOTS benchmark [18], and the Starbench benchmark [5]. Then, template programming packages like Jinja [35] are used to create synthetic programs from the templates collected from the mentioned benchmarks. The dataset contains 200 Do-all (Private), 200 Reduction, and 300 Stencil loops.
Results: We used  $80\%$  of the dataset for training and  $20\%$  for testing. Table 4 represents our findings. The results of Pragformer [25] and Graph2par [11] are reported from [11]. We compare with these two approaches as they are specifically developed for solving this problem. For generating the results with PROGRAML, we used the PROGRAML representation in our pipeline. We can see PERFOGRAPH achieves an impressive  $99\%$  accuracy on the OMP_Serial Parallel Pattern dataset. It surpasses the state-of-the-art PROGRAML model by  $3\%$ . This indicates the strength of PERFOGRAPH to capture the syntactic and structural patterns embedded into source programs. From Table 4, we can also see PERFOGRAPH has high precision for all three patterns and achieves a high precision score for Do-all and Stencil patterns while maintaining very good accuracy.
Table 4: Performance comparison for the parallel pattern detection task with PERFOGRAPH on the OMP_Serial Dataset.
Approach	Pattern	Precision	Recall	F1-score	Accuracy
Pragformer	Do-all	0.86	0.85	0.86
Reduction	0.89	0.87	0.87	0.86
Stencil	N/A	N/A	N/A
Graph2Par	Do-all	0.88	0.87	0.87
Reduction	0.9	0.89	0.91	0.9
Stencil	N/A	N/A	N/A
PROGRAML	Do-all	0.92	0.90	0.91
Reduction	0.92	0.92	0.92	0.96
Stencil	0.98	1	0.99
PERFOGRAPH	Do-all	1	0.97	0.99
Reduction	0.97	1	0.99	0.99
Stencil	1	1	1
5.5 NUMA and Prefetchers Configuration Prediction
Problem Definition: An appropriate configuration of Non-Uniform Memory Access (NUMA) and hardware prefetchers significantly impacts program performance. In this experiment, we define the task of NUMA and prefetcher selection as predicting the right configuration within a given tuning parameter search space. We evaluate the performance of both PROGRAMML and PERFOGRAPH for this task by converting each program in the dataset to PROGRAML and PERFOGRAPH graphs following the approach in [38].
(a) Error distribution for Sandy Bridge architecture.
(b) Error distribution for Skylake architecture.
Figure 6: Breakdown of the NUMA and prefetchers configuration prediction per fold [lower is better].
Dataset: We use the dataset in [38], which includes a diverse set of intermediate representation files coupled with the optimal configuration [36]. The dataset incorporates various LLVM compiler optimization flags to produce different forms of the same program. There are 57 unique kernels (IR files) in this dataset, and around 1000 optimization flags are applied, resulting in 57000 IR files in total. Each IR file within the dataset is accompanied by its runtime on two architectures, Sandy Bridge and Skylake, across thirteen different NUMA and prefetcher configurations.
Results: Following the approach in the study of TehraniJamsaz et al., we partition the dataset into ten folds for cross-validation. Figure 6a and 6b illustrate the performance results in terms of error rates. On average, PERFOGRAPH outperforms PROGRAML by achieving  $3.5\%$  and  $1.8\%$  better error rates on average for the Sandy Bridge and Skylake architecture, respectively. These improvements demonstrate the effectiveness of PERFOGRAPH compared to the state-of-the-art PROGRAML.
5.6 Thread Coarsening Factor (TCF) Prediction
Problem Definition: Thread coarsening is an optimization technique for parallel programs by fusing the operation of two or more threads together. The number of threads that can be fused together is known as the Thread Coarsening Factor (TCF). For a given program, the task is to predict the coarsening factor value (1, 2, 4, 8, 16, 32) that leads to the best runtime. The running time with coarsening factor 1 is used as the baseline for calculating speedups. For this task, we compare PERFOGRAPH against DeepTune [13], Inst2Vec [6] and PROGRAMML [14]. The results of the baselines are quoted from [6]. Since PROGRAMML has not been evaluated on this task in the past, we apply PROGRAMML representation in our setup for comparison.
Dataset: We use the dataset of Ben-Nun et al. [13]. The dataset contains only 17 OpenCL kernels. For each kernel, the dataset has the runtime information on four different GPUs for the different thread coarsening factor values. Hence, for each kernel, we have the runtime corresponding to each thread coarsening factor value on a specific GPU device.
Results: we design the problem as a multi-class classification problem where, given a kernel, we try to predict which thread coarsening factor
Figure 7: Correct TCF found by PROGRAML vs PERFOGRAPH [higher is better].
provides the highest performance. As the dataset is very small, we apply a 17-fold cross-validation approach. In each fold, we train our model on 16 data points, and the model is tested on the one unseen data point that is left out of the training set. Figure 7 shows the comparison of kernels with the correct Thread Coarsening Factor (TCF) found by PROGRAML and PERFOGRAPH. Across the four platforms in total PERFOGRAPH is able to correctly predict the TCF for 17 cases, whereas PROGRAMML is able to find only 9 cases. In two of the platforms (AMD Radeon HD 5900 and NVIDIA GTX 480) where PROGRAMML failed to find any kernel with the correct TCF, PERFOGRAPH can find three kernels in both of the platforms with the correct TCF value. As shown in 5, even though PERFOGRAPH outperforms PROGRAML on most computing platforms, it falls behind inst2vec. We posit the reason is that inst2vec has a pretraining phase where it is trained using skip-gram. On the other hand, 17 kernels are very small. Therefore, a DL-based model is not able to generalize enough. However, we can see that even on a smaller dataset, PERFOGRAPH achieved comparable speedups with respect to the current state-of-the-art models.
Table 5: Speedups achieved by coarsening threads
Computing Platform	DeepTune	inst2vec	PROGRAML	PERFOGRAPH
AMD Radeon HD 5900	1.1	1.37	1.15	1.19
AMD Tahiti 7970	1.05	1.1	1.00	1.14
NVIDIA GTX 480	1.1	1.07	0.98	1.03
NVIDIA Tesla K20c	0.99	1.06	1.03	1.01
5.7 Algorithm Classification
Problem Definition: Previous downstream tasks showed that in most of the cases, PERFOGRAPH outperforms the baselines. Those tasks were mostly performance-oriented. We go further by applying PERFOGRAPH on a different downstream task, which is algorithm classification. The task involves classifying a source code into 1 of 104 classes. In this task, we compare the results of PERFOGRAPH to those of inst2vec, PROGRAML. The results of the baselines are quoted from [14].
Dataset: We use the POJ-104 dataset [30] in a similar setup as [14] that contains around 240k IR files for training and 10k files for testing.
Results: For this task, inst2vec has error rate of 5.17, whereas PROGRAMML has error rate of 3.38. PERFOGRAPH yields an error rate of 5.00, which is better than inst2vec and slightly behind PROGRAMML. One of the reasons is that PROGRAMML already has a very small error rate in this task, leaving a very small gap for improvement; however still PERFOGRAPH's result is very close to that of PROGRAMML. We could not reproduce the results in PROGRAMML paper in our setup. When we applied PROGRAMML in our setup, the error rate of PROGRAMML was 6.00. Moreover, we posit that for algorithm classification, numbers are not a significant factor. Therefore, numerical awareness can confuse the models a little bit. However, this experiment shows that PERFOGRAPH is very close to PROGRAMML's performance in this task and shows the applicability of PERFOGRAPH to a wider range of downstream tasks.
5.8 Ablation Study
We further analyzed how each one of the enhancements in PERFOGRAPH affects the end results. We performed an ablation study on the Device Mapping task and trained our GNN models on variations of PERFOGRAPH.
Results without aggregate data type nodes: First, aggregate data type nodes are removed from PERFOGRAPH representation. Please note that in this setup, the Digit Embedding is still applied. Table 6 and 7 shows the results. It can be seen that when the representation does not support aggregate data type, the error rate increases to  $13\%$  in AMD and  $15\%$  in the NVIDIA dataset. This clearly indicates that having aggregate-type nodes in the representation helped the model to learn the code features more accurately.
Results without Digit Embedding: Then Digit Embedding is removed from the pipeline for the second experiment and the aggregate data type nodes are kept. Table 6 and 7 shows the results. We can see that unlike having aggregate data type nodes, removing digit embedding does not hurt the error rate that much for the task of device mapping. However, we can still see a small increase (1.1%) in the error rate for the AMD dataset. For the NVIDIA dataset, the error rate increases from 10.0 to 10.6%.
Table 6: Summarizing PERFOGRAPH results for AMD device.
Approach	Error (%)
DeepTune [13]	28.1
inst2vec [6]	19.7
PROGRAML [14]	13.4
PERFOGRAPH (without aggregate data type nodes)	13.0
PERFOGRAPH (without digit embedding)	7.1
PERFOGRAPH (aggregate data type nodes + digit embedding)	6.0
Table 7: Summarizing PERFOGRAPH results for NVIDIA device.
Approach	Error (%)
DeepTune [13]	39.0
inst2vec [6]	21.5
PROGRAML [14]	20.0
PERFOGRAPH (without aggregate data type nodes)	15.0
PERFOGRAPH (without digit embedding)	10.6
PERFOGRAPH (aggregate data type nodes + digit embedding)	10.0
Finally, we can conclude that both components in our representation helped the model to learn the code features better to some extent. However, aggregate data type nodes in the embedding helped our model more than Digit Embedding for the task of device mapping. The reason can be that there are not many numbers in the dataset. However, in tasks where there are many numbers, Digit Embedding can play a significant role.
In this paper, we presented PERFOGRAPH, an LLVM IR-based graph representation of programs that supports aggregate data types such as arrays and vectors and is numerical aware. Moreover, it addresses several limitations of the previous IR-based graph representations. PERFOGRAPH is evaluated on various downstream tasks, and experimental results indicate that PERFOGRAPH is indeed effective, outperforming state-of-the-art in most of the downstream tasks. PERFOGRAPH numerical awareness capability is limited to the numerical values that are available at the compile time. For future work, we intend to augment our representation by adding support for dynamic information and checking the possibility of integrating hardware performance counters with our representation. Moreover, we plan to develop a pre-trained embedding model using our representation. Having a pre-trained model will help to solve the problem of limited training samples in some downstream tasks.
This project was funded by NSF (#2211982) and Intel Labs. We would like to thank them for their generous support. Additionally, we extend our gratitude to the Research IT team* of Iowa
* https://researchit.las.iastate.edu/
State University for their continuous support in providing access to HPC clusters for conducting the experiments of this research project.