Structural Code Representation Learning for Auto-Vectorization
Authors: Yao Xiao
Nesreen Ahmed
Mihai Capotă
Guixiang Ma
Theodore L. Willke
Shahin Nazarian
Paul Bogdan
Venue: Submitted to ICLR 2023 (2023)

Abstract: The single instruction multiple data (SIMD) capability in modern processors is critical to improving the performance of current compute-intensive programs. SIMD allows architectures to exploit the natural data parallelism that exists in a wide-range of real applications (e.g., games, signal processing, etc) by executing a single instruction on multiple data items simultaneously. Modern compilers use vectorization techniques to exploit the SIMD capability, by detecting data parallelism in scalar source code and transforming a group of scalar instructions into vector-based instructions. In this work, we focus on one of the most common vectorization techniques called \emph{loop-based vectorization}, which targets loops and optimize their performance by grouping multiple occurrences of the same operation across loop iterations into single SIMD instructions. This is achieved by setting two key parameters: (1) the vectorization factor (VF), and (2) the interleaving factor (IF). Unfortunately, vectorizing loop computations effectively is a key challenging problem for both programmers and compilers due to the large search space. For example, manual vectorization of each loop puts a huge burden on the programmer, is more error-prone, and/or requires expert knowledge of both the software and the architecture. Alternatively, current compilers use fixed-cost models based on expert heuristics to make automatic vectorization decisions. However, these models often ignore the data dependencies, as well as the underlying computation graph. In this paper, we propose a data-driven graph-based learning framework for automatic vectorization, called \emph{autograph}, which takes an input program, extracts the loops, then learns a structured representation to automatically predict the correct VF/IF factors. Our proposed framework utilizes deep reinforcement learning to learn an optimal policy (observations to actions) from an intelligent agent in a SIMD environment, and automatically injects the predicted vectorization pragmas into the input program. We conducted an extensive evaluation on multiple benchmark datasets and comparisons with state-of-the-art baselines. Our experimental results show that the proposed framework can achieve up to 1.02x-2.26x and 1.06x-4.27x performance improvement, compared to state-of-the-art baseline and LLVM -O3 respectively.

Under review as a conference paper at ICLR 2023
STRUCTURAL CODE REPRESENTATION LEARNING FOR AUTO- VECTORIZATION
Anonymous authors Paper under double-blind review
ABSTRACT
The single instruction multiple data (SIMD) capability modern processors is critical to improving the performance of current compute-intensive programs. SIMD allows architectures to exploit the natural data parallelism that exists in a wide-range of real applications (e.g., games, signal processing, etc) by executing a single instruction on multiple data items simultaneously. Modern compilers use vectorization techniques to exploit the SIMD capability, by detecting data parallelism in scalar source code and transforming group of scalar instructions into rector-basee instructions. In this work, focus one of the most common vectorization techniques called loop-based vectorization, which targets loops and optimize their performance by grouping multiple occurrences of the same operation across loop iterations into single SIMD instructions. This achieved by setting two key parameters: (1) the vectorization factor (VF), and (2) the interleaving factor (IF). Unfortunately, vectorizing loop computations effectively is a key challenging problem for both programmers and compilers due the large search space. For example, manual vectorization of each loop puts huge burden on the programmer, is more error- error-prone, and/or requires expert knowledge of both the software and the architecture. Alternatively, current compilers use fixed-cost models based on expert heuristics to make automatic vectorization decisions. However, these models often ignore the data dependencies, as well the underlying computation graph. In this paper, we propose a data-driven graph-based learning framework for automatic vectorization, called autograph, which takes input program, extracts the loops, then learns a structured representation to automatically predict the correct VF/IF factors. Our proposed framework utilizes deep reinforcement learning to learn an optimal policy (observations to actions) from an intelligent agent in a SIMD environment, and automatically injects the predicted vectorization pragmas into the input program. We conducted an extensive evaluation on multiple benchmark datasets and comparisons with state-of-the-ar baselines. Our results show that for Polybench, autograph achieves on average 2.47x performance improvement for Polybench compared to neurovectorizer and 3.61x compared the baseline.
INTRODUCTION
The single instruction multiple data (SIMD) mechanisms have been widely incorporated in modern processors such as gaming machines, massively parallel supercomputers, as well as general-purpose processors (Nuzman et al. 2006 Bachega et 2004 Peleg & Weiser 1996). These mechanisms allow architectures to exploit the natural parallelism that exists in software for real-world applications (e.g., games, signal processing, etc.), by simultaneously executing the same instruction on multiple elements of the input data. Modern compilers use vectorization techniques to exploit the SIMD capa- bility of these architectures. Vectorization techniques allow the compiler reveal the data parallelism in the scalar source code and converts the code from scalar implementation to the corresponding cunctionally-corree vector implementation. This allows portions of the code to run on the processor's s high-throughput SIMD units, without any additional effort from the programmer (Porpodas et al.) 2018). With the SIMD architecture, such operations can run in fewer cycles while using less energy to boost performance in applications with vector computations.
Vectorization can be classified into two major methods: (i) the loop vectorizer, which operates on loops, and (ii) the superword-level parallelism (SLP) vectorizer (Porpodas) 2017 Mendis al.

---

Under review as a conference paper at ICLR 2023
(a)
Dependency Graph Regreeentaaion
Stractured Representation Learning (GNNs)
Oode Graph
Autograph Compile + Prediction Time
Brute-force Search Time
(b)
Embodddingg
11
Autograph Prediction Time
Intermedibte thee Reproserrattio
10
[og(Time]
Loop Extraction
Reinforcement
Leaming Agent
Input Program
-60000x
Complier
Reward
</>
Racormenndad
(VFIF)
200
6:00
800 1000
1200
1400
Kernel ID
Figure 1: (a) Overview of the proposed autograph framework. Autograph extracts loops from source code in the intermediate representation format (LLVM IR). Then using compiler dependency analysis, autograph constructs dependency graphs to capture the flow of information in the code, as well as the semantics of the code by using the full text features of the instructions. Autograph then learns structured representation using an inductive GNN approach. Finally, autograph expolits deep reinforcement to learn a mapping from embeddings to vectorization factors. (b) Comparison between brute-force and prediction-based autograph. For each kernel, the brute-force search time measured by summing up the compile times and execution times shown in log10 scale of nanoseconds with different VFs and IFs. Although the frute-force can eventually find the best vectorization parameters, it is 60 000x slower compared to autograph that only predicts the parameter without exhaustively searching the space.
2019), which operates straight-line code. Loops in repetitive tasks commonly used in modern programs to save time and code size. Therefore, in this work, we focus on the loop vectorizer. One of the key challenges is to define the vectorization factor (VF) and the interleaving factor (IF) (Nuzman et al. 2006). The VF determines how many instructions to pack together from different iterations of the loop. The IF determines the stride of the memory accesses of the packed instructions (Haj-Ali) iet al. 2020). Hence, the goal of loop vectorization is to search for the optimal VF and IF, given program. As shown in Figure [b), even though brute-force search has the ability to find the optimal/best vectorization parameter, the compiler cannot support the exhaustive search due to its time-consuming process. The brute-force method exhaustively searches the vectorization parameter for each loop in kernel while taking significant amount of time. For a kernel with N loops with M interleaving factors (IFs) and vectorization factors (VFs), the search space O(NKK).
Since manual vectorization is error-prone and difficult to maintain, modern compilers such as LLVM use nuto-vectorization techniques that rely on linear and constant-cost models to predict the vectorization factors (Tian et al. [2066) Trifunovic et al. 2009). However, these cost models don't 't consider the computation graph or loop dependencies, thus, they are not capable of capturing the natural structural dependencies and semantics of the wide-range of software programs that exist in today's ' real-world applications. Machine learning has been proposed to improve these cost models (Stock al., 2012 Wang & O' Boyle 2018) by extracting hand-engineered features from assembly code and using supervised learning techniques to predict the vectorization factors. However, these methods are still incapable of automatically learning representation that can capture the computation graph, as well as the dependencies of input codes. As we will show in evaluation, by accounting for structural dependencies we improve performance by 1.26x.
In this work, we propose a framework that learns a representation that is capable of reasoning about the flow of information and the semantics in the code, while capturing the structural dependencies in the computation graph. More specifically, in Figure [Ta. a). we propose an end-to-end graph-base based deep learning framework for compiler vettovvettoizztionn, called autograph, that takes an input code, extracts the loops, then learns a structured representation to automatically predict the correct VF/IF factors. Autograph utilizes deep reinforcement learning to learn an optimal policy (graph embeddings to VI/IF pairs) from an intelligent agent in SIMD environment, and automatically injects the predicted vectorization pragmas into the input program to achieve better performance. We conducted an extensive evaluation multiple benchmarks and comparisons with oftte-ofthhe-- art baselines. Our experimental results show that for Polybench, autograph achieves on average 2.47x performance improvement for Polybench compared to neurovectorizer and 3.61x compared to the baseline.
2

---

Under review as a conference paper at ICLR 2023
We summarize the main contributions follows:
We propose an end-to-end graph-based deep learning framework for compiler auto- vectorization, autograph, that is capable reasoning about the flow of information and the semanties in the code, while capturing the structural dependencies in the computation graph.
We develop deep reinforcement learning approach within the loop vectorizer capable of learning the optimal vectorization factors of a complex program
The proposed autograph framework is general and flexible to accommodate different repre- sentations of code predict the vectorization factors.
2 RELATED WORK
Automatic vectorization has been widely studied for improving application performance on SIMD architectures (Nuzman et al. 2006 Tian et al. (016)) Some of the existing work in nuto-vectoriization focuses on designing cost models for calculating expected execution time of vectorized loops (Trifunovic et al., 2009; Nuzman al. 2011). For example, (Trifunovic al. (2009) develops cost model to compare the performance of various vectorization alternatives and their interactions with other loop optimizations. Other work relies on heuristics for automatic vectorization, such as the instor-inttruction aware and ceuristic-guided search system proposed in (McFarlin et al. (011T for automatic vectorization of fast Fourier transforms.
In recent years, some research has also explored machine learning strategies for compiler optimiza- tion (Leather & Cummins 2020; Trofin al., 2021, Wang & Booyee 2018). For instance, the
SuperGraph- (Porpodas (017)) and VW-SLP (Porpodas et al. 2018) propose improved vector- ization algorithms based on Superword
Parallelism (SLP). The SuperGraph- -SLP operates larger region enable it vectorize code that previously unreachable. The VW-SLP varies the vector width at the granularity of the instruction level. This allows the algorithm to better adjust the code in SIMD and provide more parallelism. The MLGO framework (Trofin al. 2021
introduces machine learning methods such as policy gradient and evolution strategies to LLVM compiler optimizations. [Mammadi al. (2020) employs deep reinforcement learning (DRL)
method to the phase ordering in one of the compiler optimizations. Neuro-vectorizer (Haj-Ali et al.
2020) develops DRL approach for loop-level nuto-vectorization. However, (Wang OoBoee 2018) only provides a general survey of how to do feature engineering and provide optimization for compilers using machine learning. In addition, (Mendis et al.) 2019) focuses on SI.P vectorization whereas the focus of this work is loop vectorization. Therefore, both of them are not proper baselines.
In contrast to neuro-vectorizer, autograph is the first to propose a structured learning framework for loop auto-vectorization, which converts codes that contain loops to graphs and learns structured representations from code graphs. This structured representation can better capture the dependencies among instructions compared previous work that only relies on the code text features. Autograph can be used across different architectures with different SIMD capabilities. The only requirements of applications such architecture we collect the true label; and (2) compiler support to compile code with VF and IF into SIMD instructions.
autograph are (1) the ability
PROPOSED FRAMEWORK
In this work, we propose an end-to-end graph-based deep learning framework, called autograph, to predict vectorization factors. The proposed autograph framework is shown in Figure ma). Target programs are translated to obtain low-level virtual machine (LLVM) intermediate representation (IR) instructions of extracted loops to be vectorized. The IR instructions are then analyzed via compiler dependency analysis terms of data, control, and function calls to construct dependency graphs. The graphs are then fed into structured representation learning module that includes graph unsupervised learning and supervised learning to generate hidden embeddings. Finally, these embeddings are fed to the DRL agent to predict the vectorization factors VF and IF. The agent automat- ically injects vectorization pragmas, e.g., #pragma clang loop vectorize ze __ee__iidd width (32) interleawe _av__counttttttaeeee (16). The agent uses the clang compiler generate the executable of the program and runs it to gather the runtime, which is further used to calculate a reward for the RL agent. During inference testing, the well trained agent can be exploited to predict VF/IF pairs and find
3

---

Under review as a conference paper at ICLR 2023
the corresponding runtime information. The autograph framework injects pragmas only hints to help the compiler to generate the vectorized assembly instructions of loops such as Advanced Vector Extensions (AVX) in Intel architecture.
Graph representation of code. Code captures computations and communications within and between different high-level functions. Therefore, it of great importance to understand the heterogeneous structural organizations and dynamics of programs such as data flow from one instruction to another. However, traditional data-flow and control-flow graphs do not reveal the relationship between programs and underlying resources. Many recent graph representations of code Xiao al. (2019 2021 2017) can identify the spatio-temporal information flow by analyzing data dependencies between LLVM IR instructions collected at run-time. Moreover, PrograMI. Cummins al. (202T) is a graph representation for programs as input to machine learning model. The graph incorporates data-flow, control-flow, and call-flow that closely match the data structures used traditionally in inter-procedural data flow analysis. In our implementation, we use PrograML to generate code graphs.
Inductive representational graph unsupervised learning. From the code graphs, the autograph framework learns a representation that can reason about the information flow and the semantics in the code, while capturing the structural dependencies in the computation graph. Therefore, we propose inductive graph representation learning to learn embeddings of the nodes in an unsupervised way to understand semantics and structures of the code. The objective is formally formulated as follows.
Given graph G(V, E), Jearn an embedding of the nodes from the graph topological structures and the node attributes such as LLVM IR, without using any node labels.
The node embeddings are learned in a classification task: given a of positive node pairs generated from random walks on the target graph and negative node pairs that are randomly selected from the graph, learn a binary classifier that predicts whether the existing node pairs in the graph are likely to appear in a random walk. In this way, the classifier is able to learn a mapping from each graph with node attributes and structural dependencies in the code into an embedding.
The architecture of the classifier is defined as follows. Node pairs together with initial features are passed into a layer that (1) uses mean aggregator that aggregates the representations of each node in its immediate neighborhood into a vector, (2) concatenates the node's current representation, and (3) is fed through fully connected layer with the ReLU activation function followed by the L2 normalization. Therefore, the classifier generates the embedding for each node in graph. The general math operation of the forward pass of this model can be described as follows.
h N()
mean_ (ccen_aggregate/),.. Vj E N())
(1)
4 (I+1) o(W x concat(h, (h4, (() +
(1+1)
(2)
h/ (++)) L2 (nnomm)m))
where h encodes the hidden representations of the nodes; N is a neighborhood function: + mean mean_aggraeneeeea Laggregate is differentiable mean aggregator function; a a fully connected layer followed by the ReLU activation function; W is the weight matrix; L2_norm is the L2 normalization function.
2
Next, we use the dot product of the incident nodes' representations for each edge to calculate the edge score. Once the node-level embedding and edge score are calculated, we use binary cross entropy loss with the help of scores as the loss function as follows.
Sumo (1+1) ((+1))
==
Surve (owwollo (Durr) + Vuwe)log(1 )uww))
(5)
NNEEN
where nodes u and v are selected from node pairs N; Yuse and Dnon are the predicted and true link labels between and v. respectively.
Feature Representation. The initial features used in the unsupervised graph model are directly collected from the full text feature in each node of graph. The graphs are generated from PrograMI.
4

---

Under review as a conference paper at ICLR 2023
to capture the information flow between instructions. Therefore, associated with each node is one LLVM IR instruction. We tokenize all of the instructions to create a vocabulary that contains high- level structures, types, constants, and syntax. Next, for each full text in node, if a token exists in this vocabulary, we mark the corresponding index as 1, and O otherwise. Therefore, by adopting this approach, the size of the node embedding is the same the size of the vocabulary.
Refined latent embedding supervised learning. One drawback in the graph representation of codes is that two graphs could be identical but the vectorization parameters totally different, due to different number of iterations in the loops. This makes it difficult to learn optimal VF/IF pairs for performance improvement.
Therefore, in order to mitigate this issue, based on the feature embeddings learned from the unsupervised model, we further refine them into the supervised model to provide better accuracy and performance improvement. The architecture of the supervised model one layer of graph neural network (GNN), connected with one fully connected reed-foward neu- ral network (FCNN), shown in Figure In order to take into consideration the number of iterations in each for-loop, we insert them into the FCNN. There- fore, the hidden embedding size going into the FCNN is NaNN + N', where NONN is the embedding size for the GNN and N' is the number of loops in each file of benchmark.
DRL
Embeddings
NGNN
GNN
FCNN
N'
Label
Loop Count
Figure 2: The architecture of the supervised model.
Deep reinforcement learning. So far, autograph generates structural embeddings for code graphs via graph unsupervised learning and further refines them in terms of better accuracy and performance improvement over O3 via graph supervised learning. Next, autograph learns the optimal mapping from the embeddings the VF and pairs in an end-to-en fashion with reinforcement learning (RL). The rationale of employing the proposed end-to-end RL framework is that (1) it does not require labels to train the agent to prevent the consuming exhaustive search; (2) compared to supervised learning that optimizes the accuracy, it relies on the reward function that involves the normalized execution time to improve the system-level speedup; (3) it is hardware-agnostie meaning that while switching to another hardware platform, this RL framework does not require the exhaustive search to collect new labels before training compared to supervised learning. In RL, an agent continually interacts with an environment to learn and maximize the reward. Unlike supervised learning, RL can be tuned to co-optimize multiple objectives and be more efficient in terms of samples as it does not require a label.
Therefore, in this section, we propose a deep RL (DRL) based mapper. We seek to overcome the drawbacks of the brute force, random search, and supervised learning to improve the accuracy and the performance of the vectorized code over O3 at run-time. We represent the process of mapping the embeddings to the VF and IF pairs as Markov decision process (MDP) and apply DRL learn the optimal mapping given an embedding as state and a reward function for the quality of a mapping. We represent the vectorized kernels running on a specific hardware platform as an environment, and VF/IF pairs as set of actions to maximize the reward function.
Markov decision process (MDP). MDP discrete-ting time stochastic control process that can solve memoryless RI. problems in decision making. At each time step I, the process is in some state 8t and an action a; is chosen to find the next state 8141 with an unknown probability p(#141 (8t, at). An agent A A(az) s;; 0) selects sequence of actions followed by optimal policy xt via learned parameters The agent activates the action aj under the state s, in the environment and the reward r(st, ax) is returned back to the agent.
We consider the selection of vectorization parameters VF IF as an MDP. The state space consists of hidden feature embeddings leamned from the supervised model. The action space consists of different discrete vectorization parameters. The agent's 's task, then, to pick VF and IF for a given embedding from the code given 8 to maximize reward function under the environment.
5

---

Under review as a conference paper at ICLR 2023
Reward function. The reward function is used to evaluate and learn the optimal policy. We design an immediate reward below for the current state s under the action az.
r r8t; ae) (khaseline 1(8,, (x))
Ubaseline
where bbaseline is the execution time of a file running with the -3 flag in the LLVM workflow; tss,as the execution time for the current VF IF pair.
We calculate the difference of runtimes between the current vectorization for the kemnl 8, and the baseline running with the compiler flag O3 LLVM. Next, is normalized by the bascline runtime so that the reward function is robust variations in the programs' execution times.
Agent: fully connected neural network (FCNN). The agent is used to select an action aN (VF/IF pair) from the state 8t (embeddings) find the optimal policy that maximizes the cumulative reward R = Σ 7'r(a,, st) where ? E [O, 1] decay factor. The inputs the agent are feature embeddings extracted from graphs via supervised models. Q values are used to represent the maximum cumulative reward the agent can obtain by taking the action d, in the state 8t. Then, the optimal value Q" (st, 0x) can be calculated by Bellman equation Sutton & Barto (2018) as follows.
Q' ("1,,a as) = Elre+1 + max Q (8t+1, At+1 at)]
(7)
ae-1
We then follow Q-learning to update Q values.
Q1+1 ae) (1 a a)Qa(st a:) + afr(st, 0z) + max Qc(8++1,
act1
where a E [0, 1] is the learning rate.
Environment: vecforized code running on hardware. The environment interacts with the agent receive possible actions (VF/IF pairs). Once an action is obtained, the environment injects the vectorization pragma into the code, compiles into the executable, and runs it to get the runtime information. Next, the environment provides an immediate reward (eq. 6p. This reward is returned back the agent help make better decisions next time.
EVALUATION
In this section, discuss the hardware configurations, baselines, and datasets, along with experi- mental results to investigate the effectiveness of the proposed autograph framework.
4.1 EXPERIMENTAL SETUP
Configurations. The unsupervised learning, supervised learning, and DRL are performed in Ubuntu 20.04.3 LTS with 64 CPU threads and 2 GPUs. The CPU is an Intel(R) Xeon(R) Gold 5218 CPU at 2.30 GHz. One GPU is an NVIDIA TESLA V100, the other is an NVIDIA TITAN RTX TU102.
2017) extract feature embeddings in an unsupervised fashion, GCN (Kipf & Welling 2016) to further refine the embeddings to improve accuracy and performance, and DRL. DRL is implemented with RLlib (Liang el al.] 2017) and Tune (Liaw et al., 2018), built on top of Ray (Moritz et al. 2018). open- source libraries for RL that offer high scalability. parae-ea tuning and numerous application interfaces. In the GraphSAGE setup, the number of layers is and learning rate 0.001 with the dropout rate of 0. In the GCN setup, the number of layers is 2 and learning rate 0.001. In the DRL setup, the learning rate is 0.0001 with 800,000 steps and 15,000 training batch size. The sizes of hidden layers are both 256. The neuro-vectorin vectorizer framework incorporates code2vec (Alo et al. 2019) and DRL. The open-source code code2vec is modified to work with the RL agent implementation (Haj-Ali et al., 2020). The DRL hyperparameters are the same as autograph. We run a brute-force search on all of the datasets to find the optimal run-times and the best vectorization factors VF and IF, and use them as labels for supervised models. We also implement two supervised models, one with code2vec and FCNN and another one with GCN and FCNN. The batch size is 64. Both FCNNs have 2 layers with 0.001 learning rate. We run all of the models times and report mean and standard deviation.
Baselines.
The autograph framework incorporates GraphSAGE (Hamilton et al.

---

Under review as a conference paper at ICLR 2023
Table Impact of different pipelines.
Table Impact of different GNN architectures.
Pipelines GraphiSAGE-
Accuracy
Speedup
GNN
Accuracy
Speedup
18.45%
1.24%
1.47 0.14
GCN
18.45%
1.24%
1.47 0.14
GCN
13.22% 16%
1.2311021
GAT
15.30%
1.17%
1.32 1 0.11
GraphSAGE- CripPoAAGE
12.67% 11.45%
1,1710.15
GGNN
19.12% 10.93%
1,52.1 0.12
Datasets. We use LORE suite of kernels (Chen et al. 2017) to train and test DRL. Specifically, we use the SPEC 2006 (Henning) 2006), NPB (Bailey et 1995), and Polybench (Pouchet 2012) kernels as testing, and the rest of benchmarks as training. In addition, we transfer the learned model in DRL to further test the GCC (Haj-Ali Ali et al. 2020) and Mibench (Guthaus et al.) 2001) benchmarks. GCC contains a set of loops that was created to test the auto-vectorization capabilities of GCC. We parameterized the loops terms of a number of arrays, arithmetic operations and data sizes to increase the number of kernels to 744. Mibench is a set of representative embedded benchmarks such as telccommunication, networking, security, office, and automation; we use the MiBench subset selected in the neuro-vectorizer paper.
4.2 FRAMEWORK COMPARISON
Training loss and reward mean. We validate how well autograph is trained in terms train- ing loss and reward mean for LORE with different batch sizes and fully-connected neural net- works (FCNN). As we can see from Figure 3] if the number of timesteps is small (< 100,000), the agent in DRL is unable to explore most of the high-dimensional state space and thus fails to converge to an optimal policy. Therefore, training loss is high and reward mean is small. But as we gradually increase the number of timesteps, the agent can explore the region and find better policy. Note that there is also a diminishing return when we increase the num- ber of timesteps further beyond a point where the training loss and reward almost converge to a number. Therefore, in the following evaluation, we set the number of timesteps to 800, 000 to ensure that the agent is well trained, leading to small training loss and high reward mean.
Ablation different pipelines.
Different batch sizes
8A
Different architectures
In this part, investigate the impact of graph unsupervised learning (to understand the struc-
(a)
(c)
Mevord mon
02
10000 230000 30000 400000 50000 500033
Howard
E
tures of graphs) and graph su- pervised learning (to further re- fine the features). As shown in Table [ with graph unsuper- vised learning such as Graph- SAGE, GrEphSAGE-GGN can provide 1.40x higher in terms of accuracy and 1.20x in terms of speedup, compared to only GCN, With the graph supervised learning model such GCN, GraphSAGE-Cr GCN can provide 1.46x higher in terms of accuracy and 1.26x in terms of speedup, compared to PorppsAAGG-PPoing-FFCCC
+000 -0-
2003
- 4000
--- [32, 30]
194
-+-1324 221)
Timesiops
Tieekkgs
Different boich sizes
14
Different FCNN anchitecturs
- 1000 2000
1300
(d)
12, [04
122
02
320000 200000
300000 430000 Timertops
500000
500000
100003
300000 500000
Tinssiigs
Figure 3: Reward mean and training loss as function of the number of timesteps with different batch sizes (a-b) and FCNN architectures (c-d).
Ablation GNN architectures. In this part, we investigate the effect of different GNN architectures such as GCN, GAT, and GGNN used in the supervised learning on the overall accuracy and speedup. Experimental results in Table Dableaeindicate does not contribute significantly to higher accuracy and speedup compared GCN with only 1.03x improvement. However, training GAT and GGNN takes more time compared to GCN. Therefore, we choose GCN in the current implementation.
indicate that using more complicated models such as GAT and GGNN
Reward. We compare the DRL part in auto-
graph and neuro-vectorizer in terms of the re-
17.5
autograph (geometric mean 1.17) neuro-vectorizer (geometric 1.09)
15.0
7
12.
Reward
10.0

---

Under review as a conference paper at ICLR 2023
wards of all kernels in the LORE dataset. As can see in Figure 4 many kernels have larger rewards the red line is above the blue line. This indicates that in general, autograph could provide better performance improvement pared to neuro-vectorizer. It could be also val- idated by comparing the geometric mean of the rewards from autograph (1.17) and neuro- vectorizer (1.09).
Distributions of true labels and predictions. We compare the distributions of true labels rang-
ing from to 24 due to VFs and IFs, the neuro-vectorizer predictions, and autograph predictions in LORE. As shown in Figure 5 the cumulative distribution of the autograph predictions is much closer to that of true labels than neuro-vectorizer. Furthermore, the Kullback-Leibler (KI.) divergence (DIV) between autograph and true labels ( (1034.10 bits) is much smaller than the KL DIV between neuro- vectorizer and true labels (3995.6 66 bits). Therefore, it validates that autograph has a wider action distribution and is more similar the distribution of true labels.
Average accuracy and normalized speedup. We train and test all of the models on LORE. Accuracy is measured by calculating the num- ber of times where the ground truth is the same the predicted action, divided by the total num- ber of kernels. The ground truth is collected from choosing the optimal execution time by exhaustively searching from the space of the vectorization parameters. The predicted action is collected from the results from each model when a new kernel arrives.
1.0
autograph
neuro-vectorzzer brute-force
Cumulative distribution
0.6-
0.2
0.0
Class label
Tables 3 and show the average accuracy and
Figure 5: Distributions of true labels, autograph. speedup normalized to 03 runtimes for neuro- and neuro-vectorizer predictions.
vectorizer, autograph, code2vec, and GCN, respectively. In NPB, Polybench, and SPEC 2006 kernels, autograph outperforms neuro-vectorizer by 5.97x, 2.2x, and 1.31x in terms of the accuracy and 1.06x, 1.16x, and 1.18x in terms of the geometric speedup compared to O3. This is because autograph learns the graph structures that can be used to predict the labels. Furthermore, compared to supervised learning, the DRL-based approaches achieve 1.22x. 1.35x, and 19x better in terms of speedup.
Table 3: NPB kernels.
Original vectorization
Effective vectorization'
MI. models
Accuracy
Speedup
Accuracy
Speedup
autograph
19.41%-11.42
1.16=0.05
20.19% 1.89%
1.17+00.07
neuro-vectorizer
3.25%=1.88
1.09_0.04
3.80% 1,96
1.11
code2vec
7.72%-10.653.
0.95=0.02
0.97+00.3
GCN
8.91%
1.04=0.03
9.67%+00.3%%
1.07=00.05
It is measured by pruning some kernels with less than 1.02x brute-force speedup compared to O3 baseline.
Table 4: Polybench kernels with original vectorization. Polybench is benchmark for polyhedral transformations that operate on loops, so pruning leads to the same kernels.
ML models
Accuracy
Speedup
autograph
19.87% = 0.53%
1.31 + 0.05
neuro- vectoriec
9.05% + 1.23%
1.13 + 0.12
code2vec
8.22% + 0.58%
0.97 0.04
GCN
9.80% + 0.77%
1.09 + 0.08
8

---

Under review as a conference paper at ICLR 2023
Normalized Performance
15
Normalizzed Performance
(b)
Normalized Performance
(c)
brute-force
10
aulograph
neuro-vectorizer
code2vec
5
Kemels
Kemels
Kemels
Figure 6: Performance in the SPEC 2006 (a), NPB (b), and Polybench (c) kernels with highest brute-force speedups normalized to O3.
Table 5: SPEC 2006 kernels.
Original vectorization
Effective vectorization
ML models
Accuracy
Speedup
Accuracy
Speedup
autograph
6.42%:00.37%
1.04:0.02
1..5%%
1.24::0.04
neuro-vectorizer
4.95%
1.02+:.01
9.52%=11.08%
1.05:E0.02
code2vec
3.32%=10.22%
0.94
6.19%
1.04
GCN
1..4%%
0.97
9.15%:10.77%
1.09::0011
Normalized performance for top kernels. Figure a--c) shows the normalized performance on SPEC 2006 (a), NPB (b), and Polybench (c) for brute-force search, supervised code2vee, supervised GCN, neuro-vectorizer, and autograph. These kernels are selected based on the top normalized speedup of brute-force compared to O3. Compared neuro- vectorvec,,,,,rr, autograph can provide at most 6.51x xx performance improvement over 03 for some files because autograph represents kernels structured graphs and learns their representational embeddings unveiling their hidden structures.
Table GCC with original vectorization. GCC a synthetic benchmark for loop transformations, so pruning leads to the same kernels.
ML models
Accuracy
Speedup
autograph
26.51% = 0.36%
1.62 + 0.07
neuro-vectorizer
9.73% + 1.23%
1.22 0.16
code2vec
8.07% + 0.61%
1.08 + 0.06
GCN
8.80% + 0.79%
1.15 1 0.08
Table 7: MiBench subset. We use the same subset of Mibench as in neuro-vectorizer to validate the effectiveness of autograph.
Original vectorization
Effective vectorization
ML models
Accuracy
Speedup
Accuracy
Speedup
autograph
1..4%%
1.140.03
16.92%
1.41:00.07
neuro-vectorizer
5.21% 1211%11 1..55%
1.06.10.02
11.66% 11.%+11..230
1.12::0.04
code2vec
3.78% 10.24%
0.953
+0.48%
1.0940.06
GCN
4.71% 4..45%
1.03+0.00
9.75%10.92%%
1.11+0.08
Transfer learning on GCC and MiBench. It is important to see how well the machine learn- ing models generalize to completely new datasets. To that end, we evaluate on two benchmarks different models that are trained LORE. GCC, compared to MiBench, is more beneficial to auto-vectorization because it is synthetic benchmark for loops that are designed to be vectorized. As shown in Table 6 and 7 autograph can achieve average 2. 72x higher accuracy and 1.33x higher performance than neuro-vectorize Compared to the supervised learning models, autograph can provide average 1.5x normalized performance improvement baseline O3. We conclude

---

Under review as a conference paper at ICLR 2023
that the auto-ves vettooecctriztiinnnnnno is limited to code with rich portions of loops that can be vectorized. It is not suitable for benchmarks such as MiBench where the loops constitute a small portion of the code.
CONCLUSION
We propose Autograph, an end-to-end framework for compiler auto-vectorization that automatically extracts loops, constructs dependency graphs, and learns structured representations that capture both the structural dependencies of the computation graph and the semantics of the code. Autograph uses deep reinforcement learning approach predict vectorization factors and injects the vectorization pragmas with the optimal VF/IF factors in the code to achieve better performance. Our extensive experiments and comparisons multiple benchmark datasets show that for Polybench, autograph achieves on average 2.47x performance improvement for Polybench compared to neurovectorizer and 3.61x compared the baseline.
10

---

Under review as a conference paper at ICLR 2023
REFERENCES
Uri Alon, Meital Zilberstein, Omer Levy, and Eran Yahav. code2vec: Learning distributed rep- resentations of code. Proceedings of the ACM on Programming Languages, 3(POPL): 2019.
Leonardo Bachega, Siddhartha Chatterjee, Kenneth A Dockser, John A Gunnels. Manish Gupta, Fred G Gustavson, Christopher Lapkowski, Gary K Liu, Mark P Mendell, Charles D Wait, et al. A high-performance simd floating point unit for bluegene/l: Architecture, compilation, and algorithm design. In Proceedings. 13th International Conference on Parallel Architecture and Compilation Techniques, 2004, PACT 2004., pp. 85-96. IEEE, 2004.
David Bailey, Tim Harris, William Saphir, Rob Van Der Wijngaart, Alex Woo, and Maurice Yarrow. The parallel benchmarks 2.0. Technical report, Technical Report NAS-95-020, NASA Ames Research Center, 1995.
Zhi Chen, Zhangxiaowen Gong, Justin Josef Szaday, David C Wong, David Padua, Alexandru Nicolau, Alexander V Veidenbaum, Neftali Watkinson, Zehra Sura, Saced Maleki, et al. Lore: A loop repository for the evaluation of compilers. In 2017 IEEE International Symposium on Workload Characterization (IISWC), pp. 219 129.228. IEEE, 2017.
Chris Cummins, Zacharias Fisches, Tal Ben-Nun, Torsten Hoefler, Michael O' O'Boyle, and Hugh Leather. ProGraML: A Graph-based Program Representation for Data Flow Analysis and Compiler Optimizations. In Thirty-eighth International Conference on Machine Learning (ICML), 2021.
Matthew R Guthaus, Jeffrey S Ringenberg. Dan Ernst, Todd M Austin, Trevor Mudge, and Richard B Brown. Mibench: A free, commercially representative embedded benchmark suite. In Proceedings of the fourth annual IEEE international workshop on workload characterization. WWC-4 (Cat, No. 01EX538), pp. 3-14. IEEE, 2001.
Ameer Haj-Ali, Nesreen K Ahmed, Ted Willke, Yakun Sophia Shao, Krste Asanovic, and Ion Stoica. Neurovectorizer: End-to-end vectorization with deep reinforcement learning. In Proceedings of the / 18th ACM/IEEE International Symposium on Code Generation and Optimization, pp. 242-255, 2020.
Will Hamilton, Zhitao Ying, and Jure Leskovec. Inductive representation learning on large graphs. Advances in neural information processing systems, 30, 2017.
John Henning. Spec cpu2006 benchmark descriptions. ACM SIGARCH Computer Architecture News, 34(4):1-17, 2006.
Thomas N Kipf and Max Welling, Semi-supervised classification with graph convolutional networks. arXiv preprint 1rXiv: 1609.02907, 2016.
Hugh I Leather and Chris Cummins. Machine learning in compilers: Past, present and future. In 2020 Forum for Specification and Design Languages (FDL., pp. 1-8. IEEE, 2020.
Eric Liang, Richard Liaw, Robert Nishihara, Philipp Moritz, Roy Fox, Joseph Gonzalez, Ken Goldberg, and Ion Stoica. Ray rllib: A composable and scalable reinforcement learning library. arXiv preprint arXiv: /712 85, 2017.
Richard Liaw, Eric Liang, Robert Nishihara, Philipp Moritz, Joseph E Gonzalez, and Ion Stoica. Tune: research platform for distributed model selection and training. arXiv preprint 1XX,x:800.05111 18, 2018.
Rahim Mammadli, Ali Jannesari, and Felix Wolf. Static neural compiler optimization via deep reinforcement learning. In 2020 JEEE/ACM 6th Workshop on the LLVM Compiler Infrastructure in HPC (LLVM-HPC) and Workshop on Hierarchical Parallelism for Exascale Computing (HiPar), pp. 1-11. IEEF, 2020.
Daniel S McFarlin, Volodymyr Arbatov, Franz Franchetti, and Markus Püschel. Automatic simd vectorization of fast fourier transforms for the larrabee and avx instruction sets. In Proceedings of the international conference Supercomputing, pp. 274, 2011.
11

---

Under review as a conference paper at ICLR 2023
Charith Mendis, Cambridge Yang, Yewen Pu, Dr Amarasinghe, Michael Carbin, et al. Compiler vettooectorizttoon with imitation learning. Advances in Neural Information Processing Systems, 32, 2019.
Philipp Moritz, Robert Nishihara, Stephanie Wang. Alexey Tumanov, Richard Liaw, Eric Liang, Melih Elibol, Zongheng Yang, William Paul, Michael Jordan, al. Ray: A distributed framework for emerging {AI] applications. In 13th USENIX Symposium on Operating Systems Design and Implementation (OSDI 18), pp. 561-577, 2018.
Dorit Nuzman, Ira Rosen, and Ayal Zaks. Auto-vectorization of interleaved data for simd. ACM SIGPLAN Notices, 41(6):132-143; 2006.
Dorit Nuzman, Sergei Dyshel, Erven Rohou, Ira Rosen, Kevin Williams, David Yuste, Albert Cohen, and Ayal Zaks. Vapor simd: Auto-vectorize once, run everywhere. In International Symposium on Code Generation and Optimization (CGO 2011), pp. 151-160. IEEE, 2011.
Alex Peleg and Uri Weiser. Mmx technology extension to the intel architecture. IEEE micro, 16(4): 42-50, 1996.
Vasileios Porpodas. Supergraph-slp auto- nittovvettrizationn In 2017 26th International Conference on Parallel Architectures and Compilation Techniques (PACT), pp. 330-342. IEEE, 2017.
Vasileios Porpodas, Rodrigo CO Rocha, and Luís FW Góes. Vw-sip: auto-vectorization with adaptive vector width. In Proceedings of the 27th International Conference on Parallel Architectures and Compilation Techniques, pp- 1-15, 2018.
L.-N. Pouchet. http://www.cs..cl...dddeee www.eeecwwuuuuucceeeseeeeeeeeeee 2012.
Polybench:
The polyhedral
benchmark
suite.
In
Kevin Stock, Louis-Noël Pouchet, and P Sadayappan. Using machine learning to improve automatic vectorization. ACM Transactions on Architecture and Code Optimization (TACO), 8(4):1-23, 2012.
Richard Sutton and Andrew G Barto. Reinforcement learning: An introduction. MIT press, 2018.
Xinmin Tian, Hideki Saito, Emesto Su, Abhinav Gaba, Matt Masten, Eric Garcia, and Ayal Zaks. Llvm framework and ir extensions for parallelization, simd vectorization and offloading. In 2016 Third Workshop on the L.VM Compiler Infrastructure in HPC (LL.M--HCC, pp. 21-31. IEEE, 2016.
Konrad Trifunovic, Dorit Nuzman, Albert Cohen, Ayal Zaks, and Rosen. Polyhedral guided loop-nest nnto-vvectoiizatoon. In 2009 18th International Conference on Parallel Architectures and Compilation Techniques, pp. 327-337. IEEE, 2009.
Mircea Trofin, Yundi Qian, Eugene Brevdo. Zinan Lin, Krzysziof Choromanski, and David Li. Mlgo: a machine learning guided compiler optimizations framework. arXiv preprint arXiv:2101, 04808, 2021.
Zheng Wang and Michael O' Boyle. Machine learning in compiler optimization. Proceedings of the IEFF, 106(1 11111))18791901,, 2018.
Yao Xiao, Yuankun Xue, Shahin Nazarian, and Paul Bogdan. A load balancing inspired optimization framework for exascale multicore systems: A complex networks approach. In 2017 IEEF/ACM International Conference on Computer-Aided Design (ICCAD), pp. 217-224. IEEE, 2017.
Yao Xiao, Shahin Nazarian, and Paul Bogdan. Self-optimizing and self solf-programiing computing systems: A combined compiler, complex networks, and machine learning approach. IEEE Transactions on Very Large Scale Integration (VLSI) Systems, 27(6):1416-1444 2019.
Yao Xiao, Shahin Nazarian, and Paul Bogdan. Plasticity-on-chip design: Exploiting self-similarity for data communications. IEEE Transactions on Computers, 70(6)955-992,, 2021.
A APPENDIX
You may include other additional sections here.
12


Reviews:
[{"review_id": "eLWHOv2UVFH", "reviewer": "Reviewer_NSgq", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper presents a learning-based auto-vectorization method. The idea is to represent the code as a graph, learning embeddings for nodes in the graph, and use the embeddings to predict two vectorization parameters: VF and IF. ", "summary_of_the_review": "The paper is easy to follow. The techniques look new to me. However, for such learning-based code optimization work, a real implementation is important. For this reason, I give the paper a weak reject. ", "strength_and_weaknesses": "## Strength\nThe paper is well writing. The presentation of the technique is clear. \n\n## Weakness\n1. While the presentation of the technique is clear, it is unclear why it outperforms previous method. Can you give an example that shows your method obtains better vectorization than previous methods? \n\n2. I do not find any source code. To make your work more convincing, please consider open source your code. Auto-vectorization is a well studied topic in compiler community and many techniques are implemented in real compilers. If you want to claim an advantage over the existing work, you need to show the code. ", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "DqfoLrZkr9", "reviewer": "Reviewer_9X1Y", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper mainly focuses on the problem of loop-based vectorization and proposes a graph-based learning framework named autograph. Autograph aims to automatically predict the correct vectorization factor and the interleaving factor. Experimental results based on several data sets are reported.", "summary_of_the_review": "The rationale of employing deep reinforcement learning in the proposed method remains unclear. Important baselines are missing in the experiment. In addition, the loop-based vectorization problem is likely to have an impact within only a subfield of learning representation.", "strength_and_weaknesses": "There are some weaknesses of this paper:\n\n-\tThe rationale of employing deep reinforcement learning in the proposed method remains unclear. More discussion and empirical evidence (e.g., the ablation study) should be provided about how the deep RL helps the loop-based vectorization problem.\n-\tImportant baselines are missing in the experiment. The state-of-the-art loop-based vectorization method (e.g., Ref Wang and O’Boyle, 2018) is missing in the comparison, which brings more concerns about the performance improvement of the proposed method.\n-\tIn addition, the loop-based vectorization problem is likely to have an impact within only a subfield of learning representation, which reduces the significance of this paper.\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "9k0xUXHOopA", "reviewer": "Reviewer_pVcf", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper introduces a DRL based approach to perform Loop Vectorization in LLVM IR. They represent LLVM IR as a graph and then learn an embedding in an unsupervised manner using a GNN, which they later refine using a FCNN to decide the VF and IF in the loop vectorizer.", "summary_of_the_review": "IMO, the paper lacks novelty. Important related work are missing and this leads to wrong claims. I have mentioned them under novelty. Further, important work related to cost models (e.g. [1],[2] ) and how representation learning on graphs are used to learning embeddings that can be used for this task is missing. \n\nI find the ablations on pretraining (unsupervised learning) to be informative. However, it is unclear how the random walks are done to get negative examples. This is not the usual code language model pretraining that is used here. Better writing of this section would help. Further, it is unclear why this unsupervised learning objective was chosen, given that code language models usually opt for MLM based pertaining at least in the contextual sense.\n\nAlso, I found the \"upto\" speedups a little misleading considering the geomean speedups that are reported.\n\nOverall, I think this paper is below bar for acceptance at ICLR. I am more concerned with novelty related claims.\n\n[1] Baghdadi et al. \"A Deep Learning Based Cost Model for Automatic Code Optimization\" MLSys 2021\n\n[2] Mendis et. al \"Ithemal: Accurate, Portable and Fast Basic Block Throughput Estimation using Deep Neural Networks\", ICML 2019", "strength_and_weaknesses": "Strengths:\n* The paper improves on structural representation of the code from the NeuroVectorizer paper, even though the basic premise is the same.\n* Encouraging results from the LORE suite.\n\nWeaknesses\n*  Important related work that uses the structure for compiler auto-vectorization is missing. This leads to wrong novelty claims made at the end of section 2. This is a major flow.\n* The paper is low in novelty, where essentially they added learning based on structure to the NeuroVectorizer. Adding structure to learning in the context of vectorization is also not new as mentioned above.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "1: The contributions are neither significant nor novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "2_wdaubtV5", "reviewer": "Reviewer_Coc4", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper presents autograph, a GNN-based approach to automatic loop vectorization.\nAutograph represents the program as a graph, computes an embedding for each node using a standard GNN forward pass, then trains a RL agent to predict vectorization factors based on node embeddings.\nThe paper evaluates on a set of standard CPU benchmarks, showing that autograph outperforms the most related prior work in this area, neuro-vectorizer.\n", "summary_of_the_review": "Reject.\nThe method is not sufficiently clear, such that a reasonable reader could not reimplement the approach.\nThe evaluation is not sufficiently high quality, such that the presented results are not convincing.\n\nI would be willing to raise my score conditioned on:\n* The authors providing a significantly more clear description of the approach.\n* The authors explaining the choice of accuracy metric in the evaluation (or present a different metric)\n* The authors clarifying the train/validation/test split methodology and the transfer learning methodology\n* The authors explaining the discrepancy between the neuro-vectorizer results in this paper and the results in the neuro-vectorizer paper\n\n\n-------\n\n## Update after author response\n\nThanks to the authors for the detailed response. Based on this response, I'm raising my score to a weak accept. Specifically:\n* The revision includes a much more clear description of the approach (page 4)\n* The authors correctly pointed out that I missed the definition of accuracy in the original submission. Though I still think this is a weak metric, I agree with the authors that the speedup metric is ultimately the one that matters (which the paper does report).\n* The author response does not state whether or not any kernels are shared between the training/test dataset (and further, do not mention any validation set). The author response is also not sufficiently clear on the response about transfer learning: my point of confusion was whether or not any fine-tuning is performed on the other task (as is typically implied by the term transfer learning). The author response implies that no fine-tuning is performed, but is not 100% clear on that.\n* The explanation of the neuro-vectorizer performance is satisfactory. Though it's surprising that neuro-vectorizer does not perform anywhere near as well as in the original paper (and raises some questions about whether it is a fitting baseline as-is), I don't view this as a reason to reject this paper.\n\nIf the paper is accepted, I would still like for the authors to include the following in the final version of the paper:\n* Stating to what extent the different applications in the train and test set share kernels (or if they do not share any kernels)\n* Stating how hyperparameters were tuned (given the lack of a validation set, I'm assuming that everything was tuned to maximize training set accuracy, but this should be explicitly stated)\n* Making clear whether any fine-tuning or other transfer learning methodology was performed, or whether the transfer learning test is just a test of out-of-distribution generalization\n* Including this discussion of neuro-vectorizer results\n* Further improvements to the clarity of the approach", "strength_and_weaknesses": "Strengths:\n* The paper providers a novel approach to a relevant problem, automatic loop vectorization\n* The results, if they stand up (see weaknesses) are strong, significantly outperforming O3 and prior work in neural loop vectorization\n\nWeaknesses:\n* The clarity of the approach is low (such that the approach could not reasonably be implemented by a reader) -- see the clarity section below\n* The quality of the evaluation is low, such that I do not have confidence that the approach actually beats the baselines (especially the comparison to neuro-vectorizer) -- see the quality section below\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "Not applicable"}, "extra_text": {}}]

Author rebuttal:

Decision: Reject