Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: This paper mainly focuses on the problem of loop-based vectorization and proposes a graph-based learning framework named autograph. Autograph aims to automatically predict the correct vectorization factor and the interleaving factor. Experimental results based on several data sets are reported.
strength_and_weaknesses: There are some weaknesses of this paper:

-	The rationale of employing deep reinforcement learning in the proposed method remains unclear. More discussion and empirical evidence (e.g., the ablation study) should be provided about how the deep RL helps the loop-based vectorization problem.
-	Important baselines are missing in the experiment. The state-of-the-art loop-based vectorization method (e.g., Ref Wang and O’Boyle, 2018) is missing in the comparison, which brings more concerns about the performance improvement of the proposed method.
-	In addition, the loop-based vectorization problem is likely to have an impact within only a subfield of learning representation, which reduces the significance of this paper.

clarity,_quality,_novelty_and_reproducibility: The writing of this paper needs further improvement. Important baselines are missing in the experiment. The rationale of employing deep reinforcement learning in the proposed method remains unclear.
summary_of_the_review: The rationale of employing deep reinforcement learning in the proposed method remains unclear. Important baselines are missing in the experiment. In addition, the loop-based vectorization problem is likely to have an impact within only a subfield of learning representation.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: This paper presents a learning-based auto-vectorization method. The idea is to represent the code as a graph, learning embeddings for nodes in the graph, and use the embeddings to predict two vectorization parameters: VF and IF. 
strength_and_weaknesses: ## Strength
The paper is well writing. The presentation of the technique is clear. 

## Weakness
1. While the presentation of the technique is clear, it is unclear why it outperforms previous method. Can you give an example that shows your method obtains better vectorization than previous methods? 

2. I do not find any source code. To make your work more convincing, please consider open source your code. Auto-vectorization is a well studied topic in compiler community and many techniques are implemented in real compilers. If you want to claim an advantage over the existing work, you need to show the code. 
clarity,_quality,_novelty_and_reproducibility: The writing is clear. The idea is new. No code is provided. 
summary_of_the_review: The paper is easy to follow. The techniques look new to me. However, for such learning-based code optimization work, a real implementation is important. For this reason, I give the paper a weak reject.
review_title: null

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: The paper introduces a DRL based approach to perform Loop Vectorization in LLVM IR. They represent LLVM IR as a graph and then learn an embedding in an unsupervised manner using a GNN, which they later refine using a FCNN to decide the VF and IF in the loop vectorizer.
strength_and_weaknesses: Strengths:
* The paper improves on structural representation of the code from the NeuroVectorizer paper, even though the basic premise is the same.
* Encouraging results from the LORE suite.

Weaknesses
*  Important related work that uses the structure for compiler auto-vectorization is missing. This leads to wrong novelty claims made at the end of section 2. This is a major flow.
* The paper is low in novelty, where essentially they added learning based on structure to the NeuroVectorizer. Adding structure to learning in the context of vectorization is also not new as mentioned above.
clarity,_quality,_novelty_and_reproducibility: Novelty: Very low, important related work missing with no comparison. Consider [1], they use the graph structure of the program in their auto-vectorization scheme. The authors have not cited or compared against their approach, which leads to the wrong novelty claim, "In contrast to neuro-vectorizer, autograph is the first to propose a structured learning framework for auto-vectorization". I find this a major flaw in the paper and diminishes the value of the contributions.

Clarity: The unsupervised training part of the paper was confusing to me, specially its training objective. Rest of the paper was ok to read.

[1] Mendis et. al "Compiler Auto-Vectorization with Imitation Learning", NeurIPS 2019.
summary_of_the_review: IMO, the paper lacks novelty. Important related work are missing and this leads to wrong claims. I have mentioned them under novelty. Further, important work related to cost models (e.g. [1],[2] ) and how representation learning on graphs are used to learning embeddings that can be used for this task is missing. 

I find the ablations on pretraining (unsupervised learning) to be informative. However, it is unclear how the random walks are done to get negative examples. This is not the usual code language model pretraining that is used here. Better writing of this section would help. Further, it is unclear why this unsupervised learning objective was chosen, given that code language models usually opt for MLM based pertaining at least in the contextual sense.

Also, I found the "upto" speedups a little misleading considering the geomean speedups that are reported.

Overall, I think this paper is below bar for acceptance at ICLR. I am more concerned with novelty related claims.

[1] Baghdadi et al. "A Deep Learning Based Cost Model for Automatic Code Optimization" MLSys 2021

[2] Mendis et. al "Ithemal: Accurate, Portable and Fast Basic Block Throughput Estimation using Deep Neural Networks", ICML 2019
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: The paper presents autograph, a GNN-based approach to automatic loop vectorization.
Autograph represents the program as a graph, computes an embedding for each node using a standard GNN forward pass, then trains a RL agent to predict vectorization factors based on node embeddings.
The paper evaluates on a set of standard CPU benchmarks, showing that autograph outperforms the most related prior work in this area, neuro-vectorizer.

strength_and_weaknesses: Strengths:
* The paper providers a novel approach to a relevant problem, automatic loop vectorization
* The results, if they stand up (see weaknesses) are strong, significantly outperforming O3 and prior work in neural loop vectorization

Weaknesses:
* The clarity of the approach is low (such that the approach could not reasonably be implemented by a reader) -- see the clarity section below
* The quality of the evaluation is low, such that I do not have confidence that the approach actually beats the baselines (especially the comparison to neuro-vectorizer) -- see the quality section below

clarity,_quality,_novelty_and_reproducibility: ## Clarity

Overall, the paper is not sufficiently clear.
Primarily, the approach is not described precisely enough for a reasonable reader to implement the approach described in the paper.
I included a number of quotations below that were not sufficiently clear to me.

* The definitions of "vectorization factor" and "interleaving factor" in Section 1 are not clear.
* "In our implementation, we use PrograML to generate code graphs.". Given that PrograML is used, what is the purpose of learning the binary classifier for link prediction
* "The forward pass allows the model to find the probability of the existence of an edge by computing a score between incident nodes with a function f". What is this function $f$?
* "Given a sequence of graphs G(V, E)..." What do these sequences of graphs represent? Is this a sequence of graphs for a specific input example, or is this sequence describing the training procedure?
* "We next learn a binary classifier ... by performing link prediction". To be clear, the binary classifier is predicting whether a link exists or not? Or does the training process use link prediction?
* "... evaluated with any metrics such as area under curve..." is AUC the evaluation metric you use?
* "...link score..." What is a link score?
* I don't understand the forall notation in equation 2.
* Equations 3 and 4 both define h_i^(l+1). Is this a typo?
* "Therefore, by adopting this approach, the size of the node embedding is the same as the size of the vocabulary". Is this ultimately the dimensionality of embedding that the GNN learns?
* "... is that two graphs could be identical but the vectorization parameters are totally different, due to different number of iterations in the loops". Maybe I misunderstand the embedding technique, but if two graphs are identical wouldn't that imply that their loop conditions are identical? If that is the case, how would there be different numbers of loop iterations?
* What are loop counts? I see a potential definition of these as "... the number of loops in each file of a benchmark", but I don't understand what this is.
* "... and further refines them in terms of better accuracy and performance improvement over O3 via graph supervised learning" what refining is this talking about?
* "...the state space consists of hidden feature embeddings learned from the supervised model". Do these states change during the execution of the agent? How?
* "The autograph framework incorporates GraphSAGE (Hamilton et al., 2017) to extract feature embeddings in an unsupervised fashion..." What are these feature embeddings used for? I don't see this described anywhere in Section 3.


## Quality

The quality of the evaluation is also not sufficient.
I am not convinced that the presented metrics (accuracy) are appropriate for evaluating the approach.
I am also not convinced by the speedup results: the train/validation/test splits aren't clear, I'm not convinced by the choice of baselines, there's no information provided on the transfer learning methodology, and the presented results for neuro-vectorizer do not align with those in the neuro-vectorizer paper.

* How is the dataset split into train/validation/test? Given that these benchmarks may share kernels, is there any deduplication performed? (Or is this not an issue)
* What is the definition of accuracy for the experimental results? Assuming it means just matching the brute-forced solution, is comparing the KL true label accuracy / divergence the best metric? This is not a binary problem -- some incorrect choices are better than others.
* How long (in terms of wall-clock time) does it take to train the model? To evaluate? To generate the brute-force solutions?
* Does autograph ever result in kernels slower than O3?
* The neuro-vectorizer paper (Figure 8) shows that neuro-vectorizer consistently achieves something close to 90% of brute force's speedup. Figure 6 in this paper shows a result closer to 50%. Why is there such a large discrepancy?

## Novelty

The paper is sufficiently novel, in that it proposes a new architecture for loop vectorization.
I do have one concern, which is that the paper itself overclaims the novelty:
* "autograph is the first to propose a structured learning framework for auto-vectorization" (Section 2): I don't think that this is true. https://dl.acm.org/doi/10.5555/3454287.3455597 also uses a graph neural network to learn a vectorization policy (albeit, using SLP rather than loop vectorization).

## Reproducibility

As discussed in the clarity section above, the paper is not sufficiently clear to be reproduced.
Beyond that, I don't see significant barriers to reproducing the results of the paper.

summary_of_the_review: Reject.
The method is not sufficiently clear, such that a reasonable reader could not reimplement the approach.
The evaluation is not sufficiently high quality, such that the presented results are not convincing.

I would be willing to raise my score conditioned on:
* The authors providing a significantly more clear description of the approach.
* The authors explaining the choice of accuracy metric in the evaluation (or present a different metric)
* The authors clarifying the train/validation/test split methodology and the transfer learning methodology
* The authors explaining the discrepancy between the neuro-vectorizer results in this paper and the results in the neuro-vectorizer paper


-------

## Update after author response

Thanks to the authors for the detailed response. Based on this response, I'm raising my score to a weak accept. Specifically:
* The revision includes a much more clear description of the approach (page 4)
* The authors correctly pointed out that I missed the definition of accuracy in the original submission. Though I still think this is a weak metric, I agree with the authors that the speedup metric is ultimately the one that matters (which the paper does report).
* The author response does not state whether or not any kernels are shared between the training/test dataset (and further, do not mention any validation set). The author response is also not sufficiently clear on the response about transfer learning: my point of confusion was whether or not any fine-tuning is performed on the other task (as is typically implied by the term transfer learning). The author response implies that no fine-tuning is performed, but is not 100% clear on that.
* The explanation of the neuro-vectorizer performance is satisfactory. Though it's surprising that neuro-vectorizer does not perform anywhere near as well as in the original paper (and raises some questions about whether it is a fitting baseline as-is), I don't view this as a reason to reject this paper.

If the paper is accepted, I would still like for the authors to include the following in the final version of the paper:
* Stating to what extent the different applications in the train and test set share kernels (or if they do not share any kernels)
* Stating how hyperparameters were tuned (given the lack of a validation set, I'm assuming that everything was tuned to maximize training set accuracy, but this should be explicitly stated)
* Making clear whether any fine-tuning or other transfer learning methodology was performed, or whether the transfer learning test is just a test of out-of-distribution generalization
* Including this discussion of neuro-vectorizer results
* Further improvements to the clarity of the approach
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: This paper proposes a method for predicting parameters of compiler optimizations related to vectorization of loops. It leverages graph representations of code and deep reinforcement learning. The main debate in the reviews and rebuttals is around novelty. Reviewers point out that the problem setup and deep RL formulation are shared with NeuroVectorizer, and that adding structure to learning in the context of vectorization has been done by Mendis et. al (2019). Thus the perceived originality is low. Authors respond by saying that Mendis et al tackle a different variant of vectorization, so it is not a proper baseline, and they, e.g., argue for the importance of their chosen in the response to Coc4 by saying that it's commonly occurring in practice. Unfortunately, IMO, this is missing the point. The question isn't whether this is the first paper to tackle this particular variant in this particular way. The question is for the ICLR audience, why the distinction matters. I wish that the paper (and rebuttals) would spend more time on what unique representation learning challenges arise in their setting, and what new representation learning insights arise based on their experiments, compared to the baselines being discussed.
justification_for_why_not_higher_score: Unresolved questions about originality with respect to the representation learning problems being tackled.
Decision: Reject