Venue: ICLR 2020 Conference

Reviews:

final_score: rating: 3: Weak Reject
review_content: Using Deep Learning and especially, GNNs seems to be a popular area of research. I am no 
expert at optimizing code performance, so please take my review with a grain of salt. The algorithmic contributions of the paper are as following: 

(a) GNN that combines static code and dynamic execution trace. 
(b) Binary encoding of features leads to better performance in comparison to categorical and scalar representations. 

The results show that the proposed method outperforms existing methods on standard benchmarks in the program execution community. 

From a machine learning stand point, the contributions are straightforward and the results make sense. I have the following questions: 

(I) Authors argue that binary representations are better because of their hierarchical nature. They mention that they can generalize even if not all combinations of bits are seen, but a subset is seen in a manner that every bit has been flipped a couple of times. I don’t agree with this reasoning, as seeing the individual bits flip has no guarantee that a NN would generalize to a new combination of bits unless the distance in the binary code makes sense. Is there some special way in which the binary code is constructed?
 
(ii) Transfer learning experiments: Its unclear to me if the comparison presented in the paper is a fair one. Comparison is made against Ben-Nun et al. pre-training on LLVM IR. I am not sure how different is LLVM IR dataset from the algorithm classification dataset. If the dataset is very different, then obviously a lot of pre-training will only result in modest performance gain. What happens with Ben-Nun method is pre-trained on the same dataset as the proposed method? Also, what is the difference in performance between the cases when the proposed method is applied to algorithm classification with and without pre-training? 

Overall, the paper is a application of GNN to optimizing code execution. The technical innovations are domain-specific and do not inform the general machine learning community. Given lack of expertise in the area of program execution, I cannot judge the significance of the performance improvements reported in the paper. 

Given my current concerns, I cannot recommend acceptance. I might change my ratings based on the review discussions and the author’s responses to the above questions. 

review_title: Official Blind Review #2

final_score: rating: 8: Accept
review_content: This paper presents a novel improvement in methodology for learning code execution (at the level of branch-predictions and prefetching).  They combine static program description with dynamic program state into one graph neural network, for the first time, to achieve significant performance gains on standard benchmarks.

I would vote to accept this paper.  They appear to have developed a new model structure and interface to the program information (i.e. inputs to the model), and the design decisions appear thoughtful, sensible, and well-justified (e.g. use of assembly code).  The presentation is mostly clear, with a good balance of background material, method description, and experiment results.  

Taken at face value, the results are impressive, although I am not familiar enough with this field to assess the fairness of comparison against the baselines.  For example, it's a little unclear what the difference is vs previous baselines just from switching to source-code-as-input to assembly-code-as-input?

The study on memory representations (categorical vs scalar vs binary) is a helpful component which adds its own value, and the context for popularity of the alternatives is described.

Few details as to implementation are discussed, although the code is included in the submission, and after a quick glance appears substantial.
review_title: Official Blind Review #1

final_score: rating: 6: Weak Accept
review_content: The paper proposes using Graph Neural Networks to learn representations of source code and its execution. They test their method on the SPEC CPU benchmark suite and show substantial improvement over methods that do not use execution. 

The paper's main question is to answer how to learn code representations. The main novelty introduced in their approach is to build a graph representation not of high level code but of assembly code. They also develop a way to use what they call a "snapshot mechanism" that feeds limited memory states into the graph. The downstream consequences of their methods are improved methods for example for branch prediction. Interestingly NCF can be also used to represent programs for use in downstream tasks. This is demonstrated via transfer learning in an algorithm classification problem. The paper is well written and the background / related work makes it easy for the reader to understand the problem's relevance within the related literature. 

The results look well justified and empirically verified. 
review_title: Official Blind Review #3

Meta-review: This paper presents a method to learn representations of programs via code and execution.

The paper presents an interesting method, and results on branch prediction and address pre-fetching are conclusive. The only main critiques associated with this paper seemed to be (1) potential lack of interest to the ICLR community, and (2) lack of comparison to other methods that similarly improve performance using other varieties of information. I am satisfied by the authors' responses to these concerns, and believe the paper warrants acceptance.
Decision: Accept (Poster)