Venue: ICLR 2022 Conference

Reviews:

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: A method to learn to reason about programs is presented. The core novelty is an intermediate program representation as a set of relations (in the logics sense), derived from existing program analysis tools. By translating each tuple into a node, and references between relations into edges, the set of relations is transformed into a graph. Finally, graph learning techniques (from the random walk view of the world) are used to learn to predict the target label from the obtained graphs.
main_review: I'm both excited and disappointed by this paper. The idea of using relations as obtained by the Semmle tool as intermediate representation is elegant, reasonably extensible, and opens a new path to integrating more knowledge from the expert practicioners into the ML4Code area. I believe that this is a valuable contribution that should be published.

On the other hand, the remainder of the paper has a number of substantial problems:
1. the translation of the database of program facts into a binary graph is not described well:
  * (Q1a) how do tuple attributes exactly contribute to the node features? how are different data types (stmt types / IDs / line spans / ...) embedded? If I understand p5 top correctly, you're simpling bag-of-words embedding WordPieces here - does this even take into account what attribute a token comes from?
  * (Q1b) how fine-grained are edge types? Concretely, when translating something like the "call" relation in Fig. 1, are separate edge types generated for callees and callers? The main text says "for any two relations R and S connected via referential integrity, an R_S edge type is defined". Alg. 2 says "l [...] is a referential integrity constraint", which is an undefined term in the paper. 
  * (Q1c) have you considered representing these relations as hypergraphs (with each named relation corresponding to a hyperedge type, and each tuple from a relation instantiation corresponding to a hyperedge) and then using HyperGNN models?

2. the random walk strategy is an interesting trick to break through the limitations of number of message passing steps in GNNs / number of layers in Transformers, but leaves many questions:
  * (Q2a) You say "each individual part of the embedding tensor gets its own learnable positional embedding" - it's not clear to me what that actually means - are you describing an embedding lookup here, or do you mean to say that the relation names / relation attributes / edge types have separate positional encodings? How do you ensure that they are well-aligned with each other? Do you have ablations that show that your positional encodings contribute anything at all?
  * (Q2b) You are using mean pooling to obtain the representation of a path, but that is likely to be sensitive to the length of the path. Have you considered using attentional pooling (e.g., most simple, introducing a [PATH] pseudo-token and use its final representation, or at least a gated, weighted sum)? After all, you're using a similar attentive approach for the aggregation of paths by going through DeepSet.

3. the experiment section feels not well-structured (some explicit guideposting what key points you are trying to support would help with this), and the considered baselines are underdescribed and seem incomplete.
  * (Q3a) The hypers described in App. A for the baselines are surprising in that they don't match the hyperparameters proposed by the original authors - in particular, GREAT (4 layers here vs. 6 or 10 layers), Code2Seq (4 LSTM layers here vs. 1 in original), and GGNN (4 layers here vs. 8 layers). How where these parameters chosen? In particular, the low number of layers for GREAT and GGNN substantially limits their performance, and so this is quite surprising.
  * (Q3b) What edges are presented to the GREAT / GGNN models, which consume more than just the AST? This is not described at all.
  * (Q3c) How sensitive are your results to different choices of the random walk specification? You list learning the spec as future work, but what is the effect of choices here? How did you arrive at your current default spec?


summary_of_the_review: The paper presents a novel way of automatically extracting valuable information from programs before using standard ML tools to process them. While the core idea is exciting, the writing is in some places imprecise (see above) and the experimental evaluation is missing many details and leaves many question marks on the empirical claims. I like this paper, but I think at this time, it's shortcomings are too significant to allow for acceptance at ICLR.


Update after rebuttal
----------------------------
The authors did a great job of responding to my concerns and answering my questions, alleviating most of my concerns.

I would note that the current draft does not integrate the further explanations and results responding to my question (Q1a).

The draft also has not been updated to include the additional experimental results and ablations that the authors provided during the discussion period, and I would encourage them to include all of these in the next revision of the paper, either embedded into the main text or in an appendix.

Overall, I'm now satisfied that the paper is presenting an empirically meaningful improvement over earlier works. As the reviewing discussion highlighted, constructing program graphs as proposed by Allamanis et al. (and following works based on the extraction of semantic program information) is highly non-trivial and not an off-the-shelf product. I view the core contribution of the current submission as the empirically validated insight that this can be solved by the Semmle static analysis tool. While many open questions about design choices here remain, this is a substantial step forward that deserves publication at ICLR, as I expect it to be of significant interest to the ML4Code subcommunity at the conference.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: The paper presents a framework for creating task-specific program representations for deep learning based on program analysis and random graph walks. The basic idea is to use program analysis to create a graph-based representation of the program, capturing both syntactic and semantic relationships between elements. This graph-based representation of the program is then encoded by embedding a set of (random) walks in the graph.
The paper evaluates the framework using four tasks: three tasks that only require intra-procedural program analysis (var-misuse, infer exception type, predict where a definition is used) and one that required inter-procedural analysis (variable shadowing).
CodeTrek is shown to outperform existing techniques in most benchmarks. 

main_review: Pros: 
* A general framework for generating a relational representation of programs for programming-related tasks
* The use of “declarative” tasks (which can be solved without a learned model, e.g. variable shadowing) to evaluate framework/approaches.
* A general methodology that specifies a TODO list (defining queries, random walk bias, anchors, etc.) for tackling a new task (it doesn’t mean that these steps are easy).
* The use of long-distance and cross modules dependencies to solve the tasks.


Cons: 
* This paper uses standard neural architectures and the representation that is used is also rather low on novelty. 
* A lot of effort is required in preprocessing, before feeding the NN. This includes an expert’s knowledge to determine task-specific queries and task-specific random walk bias (although these were not used in the experiments). To conclude, it feels like that given a new task, it’s not straightforward to apply this approach. 
* This paper has less of a contribution to the ML community but has a potential impact on the SE community. The only ML novelty I can spot here is the use of the graph to represent the schema, which can be then processed by some NN. 
* This approach is limited to cases where one has the whole project in hand, and the ability to perform (potentially heavy) static analysis of the code.

* Allamanis et al. have shown how to represent programs as graphs, including cases when the program information is enriched by a wide range of semantic information from program analysis. They used the resulting graph representation as input to GNNs, but there’s nothing that precludes the same input from being linearized (by standard graph walks) and used in sequential models. 

* Given the expressiveness of the framework, I was hoping for evaluation on tasks that require a more global and long-range kind of properties. Unfortunately, most of the paper focuses on tasks that have a local (intra-procedural) solution. While I understand this from the need to compare to existing baselines, I think that it does not play to your strengths.

* In your evaluation, you only experimented with classification tasks. It will be beneficial to see other kinds of tasks such as summarization and code generation.

* You mention in several places a “walk score”, but didn’t define it, and how it’s calculated and used.

* Table 3 is interesting. Does it mean that the expected benefit from the deep and often complicated semantic analysis is sometimes as low as 20%? Do you think sampling more paths from the AST could have narrowed the gap? 

summary_of_the_review: This paper proposes a general framework for applying deep learning (specifically DeepSet + Transformers) to code. The authors suggest the following steps for handling code-related tasks:
Building a relational database based on the code by applying pre-defined queries and user/task-specific queries.
Converting the database into a graph, defining some user/task-specific random walk bias and length, and user/task-specific anchor nodes.
Applying this random walk, encoding these using Transformers, and feeding the result to some task-specific neural network.

Overall I think it’s a good paper with a minor ML contribution. It shows that the use of the relational graph outperforms the common use of AST and data flow graphs. It has its drawbacks  - heavily dependent on expert knowledge per language and per task. Further, even experts can’t tell which are the best queries/random walk for the given task.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: This paper proposes a new program representation approach CodeTrek, which leverages a program analysis tool (Semmle) to produce the rich representation of the context for a program. Semmle is able to convert the program into a relational database that can capture the semantics behind the program, furthermore, it also supports extracting the task-specific semantics with the query language CodeQL to represent new semantic information. Then CodeTrek utilizes a biased graph-walk mechanism for pruning the context paths and feeds them to the transformer encoder to learn the path representations and followed by deepset to get a vector representation. The extensive experiments on four diver Python tasks: variable misuse, exception prediction, unused definition and variable shadowing prove that CodeTrek can outperform the current baselines i.e., Code2seq, GGNN, GREAT, CuBERT by a significant margin.
main_review: This paper is well written and the analysis of the experimental results are sufficient. But I have some questions to discuss.

(1) The novelty in the network architecture is not enough. CodeTrek spends too much weight on the description of Code2Rel and Rel2Graph procedure, however, the network architecture i.e., Transformer is simple. Furthermore, the idea of sampling walks is the same with Code2seq and the difference is mainly on the constructed graph/AST is different.

(2) This paper conducts some sensitivity analysis on the number of the sampled walks, however, the discussion of another important factor walk length (24 in the appendix) is missed especially for the exception task, which may be critical. So I would like to see the performance. 

(3) CodeTrek selects the anchor node or in other words, assign higher probabilities to some specific node types for example stmt, expr, it would be great to provide more insights on why choosing these nodes as the start point.

(4) In Rel2Graph, there are two types of relations i.e., base relations and derived relations. Derived relations are customized by the program analysis queries on the base relations, so it can be regarded as another featuring engineering. Is there a way to only use the base relations to construct the graph and see the experimental results? It would be important to see how much the derived relations weigh.

(5) In Table 1, the gap between GGNN and GREAT on Varmisuse is very obvious (0.69 VS 0.82), however, in the original Hellendoorn et al. [1], the gap is not obvious GGNN VS GREAT (0.792 VS 0.769). So what is the reason behind this, since the different dataset were used or the different graph structures? 

[1] GLOBAL RELATIONAL MODELS OF SOURCE CODE. Hellendoorn et al. 


summary_of_the_review: This paper proposes a new code representation technique that is based on the relational database to capture the program semantics for different tasks, however, the novelty in the model architecture is not enough and the proposed code representation approach is more like a feature engineering.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: The paper presents a framework called CodeTrek for processing and learning source code. The main idea is to represent code as a graph, manually design analyses to serve as additional relations, and perform a set of random walks on the graph. These walks can also be manually guided. Finally, the set of walks is encoded to represent program elements.
main_review: ## Pros
+ The proposed system shows strong empirical results.
+ The system is compared to a variety of structural and textual baselines.

## Cons
- The paper is very difficult to understand. Many details are not well defined nor explained. See below.

- Ease of use - the system has many "moving parts" that need to be manually designed. While this can be a good thing for experts, it limits the system's applicability to non-expert users and hinders its generalization to other domains, tasks, and languages. The authors claim that "A task developer need not be a machine-learning expert to bring in more semantic information about programs". However, there are many more aspects that the task developer needs to consider: which queries to write, how to write these queries, how to choose the starting node, how to bias the random walk, etc. These might make the proposed system to be very difficult to use for other tasks and languages.

- The paper describes a specific system, and its contributions in terms of machine learning are relatively minor. The paper also lacks a deeper analysis or an ablation study to pinpoint the exact contributing factors and discuss their alternative, nor concrete examples which will help readers to better understand the impact of the proposed method.


### Clarity

* The paper mentioned Semmle several times and mentioned that Semmle converts codebases into relational databases. It is not explained what is Semmle? How does Semmle work? How does it convert codebases into databases? Is it a commercial product? 

* Figure 1 is very nice visually but is still very difficult to understand. Some details are explained at the beginning of Section 2, but are still very hard to follow. If I did not miss anything, 
other details in this very detailed figure are not explained anywhere. Some explanations that do exist are very hard to follow and to connect with the figure, for example:

>"The derived information is stored in def, which, together with call, can bias the prediction of the best variable to replace a placeholder" 

I am not sure how to understand this sentence and what in Figure 1 should I look at.
This unclarity continues in Section 2, where no clear definitions nor explanations are provided. The paragraph "Code as a graph" is very unclear, and frequently uses terms that were not defined.

summary_of_the_review: Overall, the paper shows strong empirical results. However, I believe that this is an application with little novelty, and few lessons to be learned for the ICLR audience. Hence, I would tend to reject this in favor of more ML-heavy papers.
review_title: null

Meta-review: The paper presents a deep learning approach encodes codebases as databases that conform to rich relational schemas. Based on this, a biased graph-walk mechanism efficiently feeds this structured data into a transformer and deepset approach. The results shown a quite good, compared to other approaches present at ICLR. Moreover, one reviewer is strongly voting for accepting the paper, arguing that "that this paper is of significance to the ML4Code research community, as it shows how to offload the engineering cost of extracting semantic information from programs to a standard tool." Overall, I have really enjoyed reading the paper, and the use of relational database as codebase together with a transformers is sweat. On the other, it also presented in a rather engineering way, as pointed out by several reviewers, suggesting that some software engineering venue might be a better place for the work. But then ICLR had similar papers, and the present paper demonstrates a benefit of using a relational encoding. Thus, I weight the leaning towards rejects borderlines votes less and suggest an accept overall. We all should keep in mind that also deep neural architecture are full of design choices.
Decision: Accept (Poster)