Venue: ICLR 2021 Conference

Reviews:

final_score: rating: 5: Marginally below acceptance threshold
review_content: This paper proposed a code similarity system called MISIM, which uses a context-aware semantic structure to represent code and calculate the similarity scores using a neural-based scoring algorithm. The context-aware semantic structure (CASS) is a structural representation of code specifically designed to lift semantic meaning from code syntax and provide an extensible representation that can be augmented. The neural-based similarity scoring part is implemented with deep neural networks, where authors tried 1). bag of features; 2). RNN, and 3). GNN. They found GNN performs the best. The experiment results show that MISIM can outperform code representing models like NCC and code2vec to detect code similarity.  


Pros:
1.	The context-aware semantic structure for code representation is  interesting. The configurable code structure representation gives the model the flexibility to handle multiple tasks.  
2.	This paper is well organized and easy to follow. The demonstration figures are clear and understandable. 

Cons:
1.	This paper does not give detailed guidance on what configuration should be used for various circumstances. As shown in the appendix, different configurations will have a considerable influence on the experimental results. It is difficult for other researchers to apply CASS to different scenarios until a detailed configuration explanation is provided. 

2.	Another weakness is that some important baselines are missing. This paper compares 3 different baselines: code2vec, NCC, and aroma, in which code2vec and NCC are designed for general code representation. Aroma is a code search system that locates code snippets in code corpus and makes code snippets recommendation. None of them are specially proposed for pair-wised code similarity comparison, as in this paper's experimental settings. There are many pieces of research about measuring code similarities such as CCLearner, CDLH, and DeepSim. Without comparison with those approaches, it is hard to convince others of the superiority of MISIM.


3.	The author emphasizes code similarity is now a first-order problem that must be solved in the introduction part, but the lack of relevant statics and examples makes this argument weak. Besides, the claim that AST representation will mislead code similarity systems into learning too much syntax but semantics also lacks solid evidence to support. Please provide some concrete examples that modeling those "syntax" in AST actually HURTS the performance, i.e., state-of-the-art methods learning from ASTs (e.g., the methods listed below) performs worse than the proposed method.

References: 
[DeepSim] Zhao, G., & Huang, J. (2018, October). Deepsim: deep learning code functional similarity. In Proceedings of the 2018 26th ACM Joint Meeting on European Software Engineering Conference and Symposium on the Foundations of Software Engineering (pp. 141-151).

[CCLearner] Li, L., Feng, H., Zhuang, W., Meng, N., & Ryder, B. (2017, September). Cclearner: A deep learning-based clone detection approach. In 2017 IEEE International Conference on Software Maintenance and Evolution (ICSME) (pp. 249-260). IEEE.

[CDLH] Wei, H., & Li, M. (2017, August). Supervised Deep Features for Software Functional Clone Detection by Exploiting Lexical and Syntactical Information in Source Code. In IJCAI (pp. 3034-3040).

review_title: Review for "MISIM: A Novel Code Similarity System"

final_score: rating: 7: Good paper, accept
review_content: The main contribution of this paper is the CASS structual feature for source code and its high performance. The advantages of CASS are that it doesn't require compliation and even not completement, also the fast building time of the tree and the global attributes table, which lead the MISIM system can work in a live scenario.

Three kinds of deep learning models (GNN, RNN and BoF) follow the CASS representation to demonstrate its performance on two code similarity experiments. The scoring part uses a Siamese network to evaluate the different kinds of distance which is reasonable for solving this task.

Questions:
1. How CASS preserve the sementic information? It's more like syntax to me in its current form.
2. Could the CASS representation be extended to other software engineering jobs, such as code summarization, bug detection/repair or auto completion.
3. The phase 1 (CASS featurization) can be replaced by other code representation methods, such as CuBERT[1]. The power of CASS should be demonstrated through necessary comparsions.
4. Last, will the authors release their code to public?

[1] Learning and Evaluating Contextual Embedding of Source Code, ICML 2020


review_title: A new structual feature for source code

final_score: rating: 5: Marginally below acceptance threshold
review_content: ### Summary ###

The paper presents a technique for code similarity. The paper combines two ideas: (1) a new “context-aware” representation for code which is used as a basis for computing code embeddings, (2) code similarity based on cosine similarity between the embeddings.

### Strengths ###

* The experimental results significantly improve previous work.

### Weaknesses ###

* Looks like an extension of AROMA, with a slightly more general approach for feature engineering, but no significant innovation beyond that. 

* The code2vec baseline is old and weak (should have used code2seq). 

### Questions for Authors ###

* You need to provide specific human customization per programming language. Can't this customization be learned itself? For example by using a Tree RNN to represent different abstractions of subtrees as needed?

* CASS seems to be closely related to an AST. In fact, the mapping of several syntactic elements (folding compound statements) could be viewed as a particular design choice when abstracting a syntax tree to an abstract syntax tree (in your case reflected in "node lables").

* The GAT includes a very shallow abstraction of function signatures. As such, I am curious about it's contribution to the final accuracy. Do you have an ablation study of the results obtained when GAT is omitted? I assume it has a rather different contribution across different languages?

* code2vec is a weak baseline. You should at least consider code2seq as it uses a significantly more powerful encoder.

* I wonder what would be the results using a textual sequence encoder as a baseline.


### Improving the Paper ###

* It would help to have a comparison of CASS+GAT to a simple AST-based approach. Keep your entire pipeline identical, including the linearization of the tree mentioned in B.2, and show what is the gain from the CASS per-language customization. I am sure that CASS+GAT would do better, but I conjecture that their contribution would be marginal in many mainstream languages.

* The comparison with Code2Vec is not clear to me. What do you mean by "we feed the AST paths from all function(s) in a program into the neural network and train it using the metric learning task described in Section 2.2.2.". Is this trained separately per program? 

### Minor questions and comments ###

* This was a strange point to emphasize in Definition 1: "A child node is either an internal node or a leaf node. An internal node has at least one child node while a leaf node has no child nodes."

* Page 15, "pretended" -> prepended







review_title: review

final_score: rating: 4: Ok but not good enough - rejection
review_content: This paper addresses the problem of code similarity detection and proposes an approach called MISIM.

Essentially, this paper proposes a tree representation, called context-aware semantic structure (CASS), which is very similar to the abstract syntax tree (AST). Then, different neural architectures (e.g., GNN or GRU) process the CASS and encode the program as a vector for similarity modeling. 

Major concerns:

1. While the title of this paper claims "a novel code similarity system," I do not feel the model novel at all, as it just follows a standard Siamese structure. 

The proposed CASS is very similar to abstract syntax tree, and there has been many structure variants proposed in previous work, like contextual flow graph, simplified parse tree. CASS is yet another, and I do not feel the need of CASS is well motivated. 


2. Experimentally, the paper shows performance improvement on GCJ and POJ-104 datasets, compared with a few baseline systems, such as code2vec, NCC, and Aroma. 

However, there's no controlled experiments on the choice of different structures, such as the AST and contextual flow graph. To be convinced, I expect 

1) The authors show the results of controlled experiments with different structures. 

2) The authors show the generality of the CASS in different tasks, such as clone detection. 

3) The authors have a deeper analysis on CASS. What's the additional information encoded in CASS than a standard AST? 

Minor:
    code2vec, NNC --> code2vec, NCC
review_title: lack of novelty and controlled comparison

Meta-review: This paper studies the problem of computing a similarity measure between two pieces of code. The main contributions are a configurable alternative (CASS) to abstract syntax trees (ASTs) for representing code and a model for embedding these structures within a Siamese net-like architecture. While parts of the ICLR community that make use of ASTs would likely find interest in the options provided by CASS and the associated experiments, the contribution is mostly around feature engineering of AST-like structures for one specific application, which is quite niche. The machine learning modeling appears fairly standard. Thus in total, I don’t see enough here to recommend acceptance.
Decision: Reject