(Updated submission 11/20/2020) MISIM: A Novel Code Similarity System
Authors: Fangke Ye
Shengtian Zhou
Anand Venkat
Ryan Marcus
Nesime Tatbul
Jesmin Jahan Tithi
Niranjan Hasabnis
Paul Petersen
Timothy G Mattson
Tim Kraska
Pradeep Dubey
Vivek Sarkar
Justin Gottschlich
Venue:  (2021)
Keywords: Machine Programming
Machine Learning
Code Similarity
Code Representation

Abstract: Semantic code similarity systems are integral to a range of applications from code recommendation to automated software defect correction. Yet, these systems still lack the maturity in accuracy for general and reliable wide-scale usage. To help address this, we present Machine Inferred Code Similarity (MISIM), a novel end-to-end code similarity system that consists of two core components. First, MISIM uses a novel context-aware semantic structure (CASS), which is designed to aid in lifting semantic meaning from code syntax. We compare CASS with the abstract syntax tree (AST) and show CASS is more accurate than AST by up to 1.67x. Second, MISIM provides a neural-based code similarity scoring algorithm, which can be implemented with various neural network architectures with learned parameters. We compare MISIM to four state-of-the-art systems: (i) Aroma, (ii) code2seq, (iii) code2vec, and (iv) Neural Code Comprehension. In our experimental evaluation across 328,155 programs (over 18 million lines of code), MISIM has 1.5x to 43.4x better accuracy across all four systems.


Under review as a conference paper at ICLR 2021
MISIM: A NOVEL CODE SIMILARITY SYSTEM
Anonymous authors
Paper under double-blind review
ABSTRACT
Semantic code similarity systems are integral to a range of applications from code recommendation to automated software defect correction. Yet, these systems still lack the maturity in accuracy for general and reliable wide-scale usage. To help address this, we present Machine Inferred Code Similarity (MISIM), novel end-to-end code similarity system that consists of two core components. First, MISIM uses novel context-aware semantic structure (CASS), which designed to aid in lifting semantic meaning from code syntax. We compare CASS with the abstract syntax tree (AST) and show CASS is more accurate than AST by up to 1.67x. Second, MISIM provides neural-based code similarity scoring algorithm, which can be implemented with various neural network architectures with learned parameters. We compare MISIM to four state-of-the-art systems: (i) Aroma, (ii) code2seq. (iii) code2vec, and (iv) Neural Code Comprehension. In our experimental evaluation across 328,155 programs (over 18 million lines of code), MISIM has 1.5x to 43.4x better accuracy across all four systems.
INTRODUCTION
The field of machine progranminng (MP) is concerned with the automation of software develop- ment (Gottschlich et al. 2018). In recent years, there has been an emergence of many MP systems,
due, in part, to advances in machine learning, formal methods, data availability, and computing
efficiency (Allamanis et al. 2018a| Alon et al. 2018) 2019b a Ben-Nun et al. 2018 Cosentino et al.
2017 Li et al. 2017 Luan al. 2019 Odena & Sutton 2020 Tufano et al. 2018 Wei & [0]7
Zhang et al. 2019; Zhao & Huang 2018). An open challenge in MP is the construction of accurate
code similarity systems. Code similarity is the problem of determining if two or more code snippets
have some degree of semantic similarity (or equivalence) even in the presence of syntactic divergence. At the highest level, code similarity systems aim to determine if two or more code snippets are solving a similar problem, even if the implementations they use differ (e.g., various algorithms of sort () (Cormen et al. 2009)).
Historically, code similarity systems have been considered auxiliary feature that aim to improve programmer productivity with tools such as code recommendation, automated bug detection, and transformation for small kernels (c.g.. stencils), to name few (Allamanis et al.
Minguage-to-laguaee
20186 Ahmad et al. 2019; Bader et al. 2019 Barman et al. 2016 Bhatia et al., 2018 Dinella et al.
2020 Kamil et al.] 2016 Luan et al.] 2019 Pradel & Sen 2018). Yet, these systems still lack the maturity in accuracy for general and reliable wide-scale usage. In particular, without largely accurate
code similarity systems automate significant parts of our software development, we believe the explosion of heterogeneous software and hardware may become an untenable problem that software
developers will not be able to navigate (Ahmad al. 2019 Batra al. 2018 [Bgddan al. 2019) Chen et al. 2020: Deng et al. 2020; Hannigan et al. 2019)) Yet, as others have noted before us, even some of the most fundamental questions in code similarity have no clear answers, such as the proper
structural representation of code for a particular similarity problem (Alam et al. 2019 Allamanis et al. 2018b [ecker Gottschlich] 2017 Ben-Nun et al.) 2018 Dinella et al.) 2020 Iyer et al. 2020
Luan et al. 2019). In this paper, aim to address some of these questions.
While prior work has explored some structural representations of code in the space of code similarity, these explorations are far from complete. The abstract syntax tree (AST) is used in the code2vec and code2seq system (Alon et al. 2019b a). a novel structure called the contextual flow graph (XFG) is used by Neural Code Comprehension (NCC) (Ben-Nun et al.) 2018), and new structure called the simplified parse tree (SPT) is used by Aroma (Luan et al. 2201). While each of these representations have benefits in certain contexts, have found that they possess or more limitations when used

---

Under review as a conference paper at ICLR 2021
in the broader context of code similarity that may limit their practical application. For example, the AST while principally valuable for compilers - is syntax driven, which can often mislead code similarity systems into learning too much syntax and not enough semantics (i.e., the meaning behind the syntax). The XFG is obtained from an intermediate representation (IR), which requires code compilation; this limits its to only compilable code. Although the SPT is structurally driven (not syntax driven), it does not always resolve syntactic ambiguities, which may result in semantic obfuscation that prevent from observing semantic variation caused by contextual syntactic differences. In Section 4 we evaluate how these limitations impact code similarity accuracy.
Learning from these observations, we attempt to address some of the open questions around code similarity with our novel end-to-end code similarity system called Machine Inferred Code Similarity (MISIM). this paper, we principally focus on two main novelties of MISIM and how they may improve code similarity analysis: (i) its structural representation of code, called the context-aware semantic structure (CASS), and (ii) neural-basee learned code similarity scoring algorithm. These components can be used individually together as we have chosen do.
This paper makes the following technical contributions:
We present Machine Inferred Code Similarity (MISIM), a novel end-to-end code similarity system. We present MISIM's context-aware semantic structure (CASS), novel structural representation of code specifically designed to (i) lift semantic meaning from code syntax and (ii) provide an extensible representation that can be augmented as needed (c.g., as new programming languages (PLs) emerge, existing PL contextual syntax ambiguities are discovered, etc.). We also experiment AST and CASS on representing code semantics. Our preliminary result shows that CASS can be up to 1.67 more accurate.
We present MISIM's ' open-ended deep neural network (DNN) backend that learns the similarity scoring algorithm for given code corpus and show its efficacy across three DNN topologies: (i)
bag-of-fcatures. (ii) recurrent neural network (RNN), and (iii) graph neural network (GNN). [
We compare MISIM four state-of-the-art code similarity systems: (i) code2vec, (ii) code2seq. (iii) Neural Code Comprehension, and (iv) Aroma. Our experimental evaluation, across 328, 155 CIC++ programs comprising of over million lines of code, illustrates that MISIM more accurate than all four systems, across all experiments, ranging from 1.5x to 43.4x.
2
MISIM SYSTEM
Phase D: source code
Phase 1: CASS featurization
Phase 2: code simillrrty scoring
Gestllatristss Table
Fundion 1
Shippit CASSS rspresentation
Fireton
GINI
Code In C/oce
Featurined
Twin
Simiintty metricee le cocine, dot aroduct, etc.)
Score
hineton
Snippet CASS
Sub- netenek Jegs
te representation
Turetion M
Code Snippet 2 /eg. c/c-+
CASS
Featurized CASS
DNN2
Inference
Training
CASS representationn
representation
Snippet 1 EASS
Twin DMN
Swrislied
Manual
Similartty metric [eg. cosine, dot product, stc.)
Metric leamiin (6-6, cirde loss, triplet loss, etc.
Optimization
2nipper CASS CASS morresentation
Twin DNN
Neural features
Recurrent neural
Graph neural net
Figure Overview of the MISIM System.
We acknowledge that this is a non- non-exhanne exhaustive list of all possible neural network architectures that can be used with MISIM. We explore these three for two reasons: (i) to demonstrate the diversity of DNNs that can be used with MISIM and to illustrate how different DNNs have measurable impact MISIM's accuracy.
2

---

Under review as a conference paper at ICLR 2021
In Figure we provide overview of MISIM's system diagram. A core component of MISIM is the novel context-ave aware semantic structure (CASS), which aims to capture semantically salient properties of the imput code. Moreover, CASS is designed to be context-aware That is, it can capture information that describes the context of the code (e.g., parenthetical operator disambiguation between function call, mathematical operator precedence, Boolean logic ordering, etc.) that may otherwise be ambiguous without such confext-sensitivity. Once these CASSes are constructed, they are vectorized and used as input to a neural network, which produces a feature vector for corresponding CASS. Once a feature vector is generated, code similarity measurement (e.g., cosine similarity (Baeza- Yates& & Ribeiro-Neto 1999)) calculates the similarity score.
2.1 CONTEXT-AWARE SEMANTIC STRUCTURE (CASS)
We have designed CASS with the following guiding principles: (i) it should not require compilation, (ii) it should be a flexible representation that captures code semantics, and (iti) should be capable of resolving code ambiguities in both its context sensitivity to the code and its environment. The first principle (i) originates from the observation that unlike programs in higher-level scripting programming languages (e.g.. Python (Van Rossum & Drake 2009), JavaScript (Flanagan) 2006)), C/C++ programs found "in the wild" may not be well-formed (e.g., due to specialized compiler dependencies) or exhaustively include all of their dependencies (e.g., due to assumptions about library availability) and therefore may not compile. Moreover, for code recommendation systems that are expected to function a live setting, requiring compilation may severely constrain their practical application. We address this by introducing structure such that it does not require compilation (Section 2.1.I). The second (ii) and third (iii) principles originate from the observation that different scenarios may require attention to different semantics (e.g., embedded memory-boure systems may prefer to use algorithms that do not use recursion due to a potential call stack overflow) and that programming languages (PLs) evolve and new PLs continue emerge. We attempt address these issues with CASS's confi configuration categories (Section 2.1.2
2.1.1 CASS's TREES AND GLOBAL ATTRIBUTES TABLE
In this section, we provide an informal definition of CASS (a formal definition can be found in Appendix (A). The CASS consists of one more CASS trees and an optional global attributes table (GAT). A CASS tree is a tree, in which the root node represents the entire span of the code snippet. During the construction of CASS tree, the program tokens mapped its corresponding node label using the grammar of the high-level programming language. A CASS's GAT contains exactly one entry per unique function definition in the code snippet. GAT entry currently includes only the input and output cardinality values for each corresponding function, but can be extended as new global attributes are needed. A simple example of this is illustrated in Figure 2 where a code snippet and its corresponding CASS tree and GAT are shown. Haimaannd
int isom (int start, int and)
sun
for (int iestart; iseend;
9
retimS
Function:
sum()
-+1)
BouS
carcinaliy
Input
2
return suis
AIM
PHR
corcinality
Dutput
1
RWAR
AVAR
Figure 2: Sum Function and One Variant of Its Context-Aware Semantic Structure.
2.1.2 CASS CONFIGURATION CATEGORIES
In general, CASS configurations can be broadly classified into two categories: language-speciinn and language-agnostcr Options for each of the configuration categories are described in Table[]
Language-specific configurations (LSCs). Language-specific configurations are meant to capture semantic meaning by resolving a syntactic ambiguity present in the concrete syntax tree. It also introduces specificity related to the high-level programming language. For example, the parentheses operator is overloaded in many programming languages to enforce an order of evaluation of operands in expression well enclose list of function arguments, amongst other things. CASS
3

---

Under review as a conference paper at ICLR 2021
Table 1: CASS Configuration Options.
Cauegory
Type
Option
No change (Arons is original conliguration)
Language-ppesee specific
A. Node Prefix Lahel
Add prefix to each internal node lahel
prefix parenthesis node (CIC+ Specific)
0. No change (Aroma is original configunition
Compound Statements
relevant to compound statements
2. Replace with '(e)"
No change (Aroma''s original conliguraiion)
Remove all features relevant to global variables
Language-agnostic
C. Global Variables
Replace with *#GVARR
3. Replace with 'WVAK' (the label for local variables)
No change (Arema''s original configunation
D. Global Functions
Remove all features relevant to global functions
Remove function identifier and replace with "EXFUNCC
E. Function PV Cardinality
Include the inpus and ourpur cardinality per function
change
disambiguates these by explicitly embedding the semantic contextual information in the CASS tree nodes using the node prefix label (defined in Appendix A)).
(A) Node Prefix Label. The configuration options for node prefix label correspond to various levels of semantic to syntactie information. In Table []] option corresponds to the extreme case of a concrete syntax embedding, option 1 corresponds to eliminating irrelevant syntax, and option is principally equivalent to option I, except it applies only to parentheticals, which we have identified through empirical evaluation to often have notably divergent semantic meaning based on context.
Language-agnostic configurations (LACs). LACs can improve code similarity analysis by unbind- ing everly-specinee semantics that may be present in the original concrete syntax tree structure.
(B) Compound Statements. The compound statements configuration option enables the user to control how much non-terminal node information is incorporated into the CASS. Option 0 is equivalent to Aroma's SPT, option omits separate features for compound statements altogether, and option 2 does not discriminate between compound statements of different lengths and specifies special label to denote the presence of a compound statement.
(C) Global Variables. The global variables configuration specifies the degree of global variable- specific information contained in CASS. The configuration options for global variables provide the user with the ability to control the level of abstraction essentially binding or unbinding global variable names as needed. If all code similarity analysis will be performed against the same software program, retaining global variable names may help elicit deeper semantic meaning. If not, unbinding global variable names may improve semantic meaning.
(D) Global Functions. The global functions configuration serves the dual purpose of (i) controlling the amount of function-specifie information to featurize and (ii) to explicitly disambiguate between the usage of global functions and global variables (a feature that is absent in Aroma's SPT design).
(E) Function I/O Cardinality. The function I/O cardinality configuration aims to abstract the semanties of certain groups of functions through input and output cardinality (i.e., embedded semantic information that can be implicitly derived by analyzing the number of input and output parameters of a function).
We have found that the specific context in which code similarity is performed seems to provide some indication of the optimal specificity of the CASS configuration. In other words, one specific CASS configuration is unlikely to work in all scenarios. To address this, CASS provides a number of options to control the language-specific and/or language-agnostic configurations to enable tailored CASS representations for particular application contexts. We discuss this in greater detail in Appendix A.
2.1.3 CASS vs AST
Some recent research on code representation uses the baseb representation (Dinella et al. 2020) or AST paths (Alon et al. 2019b]a) a). In this subsection, we explore how AST and CASS perform on the task of code semantic representation described here.
Analytically deriving the optimal selection of node prefix labels across all C/C++ code may be untenable. To accommodate this, we currently provide levels of granularity for C/C++ node prefix labels CASS.
4

---

Under review as a conference paper at ICLR 2021
We compared the code similarity performance of ASTs and CASSes the test set of POJ-104, as shown in table by transforming both kinds of representations into feature vectors. using the same method described in (Luan al. 2019) and compute the similarity scores using dot or cosine similarity, For each program in the dataset, we extracted its CASS under three different configurations 0-0-0-0-0 | the base configuration, and 2-1-3-1-1/1-2-1-044 the best/worst performing configuration according to our preliminary evaluation of CASS (see Appendix for details), We also extracted the ASTs of function bodies in program. Each syntax node in the AST labeled by its node type, and an identifier (or literal) node also gets a single child labeled by the corresponding identifier name (or literal text).
Table 2: Test Accuracy for AST and three CASS configurations on POJ-104
Method
MAP@R (%)
AP (%)
AUPRG (%)
AST-Dot
45.12
35.98
97.29
AST-Cos
47.39
45.31
98.41
CASS (0-0-0-6-0--oul
$2,09
45.99
98.42
CASS (0.0:0.0.0))- Cos
55.12
55.4
99.07
CASS (2-1-3-1--)--oo
55.59
48.31
98.62
CASS (2-1-3-1-1)-Cos
60.78
60.42
99.31
(1-2-1-00)-Dot
CASS (1-2-1-0.0) Cas
57.99
54.75
99.06
As shown in Table 2 CASS configurations show an improvement in accuracy over the AST for up to 1.67x across three evaluation metrics described in Appendix C33 To better understand the performance difference, we investigated few solutions for the same problems from the POJ-104 dataset. One of the interesting observations we found is that for the same problem, a solution may have a different naming convention for local variables than that of another solution (e.g., English vs Mandarin description of variables), but the resulting different variable names may carry the same semantic meaning. AST uses variable names in its structure, but CASS has the option to not use variable names. Thus the erasure of local variable names in CASS might help in discovering the semantic similarity between code with different variable names. This might explain some of the performance difference between AST and CASS in this experiment.
2.2 NEURAL SCORING ALGORITHM
MISIM's neural scoring algorithm aims compute the similarity score of two input programs. The algorithm consists of two phases. The first phase involves a neural network model that maps a featurized CASS to real-valued code vector. The second phase generates a similarity score between a pair of code vectors using a similarity metric For the remainder of this section, we describe the details of the scoring model, its training strategy, and other neural network model choices.
2.2.1 MODEL
We investigated three neural network approaches for MISIM's 's scoring algorithm: (i) graph neural network (GNN) (Zhou et al. 2018), (ii) recurrent neural network (RNN), and (ii) bag of manual features (BoF) neural network. We name these models MISIM-GNN, MISIM-RNN, and MISIM-BI respectively. MISIM-GNN performs the best overall for our experiments, therefore, we describe it in detail in this section. Appendix has details of the MISIM-RNN and MISIM-Bol models.
MISIM-GNN. MISIM-GNN's architecture is shown in Figure 3 For this approach, an input program's CASS representation is transformed into a graph. Then, each node the graph embedded into a trainable vector, serving the node's 's initial state. Next, a GNN is used to update each node's ' state iteratively. Finally, a global readout function is applied to extract a vector representation of the entire graph from the nodes' final states. We describe each of these steps in more detail below.
Input Graph Construction. We represent each program as a single CASS instance, Each instance can contain one more CASS trees, where each tree corresponds to unique function of the program.
1 *Configuration 0-0-0-0-0 is duplicate SPT. As shown Table configuration 2-1-3-1-1 shows better accuracy than configuration 0-0-0-0-00 *For this work, have chosen cosine similarity as the similarity metric used within MISIM.
5

---

Under review as a conference paper at ICLR 2021
Elementwiss mean
Embedding
Elementwiss
CAS5 nodes and attributes
Embedding voctors
Kode final dtatss
Conrateturiin menn
vodovr
Figure 3: MISIM-GNN Architecture.
The CASS instance is converted into a single graph representation to serve as the input to the model. The graph is constructed by first transforming each CASS tree and its GAT entry into an individual graph. These graphs are then merged into single (disjoint) graph. For a CASS consisting a CASS tree T = (V, E') and GAT entry a, we transform it into directed graph G = (V', E', R), where V' is the set graph nodes, R is the set edge types, and E' = {(,, u, u, E V r R) is the set of graph edges. The graph is constructed as follows:
VU=V
R= (p,c),
E' = {(e, u. u) E E) u(,, v) E E}.
The two edge types, and e, represent edges from CASS tree nodes to their child nodes and to their parent nodes, respectively.
Graph Neural Network. MISIM embeds each node u E V in the input graph G into a vector by assigning trainable vector each unique node label (with the optional prefix) and GAT attribute.
The node embeddings are then used as node initial states (h (h( by a relational graph convolutional network (R-GCN (Schlichtkrull al. 2018) specified as the following:
h ReLU
Σ Σ
wh w!'ng-)
VE VEV VE [1,
EVER W
rEN
where T. is the number of GNN layers, NY = {u u,) E E') is the set of neighbors of v that connect through an edge of type r E R, and w!!) w are weight matrices to learned.
Code Vector Generation. To obtain vector representing the entire input graph, we apply both an average pooling and max pooling on the graph nodes' final states ()) The resulting two vectors
are concatenated and fed into fully-connected layer, yielding the code vector for the input program.
2.2.2 TRAINING
We train the neural network model following the setting of metric learning (Schroff et al. 2015 Hermans al., 2017 Musgrave et 2020 (unn al. 2020),, which tries map input data vector space where, under a distance (or similarity) metric, similar data points are close together (or have large similarity scores) and dissimilar data points are far apart (or have small similarity scores). The metric we use is the cosine similarity in the code vector space. As shown in the lower half of Figure we use pair-wi wise labels train the model. Each pair of input programs are mapped two code vectors by the model, from which similarity score is computed and optimized using metric learning loss function.
RELATED WORK
Although research code similarity is still in early stages, there growing body of exploratory work (Alon et al. 2019b a Ben-Nun al.) 2018 Kim et al.) [2188 Liu et al. 2018 Luan et al. 2019). In this section, we briefly discuss four state-of-ther ofhhh--rt code similarity systems: code2vec, code2seq. Neural Code Comprehension (NCC), and the Aroma system.
code2vec and code2seq. core goal of code2vec is to learn a code embedding for representing snippets of code. A code embedding is trained through the task of semantic labeling of code snippets (i.e.. predicting the function name for a function body). code2seq is proposed as a approach to generate natural language sequence from code snippet by encoding the code into vectors and

---

Under review as a conference paper at ICLR 2021
decoding sequence from them, where the sequence can be a description of the semantics of the code (e.g., the function name). As input, both systems use abstract syntax tree (AST) paths to represent code snippet. The AST is a tree structure that represents the syntactic information of the source code (Baxter et al. 1998). code2vec uses a fixed vocabulary of paths to map each path into a trainable neural
vector, while code2seq encodes the paths with RNN. They incorporate attention-ban networks to automatically identify AST paths that are more relevant to deriving code semantics. In contrast, the MISIM system uses the CASS structure that provides abstractions over code syntax, and a neural backend that can be instantiated by various learning algorithms.
Neural Code Comprehension (NCC). NCC attempts to learn code semantics based on an intermediate representation (IR) (Lattner & Adve, 2004) of the code. NCC processes source code at the IR level in an attempt to extract additional semantic meaning. transforms into contextual flow graph (XFG) that incorporates both data- and control-flow of the code, and uses XFG to train neural network for learning a code embedding. One of the constraints of NCC is that its IR requires code compilation. MISIM does not use an IR and thus does not have a compilation requirement, which may be helpful in some settings (e.g., live programming environments).
Aroma. Aroma is code recommendation system that takes a partially-writter code snippet and recommends extensions for the snippet. The intuition behind Aroma is that programmers often write code that may have already been written. Aroma leverages code base of functions and recommends extensions in a live setting. Aroma introduces the simplified parse tree (SPT) a tree structure to represent code snippet. Unlike AST, an SPT is not language-specific and enables code similarity comparison across various programming languages. To compute the similarity score of two code snippets, Aroma extracts binary feature vectors from SPTs and calculate their dot product.
Other Types of Code Comprehension. There is also a growing body of work on code comprehension that is not directly intended for code similarity, but may provide indirect value to the space of code similarity. A large body of work has studied applying machine learning to learn from the AST for completing various tasks on code (Alon et al., 2019a] Chen et al.) 2018 Hu et al.. 2018 Li et al. 2018: Mou et al.] 2016). Odena & Sutton (2020) represent program as property signatures inferred from oututt pairs, which can be used to improve program synthesizers, amongst other things. There has also been work exploring graph representations, such the following. For bug detection and code generation, Allamanis et al. (2018b), 8b); [Brockschmidi et al. (2019) represent a program as a graph with backbone AST and additional edges representing lexical ordering and semantic relations between the nodes. Dinella et al. (2020) also use AST-backboned graph representation of programs to learn bug fixing through graph transformation. [ellendoorn et al. (2020) introduce simplified graph containing only AST leaf nodes for program repair. WWi et al. (2020) extract type dependency graphs from JavaSeript programs for probabilistic type inference. In this paper, we present new code structural representation that can be consumed by different neural architectures to infer the semantic similarity of code.
4 EXPERIMENTAL EVALUATION
Table 3: Dataset statisties.
In this section, we analyze the performance of MISIM com- pared code2vec, code2seq. NCC, and Aroma on a dataset containing more than 328,000 programs. Overall, we find that MISIM has improved performance than these systems across three metrics. Then, we compare the performance of two MISIM variants, each trained with a different CASS configuration.
Split
GC
POU-104
#Problems #Programs &Problems #Programs
Training Validation
237
223,171
64
28,137
29
36.409
16
7,193
Test
31
22,795
24
10,450
Total
282,375
45,780
Datasets. Our experiments are conducted on two datasets: the GCJ dataset (Ullah et al. 2019) and the POJ-104 dataset (Mou et al., 2016). The GCJ dataset consists of solutions programming problems in Google's Code Jam coding competition 5 We use a subset of it that only contains solutions written in C/C++, solving 297 problems. The P0J-104 dataset consists of student-written C/C++ programs solving 104 problems. For both datasets, we label two programs as similar if they are solutions to the same problem. After a filtering step, which removes unparsabbe/onn-complllee programs, we split
https::/ccod in thgoog e.com/
7

---

Under review as a conference paper at ICLR 2021
each dataset by problem into three subsets for training, validation, and testing. Detailed statistics of the dataset partitioning are shown in B
Training. Unless otherwise specified, we use the same training procedure in all experiments. The models are built and trained using PyTorch (Paszke et al. 2019). To train the models, we use the Circle loss (Sun et al., 2020), state-of-the-arr metric learning loss function that has been tested effective in various similarity learning tasks. Following the P-K sampling strategy (Hermans et al.) 2017). we construct batch of programs by first randomly sampling 16 different problems, and then randomly sampling at most 5 different solutions for each problem. The loss function takes the similarity scores of all intra-batch pairs and their pair- labels as input. Further details about the training procedure and hyperparameters are discussed in Appendix C.I
Evaluation Metries. The accuracy metrics we use for evaluation are Mean Average Precision at R (MAP@R) [Musgrave et al. 2020), Average Precision (AP) (Baeza- Yates & Ribeiro-Neto (999). and Area Under -eciisionRRealll Gain Curve (AUPRG) (Flach 2015). Since these metrics are already defined, we do not detail them here, but would refer readers to Appendix C.3
100
MARel (3)
AP e5) 30
endebree
codeberd
NYC
Arana Kaar
Aroms Cue
MISEd MISIMd-GEN GREV
MISIs MISBI--PN -PNN
MrsIt- BaP
rodelvee
code 2001
NCC- Nice-are-dinss dive
ACE'
Aroms Amas Dlo
MISMM CNN
Coo
MISM BAN
MSSM BeF
cudedver
cndeaard
NCC
Arena-Ca Arons Hot
MESM GaNN
Cos
MIDI MIDDI-PANN PANN
AIttns--a BaF
NCCario Inve
NCC-arle-
(a) MAP@R on GCL.
(b) AP on GCJ.
AUPRG on GCJ.
100
60
AP
ACPPRG
50
enderve
0
calched
NCC
Atomo Chus
MIsM- GNN
AUSM RNV
MSSM
codetve
codedree
NCC
Aremia Dar
Aroe Cca
MBBM GNN
MBSM RwW
endesree
NCC-wea- iust2 frer
NCC
Arott Aroms-Co Car
Last
MSTM chtr
MIDM--N RNY
NCC-
ARC NCC-wrin-iaee
(d) MAP@R on P04.104.
(c) AP P0J-104.
AUPRG POJ-104.
Figure 4: Summarized accuracy results on the test sets for code2vec, code2seq. NCC, and Aroma and MISIM. Bar heights are the averages of the measurements over 3 runs, and error bars are bounded by the minimum and the maximum of measured values.
Configuration Identifier. In the following sections, we refer to a configuration of CASS by its unique identifier (ID). A configuration ID is formatted A-B-C-D-E. Each of the five letters corresponds to configuration type in the second column of Table [ and will be replaced by an option number specified in the third column of the table. Configuration 0-0-0-0-0 corresponds to Aroma's SPT.
Figure 4 shows the accuracy of MISIM, code2vec, code2seq, NCC, and Aroma bars show the results of the MISIM system variants trained using the baseline CASS configuration of 0-0-0-0-0. The orange bars show the results of code2vec, code2seq. NCC, and Aroma. We observe that MISIM-GNN results in the best performance for MAP@R, yielding 1.5x to 43.4x improvements over the other systems. In some cases, MISIM-BoF achieves the best AP and AUPRG scores. summary, MISIM system has better accuracy than the other systems we compared against across all three metrics.
Results.
6 The blue
Results Analysis. We provide a brief intuition for these results. The code2vec and code2seq systems use paths in an abstract syntax tree (AST), purely syntactical code representation, as the input to its neural network. We speculate that such representation may (i) keep excessive fine-grained syntactical details while (ii) omitting structural (i.e., semantic) information. This may explain why code2vec and code2seq has smaller accuracy in our experiments. The Aroma system employs manual features derived from the simplified parse tree and computes the number of overlapping features from two
'The table illustrating the same results, hut shown numerically, can be found in Appendix C.4
8

---

To Under review as a conference paper at ICLR 2021
programs their similarity score. The selection of manual features appears to be heuristic-based based and might potentially result in a loss in semantic information. NCC tries to learn code semantics from LLVM IR, low level code representation designed for compilers. The lowering process from source code to LLVM IR may discard some semantic-relevant information such as identifier names and syntactic patterns, which is usually not utilized by compilers, but might be useful for inferring code semantics. The absence of such information from NCC's input may limit its code similarity accuracy,
Table 4: Test Accuracy of MISIM-GNN Trained on Different Subsets of the Training Set. The results are shown as the average and min/max values relative to the average over 3 runs.
Sub- Training Set
Configuration
MAP@R (%)
AP (96)
AUPRG (%)
C₁
69.78 (.0.42/+0.21)
76.39 (-1.68+1155)
99.78 (-0003++00033
Ta
C2
71.99 (-0.26/+0.455
(-1.20/+0.71)
99.83 (40.02/+0.01)
C₁
63.45 (-1.588/+199 .92)
68.58 (-2.51/42.85) 85)
99.63 (-0.06/+0.00)
TB
C2
67,40 (-1.8/++1.33)
69.86 (-3.34/+1.79)
99.65 (-0.10/=0.05)
C₁
63,53 18/433
72.47 (-0.95/+1.24)
99.70 (-0.04/+0.03)
To
C2
61,23 (22.04/+1.55)
69.83 (-1.03/+
99.65 (.0.33/0.033)
C₁
61.78 (-0.46/+0.44)
66.86 (-2.31/42.81)
99.56 (-0.06/+0.07)
C2
60.86 1.599400)))
63,86 (-3.06/43.43)
99.46 14/+0.1))
4.1 SPECIALIZED EXPERIMENTAL RESULT: CASS CONHIGURATIONS
In this subsection, we provide early anecdotal evidence indicating that no CASS configuration is invariably the best for all code snippets. In other words, the configurations may need to be chosen based on the characteristics of code that MISIM will be trained on and, eventually, used for. We conducted a series of experiments that train MISIM-GNN models with two CASS configurations on several randomly sampled sub-training sets and compared their test accuracy. The two configurations used were C1, which is the CASS baseline configuration of 0-0-0-0-0 and C22, which is a non non-baseline configuration of 2-2-3-2-1 that provides a higher abstraction over the source code than C1 (e.g.. replace global variable names with a unified string, etc.). Table 4 shows the results from four selected sub-training sets, named T A. TB. Tc, and T'D from POJ-104. It can be seen that when trained on TA A or T B. the system using configuration C2 performs better than the baseline configuration in all three accuracy metrics. However, using the training sets TC or D',, the results are inverted.
To better understand this divergence, we compared the semantic features of T n B to Tc n TD We observed that some CASS-define semantically salient features (e.g., global variables see Appendix A that C2 had been customized to extract, occurred less frequently in T TTg than in n- We speculate that, in the context of the POJ-104 dataset, when global variables are used more frequently, they are more likely to have consistent meaning across different programs. As a result, abstracting them away as C2 does for T To, D. leads to loss in semantic information salient to code similarity. Conversely, when global variables are not frequently used, there is an increased likelihood that the semantics they extract are specific to a single program's embodiment. As such, retaining their names in a CASS, may increase syntactic noise, thereby reducing model performance. Therefore, when C22 eliminates them for A. B, there is an improvement in accuracy.
To TT
CONCLUSION
In this paper, we presented MISIM, an end-to-end code similarity system. MISIM has two core novelties. The first is the contexi-aware semuntics siructure (CASS) designed specifically to lift semantic meaning from code syntax. The second is neural-based code similarity scoring algorithm for learning code similarity scoring using CASS. Our experimental evaluation showed that MISIM outperforms four other of-te- code similarity systems usually by a large factor (up to 43.4x). We also provided anecdotal evidence illustrating that there may not be one universally optimal CASS configuration. An open research question for MISIM is in how to automatically derive the proper configuration of its various components for given code corpus, specifically the CASS and neural scoring algorithms which we plan to explore in future work.

---

Under review as a conference paper at ICLR 2021
REFERENCES
Maaz Bin Safeer Ahmad, Jonathan cally Translating Image Processing Libraries to Halide. ACM Trans. Graph. 38(6), November 2019. ISSN 0730-0301. doi: 10. 10.1145/335009.33355533 3355089.3356549
Alvin Cheung, and Shoaib Kamil. Automati-
Mejbah Alam, Justin Gottschlich, Nesime Tatbul, Javier S Turek, Tim Mattson, and Abdullah Muzahid. A Zero-Positive Learning Approach for Diagnosing Software H. Wallach, H. Larochelle, A. Beygelzimer, dAlch- Buc, E. Fox, and R. Garnett (eds.), Advances in Neural Information Processing Sys- tems 32, NeurIPS 2019, pp. 11623-11633 Curran Associates, Inc., 2019. URL
Performance Regressions.
http://papersnnpe.cc/aaa/e/ zero- learniiinng fop
wow.waare--eerooransseee regressions.pe
Miltiadis Allamanis, Earl T. Barr, Premkumar Devanbu, and Charles Sutton. A Survey of Machine Learning for Big Code and Naturalness. ACM Computing Surveys, 51(4). September 2018a.
Miltiadis Allamanis, Mare Brockschmidt, and Mahmoud Khademi. Learning to Represent Programs with Graphs. In International Conference Learning Representations, 2018b. 20186 URL https:
Uri Alon, Meital Zilberstein, Omer Levy, and Eran Yahav. A General Path-Based Representation for Predicting Program Properties. In Proceedings of the 39th ACM SIGPLAN Conference on Programming Language Design and Implementation, PLDI 2018, pp. 404-419, New York, NY, USA, 2018. Association for Computing Machinery. ISBN 97814 9781450356985. 3192366.3192412. UBLL://...//000000/0001111111113
doi: 10.1145/
Uri Alon, Omer Levy, and Eran Yahav. code2seq: Generating Sequences from Structured Rep- resentations of Code. In International Conference on Learning Representations, 2019a. URL .et
Uri Alon, Meital Zilberstein, Omer Levy, and Eran Yahav. code2vec: Learning Distributed Rep- resentations of Code. Proc. ACM Program, Lang., 3(POPL):40:1140022 Jamuary 2019b, ISSN 2475-1421. doi: 10. 10.1145/3290353, URL /tt.pptppp///oo.....ccr///111111
Johannes Bader, Andrew Scott, Michael Pradel, and Satish Chandra. Getafix: Learning to Fix Bugs Automatically. Proc. ACM Program. Lang., 3(0PSSLA), October 2019, doi: 10. 1453336058.
URK/https:://6..org/111110
Ricardo A. Baeza- Yates and Berthier Ribeiro-Nee Neto. Modern Information Retrieval. Addison- A Longman Publishing Co., Inc., USA. 1999. ISBN 020139829X.
Shaon Barman, Sarah Chasins, Rastislav Bodik, and Sumit Gulwani. Ringer: Web Automation by Demonstration. SIGPLAN Not., 51(10))748-766 October 2016. ISSN 0362- 13400 doi:
00011145/022671.2988488 URL /ttp://///LL//////o////////////0
Gaurav Batra, Zach Jacobson, Siddarth Madhav, Andrea Queirolo, and Nick Santhanam. Artificial- Intelligence Hardware: New Opportunities for Semiconductor Companee, 2018. URL/https
I. D. Baxter, A. Yahin, L. Moura, M. Sant' Anna, and L.. Bier. Clone detection using abstract syntax trees. In Proceedings. International Conference on Software Maintenance (Cat. No. 98CB36272), pp. 368-377, 1998.
Kory Becker and Justin Goutehhlich. Programmer: Autonomously Creating Software Programs Using Genetic Algorithms. CoRR, abs// abs/ 2017. URL /nttp::/aaxxvv.or///aa 1709.05703
10

---

Under review as a conference paper at ICLR 2021
Tal Ben-Nun, Alice Shoshana Jakobovits, and Torsten Hoefler. Neural Code Comprehension: A Learnable Representation of Code Semantics. In S. Bengio, H. Wallach, H. Larochelle, K. Grauman, and R. Gamett (eds.), Advances in Neural Information Processing Systems 31, pp. 3585-3597. Curran Associates, Inc., 2018.
N.
Sahil Bhatia, Pushmeet Kohli, and Rishabh Singh. Neuro-Symbolic Program Corrector for In- troductory Programming Assignments. In Proceedings of the 40th International Conference on Software Engineering. ICSE '1, pp. 60-70, New York. NY, USA, 2018. Association for Computing Machinery. ISBN 9781 7781450356381. doi: 10.1 14.11 00445/3180555 15553311111151111120010 80219. URL.
htttp::/ddo..rrg/
45/ 31 BO 55.3 80219
Paul Bogdan, Fan Chen, Aryan Deshwal, Janardhan Rao Doppa, Biresh Kumar Joardar, Hai (Helen) Li, Shahin Nazarian, Linghao Song, and Yao Xiao. Taming Extreme Heterogeneity via Machine Learning Based Design of Autonomous Manycore Systems. In Proceedings of the International Conference on Hardware/Software Codesign and System Synthesis Companion, CODES/ISSS '19, New York, NY, USA, 2019. Association for Computing Machinery. ISBN 9781450369237, doi: 10. 7..144//344957.33357337
Marc Brockschmidt, Miltiadis Allamanis, Alexander I.. Gaunt, and Oleksandr Polozov. Generative Code Modeling with Graphs. In International Conference on Learning Representations, 2019.
URL
Xinyun Chen, Chang Liu, and Dawn Song. Tree-to-tree Neural Networks for Program Translation. In S. Bengio, H. Wallach, H. Larochelle, K. Grauman, N. Cesa-Bianchi, and R. Garnett (eds.), Advances in Neural Information Processing Systems 31, pp. 2547-2557. Curran Associates, Inc., 2018. URL /tt/tttt///paaeaaa.../////////// http://ppaarretttttttttttteeeeee networks for for--ooorreeeeee
program wtaaaa--ttasaaaaa...d...
Yiran Chen, Yuan Xie, Linghao Song. Fan Chen, and Tianqi Tang. A Survey of Accelerator Architectures for Deep Neural Networks. Engineering, 6(3): 274, 2020. ISSN 2095-8099.
2020.01.077 URL http:t/://// http::::wwwwwwcceeeee.eeee.. doi: scceence/arrrcccee///////3333333
Kyunghyun Cho, Bart van Merriënboer, Caglar Gulcehre, Dzmitry Bahdanau. Fethi Bougares, Holger Schwenk, and Yoshua Bengio. Learning Phrase Representations using RNN Emcoder-Decodee for Statistical Machine Translation. In Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP), pp. 1724-1734, Doha, Qatar, October 2014, Association for Computational Linguistics. doi: I0D1115////144-177, URL wwtpt:ttt////cm...t.. www.wwwwwwe..rr.go
Wathology/DD4--1
Thomas H. Cormen, Charles E. Leiserson, Ronald L. Rivest, and Clifford Stein. Introduction to Algorithms, Third Edition. The MIT Press, 3rd edition, 2009, ISBN 0262033844.
V. Cosentino, J. L. Cánovas Izquierdo, and J. Cabot. A Systematic Mapping Study of Software Development With GitHub. IEEE Access, 5:7173-7192, 2017. ISSN 2169-3536. doi: 10.1109/ ACCESS. 2017..2227...22222222
S. Deng, H. Zhao, W. Fang. J. Yin, S. Dustdar, and A. Y. Zomaya. Edge Intelligence: The Confluence of Edge Computing and Artificial Intelligence. IEEE Internet of Things Journal, 77),745577749, 2020.
Elizabeth Dinella, Hanjun Dai, Ziyang Li, Mayur Naik, Le Song, and Ke Wang. Hoppity: Learning Graph Transformations to Detect and Fix Bugs in Programs. In International Conference on Learn- ing Representations, 2020. URL https: http:/::///peeeeee..eeeeee http://ommmmmmmsssssasae S
forum?
D. Feitelson, A. Mizrahi, N. Noy, A. Ben Shabat, O. Eliyahu, and R. Sheffer. How Developers Choose Names. IEEE Transactions on Software Engineering, pp. 1-1, 2020. ISSN 2326-3881. doi: 10. 109/TSF.2000. 2976920.
Peter A. Flach and Meelis Kull. Precision- recall-reeelaaeeee- gain curves: Pr analysis done right. Proceedings of the 28th International Conference on Neural Information Processing Systems Volume 1, NIPS' 15, pp. 838 846, Cambridge, MA, USA, 2015. MIT Press.
11

---

Under review as a conference paper at ICLR 2021
David Flanagan. JavaScript: The Definitive Guide. "O'Reilly Media, Inc.", 2006.
Edward M Gellenbeck and Curtis R Cook. An Investigation of Procedure and Variable Names as Beacons During Program Comprehension. In Empirical studies of programmers: Fourth workshop, pp. 65-81. Ablex Publishing, Norwood, NJ, 1991.
Justin Gottschlich, Armando Solar-Lezama, Nesime Tatbul, Michael Carbin, Martin Rinard, Regina Barzilay, Saman Amarasinghe, Joshua B. Tenenbaum, and Tim Mattson. The Three Pillars of Machine Programming.
Proceedings of the 2nd ACM SIGPLAN International Workshop on Machine Learning and Programming Languages, MAPL 2018, pp. 69-80, New York, NY, USA, 2018, ACM. ISBN 978-1-4503-5584-77 doi: 10.1145/3211346.32115. URL http: //doi.acm.org htt.://m.rrr/11.1111111111 02044//2211144..3
Eileen Hannigan, Ondrej Burkacky, Peter Kenevan, Abhijit Mahindroo, Richard John- son, Jonathon Rivait, Heather Byer, Venetia Simcock, Elizabeth Brown, Roger Draper, Gwyn Herbein, Pamela Norton, Katya Petriwsky, Charmaine Rice, John C. Sanchez, Dana Sand. Sneha Vats, Pooja Yadav, Belinda Yu. Lucia Rahilly, Michael T. Borruso, Bill Javetski, Mark Staples, and Leff Communications. McKinsey on Semiconductors. 2019. URL
http://wwwww...eeoeeeoooo/////ee Mcc i Inndyst/nndurrrr es/ wtt.uutteeemcccccoooooooooooo
Semi Saa.a//ccccccctccoooooooo/o/oann Saspa//c...cc..aaoooooooooooore2
Vincent J. Hellendoorn, Charles Sutton, Rishabh Singh, Petros Maniatis, and David Bieber. Global Relational Models of Source Code. In International Conference on Learning Representations, 2020. URL httpt:://p//peeete
wwwpt//wwwwwetettt........ forum? id= 1nbRNtwr
Alexander Hermans, Lucas Beyer, and Bastian Leibe. In Defense of the Triplet Loss for Person Re- Identification. CoRR. abs/ 703 2017. URL
Xing Hu, Ge Li, Xin Xia, David Lo, and Zhi Jin. Deep Code Comment Generation. In Proceedings of the 26th Conference on Program Comprehension, ICPC '18, pp. 200-210, New York, NY, USA, 2018. Association for Computing Machinery. ISBN 9781450357142. doi: 10.1 3196321.31999 3196334. URL
Roshni G. Iyer, Yizhou Sun, Wei Wang, and Justin Gottschlich. Software language comprehension using a program-derived semantic graph, 2020.
Shoaib Kamil, Alvin Cheung, Shachar Itzhaky, and Armando Solar-Lezama. Verified Lifting of Stencil Computations. In Proceedings of the 37th ACM SIGPLAN Conference on Programming Language Design and Implementation, PLDI 16. pp. 711-726, New York, NY, USA, 2016. ACM. doi: 10. 10.1145/29080800299111
ISBN 978- 1-881144000 470---2500-
URL http: //op:://do.... cm.org/
Kisub Kim, Dongsun Kim, Tegawendé F. Bissyandé, Eunjong Choi, Li Li, Jacques Klein, and Yves Le Traon. FaCoY: A Code-to-Code Search Engine. In Proceedings of the 40th International Conference on Software Engineering, ICSE '18, pp. 946-957, New York, NY, USA, 2018. Associ- ation for Computing Machinery. ISBN 978 450356681. doi: 10. 111453380155.31800
URL
Chris Lattner and Vikram Adve. LLVM: A Compilation Framework for Lifelong Program Analysis & Transformation. In Proceedings of the International Symposium on Code Generation and Optimization: Feedback-Diren and Runtime Optimization, CGO '04, pp. 75, USA, 2004. IEEE Computer Society. ISBN 0769521029.
Jian Li, Yue Wang, Michael R. Lyu, and Irwin King. Code Completion with Neural Attention and Pointer Networks. In Proceedings of the 27th International Joint Conference on Artificial Intelligence, IJCAP18, 18, pp. 4159-25. AAAI Press, 2018. ISBN 9780999241127.
Liuqing Li, He Feng, Wenjie Zhuang, N. Meng, and B. Ryder. Celearner: A deep learning-basse based clone detection approach. 2017 IEEE International Conference on Software Maintenance and Evolution (ICSME), pp. 249-260, 2017.
12

---

Under review as a conference paper at ICLR 2021
Bingchang Liu, Wei Huo, Chao Zhang, Wenchao Li, Feng Li, Aihua Piao, and Wei Zou. oDfff: Cross-
Cross-Version Version Binary Code Similarity Detection with DNN. Proceedings of the 33rd ACM/IEEE International Conference York, NY, USA, 2018. Association for Computing Machinery. ISBN 470355355.. doi:
Automated Software Engineering, ASE 2018. pp. 667-678, New
10. 00.1145/3238177322338
Tie- Yan Liu. Learning to Rank for Information Retrieval. Found. Trends Inf Retr. 3(3):225-333 March 2009. ISSN 1554-0669.
doi: 10.156//1500000066 URL/tttps::/ooioorg///000
1500000016
Ilya Loshchilov and Frank Hutter. Decoupled Weight Decay Regularization, In International Con- ference Learning Representations, 2019. [tL//tttss///genneeeieeeeeeeee/e
Bkg6RiCqY7
Sifei Luan, Di Yang. Celeste Barnaby, Koushik Sen, and Satish Chandra. Aroma: Code Recommenda- tion via Structural Code Search. Proc. ACM Program. Lang., 3000PSSLA;:152211522222 October 2019. ISSN 2475-1421. doi: 10. 10.1145/3360578. URL http:tpp///ddi....or///.........
3360578
Lili Mou, Ge Li, Lu Zhang, Tao Wang, and Zhi Jin. Convolutional Neural Networks over Tree Structures for Programming Language Processing. In Proceedings of the Thirtieth AAAI Conference on Artificial Intelligence, AAAI '1,, pp. 1287-1293, AAAI Press, 2016.
Kevin Musgrave, Serge Belongie, and Ser-Nam Lim. A Metric Learning Reality Check, 2020.
Augustus Odena and Charles Sutton. Learning to Represent Programs with Property Signa- tures. In International Conference on Learning Representations, 2020. URL https://
Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, Alban Desmaison, Andreas Kopf, Edward Yang. Zachary DeVito, Martin Raison, Alykhan Tejani, Sasank Chilamkurthy, Benoit Steiner, Lu Fang, Junjie Bai, and Soumith Chintala. PyTorch: An Imperative Style, High-Performance Deep Learning Library. In H. Wallach, H. Larochelle, A. Beygelzimer, F. d' A'Alhhé-Bur Buch E. Fox, and
R. Garnett (eds.), Advances in Neural Information Processing Systems 32, pp. 8024 8035. Curran Associates, /ttp:////////////oootttttaaaaaaa
Michael Pradel and Koushik Sen. DeepBugs: A Learning Approach to Name-Based Bug Detection.
Proc. ACM Program. Lang., 2(00PSLA), October 2018. doi: 10. 15543427/777.7. URL https:
//dol.ogg/0.1145/22777
Michael Sejr Schlichtkrull, Thomas N. Kipf, Peter Bloem, Rianne van den Berg, Ivan Titov, and Max Welling. Modeling relational data with graph convolutional networks. Aldo Gangemi, Roberto Navigli, Maria-Esther Vidal, Pascal Hitzler, Raphaël Troncy, Laura Hollink, Anna Tordai, and Mehwish Alam (eds.), The Semantic Web 15th International Conference, ESWC 2018, Heraklion, Crete, Greece, June 3-7, 2018, Proceedings, volume 10843 of Lecture Notes in Computer Science, pp. 593-607. Springer, 2018 doi: 10. 111110077778333399934447444 10.1007/970 1020079978-35 319- 93417-4 38
URL htttps://dii.og:
Florian Schroff, Dmitry Kalenichenko, and James Philbin. FaceNet: A Unified Embedding for Face Recognition and Clustering. The IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Jun 2015.
Yifan Sun, Changmao Cheng, Yuhan Zhang, Chi Zhang, Liang Zheng, Zhongdao Wang, and Yichen Wei. Circle Loss: A Unified Perspective of Pair Similarity Optimization. In The IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2020.
Michele Tufano, Cody Watson, Gabriele Bavota, Massimiliano Di Penta, Martin White, and Denys Poshyvanyk. Deep Learning Similarities from Different Representations of Source Code. In Proceedings of the 15th International Conference on Mining Software Repositories, MSR '1, pp. 542-54-255 553, New York, NY, USA, 2018. Association for Computing Machinery. doi: 111114//3196398.119668 URL httptttp: htttppp///oo.org/1001115511
ISBN 4581450357
31 39..399..1433
13

---

Under review as a conference paper at ICLR 2021
F. Ullah, H. Nacem, S. Jabbar, S. Khalid, M. A. Latif, F. Al-turjman, and L. Mostarda. Cyber Security Threats Detection in Internet of Things Using Deep Learning Approach. IEEE Access, 194379-124389 2019.
Guido Van Rossum and Fred L. Drake. Python Reference Manual. CreateSpace, Scotts Valley, CA. 2009. ISBN 1441412697.
Huihui Wei and Ming Li. Supervised deep features for software functional clone detection by exploiting lexical and syntactical information in source code. In Proceedings of the Twenty-Sixth International Joint Conference Artificial Intelligence, IJCAI-17, pp. 30400 2017. doi: 10.24966/iccai.20177743 /t//
Jiayi Wei, Maruth Goyal, Greg Durrett, and Isil Dillig. LambdaNet: Probabilistic Type Inference using Graph Neural Networks. In International Conference Learning Representations, 2020.
httpttppss//pganeeeeeeeeeeete/oe www.eoooornniiioooooiiiieiiie-ee HwdddHHxxhea 6kxch
W. Wulf and Mary Shaw. Global Variable Considered Harmful. Not. 80)::0334222222
1973. ISSN 0362- 1340. doi: 10. 10.114559533333 URL httpttts///0...r/1100..111
953355.9568555
Jian Zhang, Xu Wang, Hongyu Zhang, Hailong Sun, Kaixuan Wang, and Xudong Liu. A Novel Neural Source Code Representation Based on Abstract Syntax Tree. Proceedings of the 41st International Conference Software Engineering, ICSE 19, pp. 783-794. IEEE Press, 2019. doi: 20 1.00086
10. 1.11009/SE..200 2019,00086.......
Gang Zhao and Jeff Huang. DeepSim: Deep Learning Code Functional Similarity. In Proceedings of the 2018 26th ACM Joint Meeting on European Software Engineering Conference and Symposium on the Foundations of Software Engineering. ESEC/FSE 2018, 8, pp. 141-151, New York, NY. USA. 2018. Association for Computing Machinery. ISBN 978 4781550355735. doi: 10.1145/ 3236024.3236068.
Jie Zhou, Ganqu Cui, Zhengyan Zhang, Cheng Yang, Zhiyuan Liu, and Maosong Sun. Graph Neural Networks: A Review of Methods and Applications. CoRR, abs/18 181 1..08334, 2018. URL
14

---

Under review as a conference paper at ICLR 2021
A CONTEXT-AWALE SEMANTICS STRUCTURE DETAILS
The following is the formal definition of CASS.
Definition (Context-aware semantic structure (CASS)) A CASS consists of one or more CASS trees and an optional global attributes table (GAT). A CASS tree, T. is collection of nodes, denoted as T = (V, E). Each edge is from a parent node, Up 10 a child node, Vo or eg = (Vp, Ve) where ex E F. and Up, Ue E V. The root node, Ur, of the tree signiftes the beginning of the code snippet and has no parent node, i.e., pups (Up, v2) E E. A child node is either an internal node or a leaf node. An internal node has at least one child node while a leaf node has no child nodes. A CASS tree can be empty, in which has no nodes. The CASS GAT contains exactly one entry per unique function definition in the code snippet. A GAT entry includes the input and output cardinality values for the corresponding function.
V = =1v,, -2,,.,,.... ,
and edges, E {ei, E21,, fIE)
directed from
Definition (Node labels) Every CASS node has an associated label, by During the construction of CASS tree, the program tokens at each node, by are mapped to its corresponding label or l,, = (1,).. This is depicted with an expression grammar for node labels and the function mapping tokens to labels below
(bin-op)
11-19171
(unary-op)
= '++'''- ----
nleea-node-label)
LITERALI IDENT 'EVAR' 'HHVAR' ''EEXFUNC' I '#LIT' I...
(exp)
"s" I's (bin-op) 's' (unary-op) "$"1...
Linteral-nodeellbbe
== "for' '(' (exp) ';" (exp) ';' (exp) ')' (exp)';" ';"
'int' (exp) ";'
'return' (exp) ';' (exp)
(leaf-node-label) (intternl--odd--lae
if u is a leaf node otherwise
4. = f(L)
Definition 3 (Node prefix label) A node prefix label is string prefixed to node label. A node prefix label may or may not be present.
A.1 DISCUSSION
We believe there is no silver bullet solution for code similarity for all programs and programming languages. Based on this belief, key intuition of CASS's design is to provide structure that semantically rich based on structure, with inspiration from Aroma's ss SPT, while simultaneously providing range of customizable parameters to accommodate wide variety of scenarios. CASS's ainguage-gnnssin and language-specific configurations and their associated options serve for explo- ration of a series of tree variants, each differing in their granularity of detail of abstractions.
For instance, the compound statements configuration provides three levels of abstraction. Option is Aroma's 's baseline configuration and is the finest level of abstraction, as it featurizes the number of constituents in a compound statement node. Option 2 reduces compound statements to a single token and represents slightly higher level of abstraction. Option I eliminates all features related to compound statements and is the coarsest level of abstraction. The same trend applies to the global variables and global functions configurations. It our belief, based on early evidence, that the appropriate level of abstraction in CASS is likely based on many factors such as (i) code similarity purpose, (ii) programming language expressiveness, and (iii) application domain.
Aroma's 's original SPT seems to work well for a common code base where global variables have consistent semantics and global functions are standard API calls also with consistent semantics (e.g.. a single code-base). However, for cases outside of such spaces, some question about applicability
Note: the expression grammar provide is non-exhaustive due space limitations. The complete set of standard C/C++ tokens binary and unary operators is collectively denoted in shorthand as '..!.
15

---

Under review as a conference paper at ICLR 2021
arise. For example, assumptions about consistent semantics for global variables and functions may not hold cases of non-commn code-bases or non -on-standardize global function names (Wulf & Shaw 1973 Gellenbeck & Cook 1991 Feitelson et al. 2020). Having the capability to differentiate between these cases, and others, is key motivation for CASS.
We do not believe that CASS's current structure is exhaustive. With this in mind, have designed CASS to be extensible, enabling a seamless mechanism to add new configurations and options. Our intention with this paper is to present initial findings in exploring CASS's structure. Based on our early experimental analysis, presented in Section CASS seems to be a promising research direction for code similarity.
An Important Weakness. While CAST provides added flexibility over SPT, such flexibility may be misused. With CAST. system developers are free to add or remove as much syntactic differ- entiation detail they choose for given language given code body. Such overspecification (or underspecificationn, may result syntactic overload (or underload) which may cause reduced code similarity accuracy over the original SPT design, illustrate in Section C.5
B MODELS
In this section, describe the models evaluated in experiments other than MISIM-GNN, and discuss the details of the experimental procedure.
B.1 MODEL:: MISIM-BOP
The MISIM-BoF model takes a bag manual features extracted from a CASS as its input. The fea- tures include the ones extracted from CASS trees, using the same procedure described in Aroma (Luan al., (209)), as well as the entries CASS GATs. As shown in Fi the output code vector computed by taking the elementwise mean of the feature embeddings and projecting it into the code vector space with fully connected layer.
Elementwise mean
Fully connected
Embedding
features
embedding vectors
Figure MISIM BoF Architecture.
B.2 MODEL: MISIM-RNN
The input the MISIM-RNY model is a serialized representation of CASS. Each CASS tree, representing function in the program, is converted to sequence using the technique proposed in (Hu) al. 2018). The GAT entry associated with a CASS tree both prepended and appended to the tree's sequence, forming the sequence of the corresponding function. As illustrated in Figure 6 each function's sequence first has its tokens embedded, and then gets summarized a function-leve vector by a bidirectional GRU layer (Cho et al., 2014)). The code vector for the entire program is subsequently computed by taking the mean and max pooling of the function-level vectors, concatenating these two vectors, and passing the resulting vector through fully connected layer.
16

---

Under review as a conference paper at ICLR 2021
Drbeking
Function
Denalied CAS5
Teen embedding
Bdreston ORU
mean
connected
Elementwise max
Inbbdding
Function
Concatenation
Code vector
urtndding
las www.erddamnllo Vidtes
Figure 6: MISIM-RNN Architecture.
C EXPERIMENTAL DETAILS
C.I TRAINING PROCEDURE AND HYPERPARAMETERS
We use the AdamW optimizer with learning rate of 10 Loshchilov & Hutter 2019). The training runs for 100 epochs, each containing 1,000 iterations, and the model that gives the best validation accuracy is used for testing The hyperparameters used for the Circle loss ? 80 and For all of our MISIM models, we use 128-dimensional embedding vectors, hidden states, and code vectors. We also apply dropout with a probability of 0.5 to the embedding vectors. To handle rare or unknown tokens, a token that appears less than 5 times in the training set is replaced with a special UNKNOWN token.
0.4.
C.2 MODIFICATIONS TO CODE2VEC, CODE2SEQ, NCC, AND AROMA
To compare with code2vec, code2seq, NCC, and Aroma, adapt them to our experimental setting in the following ways. The original code2vec takes a function as an input, extracts its AST paths to form the input to its neural network, and trains the network using the function name prediction task. In our experiments, we feed the AST paths from all function(s)
program into the neural network and train it using the metric learning task described in Section 2.2.2 We make similar adaptions code2seq by combining AST paths from the whole program as one input sample. Additionally, we replace the sequence decoder of code2vec with an baaetioonbbaedd path aggregator used in code2vec. NCC contains pre- pre-training phase, named inst 2vec, on large code corpus for generating instruction embeddings, and subsequent phase that trains an RNN for downstream task using the pre-traue traitre embeddings. We train the downstream RNN model on our metric learning task in two ways. The first uses the pre-trained embeddings (labeled as NCC in our results). The second trains the embeddings from scratch on our task in end-to-end fashion (labeled as NCC-w/o-inst2ved For both code2vec and NCC, we use the same model architectures and smbedding/hidden sizes suggested their papers and open sourced implementations. The dimension of their output vectors (i.e., code vectors) is set to the same as our MISIM models. Aroma extracts manual features from the code and computes the similarity score of two programs by taking the dot product of their binary feature vectors. We experiment with both its original scoring mechanism (labeled: Aroma-Don and a variant that uses the cosine similarity (labeled: Aroma-Cos).
We have observed that the validation accuracy stops to increase before the 1h0 epoch all experiments.
17

---

Under review as a conference paper at ICLR 2021
C.3 EVALUATION METRICS
MAP@R R measures how accurately model can retrieve similar (or relevant) items from a database given query. MAP@R rewards a ranking system (e.g., search engine, code recommendation engine, etc.) for correctly ranking relevant items with an order where more relevant items are ranked higher than less relevant items. is defined as the mean of average precision scores, each of which is evaluated for retrieving R most similar samples given a query. In our case, the set of queries is the set of all test programs. For a program, R is the number of other programs in the same class (i.e., a POJ-104 problem). MAP@R R is applied to both validation and testing. We use AP and AUPRG to measure the performance in a binary classification setting, in which the models are viewed as binary classifiers that determine whether pair of programs are similar by comparing their similarity score with a threshold. AP and AUPRG are only used for testing. They are computed from the similarity scores of all program pairs in the test set, as well their pair-wise labels. For the systems that require training (i.e., systems with ML learned similarity scoring), we train and evaluate them three times with different random seeds.
C.4 MISIM ACCURACY RESULTS (IN TABULAR FORM)
Table shows the results of MISIM in comparison to other systems. Same results are presented in the graphical form in Figure
Table 5: Code similarity system accuracy. Results are shown as the average and min/max values, relative to the average, over 3 runs. We had to make a few modifications to adapt code2vec, code2seq. NCC and Aroma to our experimental settings. Please refer to Appendix C.2 for details.
GCI
POJ-104
Method
MAP@R (%)
AP (5)
AUPRG (%)
MAP@E (%)
AP (%))
AUPRG (%)
code2voc
7.76 (-0.79/40.88)
17.95 (-1 (-1.24141.76)
25.48 (-7,57/-10,37)
1,90 (-0-43/-0.88)
6-0.8000--00
34,97 (-20,83/-15, 15,10)
oade2seq
73)
23.09 24/2244))
(-15, 10/12.02)
(-045)40.677
(-0.37/4448))
54.97 (-7.15/47..66
NCC
17.26 (-1.11/40.57)
31.56 (-1.11/+1416)
68.76 (-1225/42646)
39.95 (-2.29/+1.61)
50.42
(-2.9%/-11))
98.86 (-0.20/-0.10)
NCC-w/o-instvve
34.88 (5.62/17.633)
56.12 (-7.63/99))
90.10 (-3 43.49)
54.19 (-3.18/
62.75 (-5. (-5.49/44422)
99.39 (-0.22/+0.17)
Aroma-Dot Dot
29.08
42.47
82,03
$2.09
45.99
98.42
Aroma-Cos
29.67
36.21
75.09
55.12
99.07
MISIM-GNY GNNN
74,90 (-L.15+0064) 92.15 (-0.97/+077
99.46 (-0.08/0.055
82.45 (-0.61/+0.4)
82.00 (-1.77/-1.66
99 36 (-0.01/-0.03)
MISIM-RNN
66 38 (-2,93442 68)
85.72
19)-69))))))
98. 78 (-0. (-011840.1)
74.01 (-2.00/4381)
81.64 (-1 52/-2,72)
99 (-0.03/-0.004)
MISIM-BoF
(-0.64/40.52)
(-0.81/+0.6))
(-0.07+0000))
(-1044+100) (-0.70/00.0)
(-0.01/+0.01)
C.5 EXPERIMENTAL RESULTS OF VARIOUS CASS CONFIGUAATINSS
In this section, we discuss our experimental setup and analyze the performance of CASS compared to Aroma's simplified parse tree (SPT). In Section C we explain the dataset grouping and enumeration for experiments. We also discuss the metrics used to quantitatively rank the different CASS configurations and those chosen for evaluation of code similarity. Section C.5.2 demonstrats that, code similarity system built using CASS (i) has greater frequency of improved accuracy for the total number of problems and (fi) is, on average, more accurate than SPT. For completeness, we also include cases where CASS configurations perform poorly.
C.5.1 EXPERIMENTAL SETUP
In this section, describe experimental setup. At the highest level, compare the performance of various configurations of CASS to Aroma's 's SPT. The list of possible CASS configurations are shown in Table
Dataset.
The experiments use the same POJ-104 dataset introduced in Section
Problem Group Selection. Given that POJ-104 consists of 104 unique problems and nearly 50,000 programs, depending on how we analyze the data, we might face intractability problems in both computational and combinatorial complexity. With this in mind, our initial approach is to construct 1000 sets of five unique, pseudo-randomly selected problems for code similarity analysis. Using this approach, we evaluate every configuration of CASS and Aroma's 's original SPT on each pair of solutions for each problem set. We then aggregate the results across all the groups to estimate their
18

---

Under review as a conference paper at ICLR 2021
overall performance. While this approach is not exhaustive of possible combinations (in set size set combinations), we aim for it to be a reasonable starting point. As our research with CASS matures, we plan to explore broader variety of set sizes and more exhaustive number of combinations.
Code Similarity Performance Evaluation. For each problem group, we exhaustively calculate code similarity scores for all unique solution pairs, including pairs constructed from the same program solution (i.e., program A compared to program A). We use G' to refer to the set of groups and g to indicate partieular group in G. We denote |G] as the number of groups in G (i.e. cardinality) and lgl as the number of solutions in group For g= Gi, where = (1,2,.... 1000), the total unique program pairs (denoted by gP) in G, ggll _ gg + 1).
To compute the similarity score of solution pair, we use Aroma's approach. This includes calculating the dot product of two feature vectors (i.e., a program pair), each of which is generated from a CASS or SPT structure. The larger the magnitude of the dot product, the greater the similarity.
We evaluate the quality of the recommendation based on average precision. Precision the ratio of true positives to the sum of true positives and false positives. Here, true positives denote solution pairs correctly classified as similar and false positives refer to solution pairs incorrectly classified as similar. Recall is the ratio of true positives to the sum of true positives and false negatives, where false negatives are solution pairs incorrectly classified as different, As we monotonically increase the threshold from the minimum value to the maximum value, precision generally increases while recall binary classifier
generally decreases. The average precision (AP) summarizes the performance under different thresholds for categorizing whether the solutions from the same equivalence class (i.e., the same POJ-104 problem) (Liu 2009). AP is calculated using the following formula over all thresholds.
1. All unique values from the M similarity scores, corresponding to the solution pairs, are gathered and sorted in descending order. Let N be the number of unique scores and
$1, $2,.... SN be the sorted list of such scores.
2. For i in 21.2...., NN, the precision ps and recall ry for the classifier with the threshold being 8; is computed.
3. Let ry = 0. The average precision is computed as:
N
AP
(ri Ti- I));
i=1
C.5.2 RESULTS
Figure 7a depicts the number of problem groups where particular CASS variant performed better (blue) or worse (orange) than SPT. For example, the CASS configuration 2-0-0-0-1 outperformed SPT 859 of 1000 problem groups, and underperformed in 141 problem groups. This equates to a 71.8% accuracy improvement of CASS over SPT. Figure shlws the two best (2-0-0-0-1 and 0-0-0-0-1), the median (2-2-3-0-0), and the two worst (1-0-1-0-4 and 1-2-1-0-0) configurations with respect certain configurations that perform better than SPT, there are also
to SPT. Although we have configurations that perform worse. We observed that the configurations with better performance have function I/O cardinality option as 1. We also observed that the configurations with worse performance have function I/O cardinality option as O. These observations indicates that function I/O cardinality seems improve code similarity accuracy, at least, for the data are considering. We speculate that these configuration results may vary based on programming language, problem domain, and other constraints.
Figure 7b the group containing the problems for which CASS achieved the best performance relative to SPT, among all 1000 problem groups. In other words, Figure 7 shows the performance of SPT and CASS for the single problem group with the greatest difference between a CASS configuration and SPT. In this single group, CASS achieves the maximum improvement of more than 30% over SPT for this problem group two of its configurations. We note that, since we tested 216 CASS configurations across 1000 different problem groups, there is a reasonable chance of observing such large difference even if CASS performed identically to SPT in expectation. We do not intend for this result to demonstrate statistical significance, but simply to illustrate the outcome of experiments.
19

---

Under review as a conference paper at ICLR 2021
Figure 7c compares the mean of AP over all 1000 problem groups. In it, the blue bars, moving left to right, depict the CASS configurations that are (i) the two best, (ii) the median, and fiii) the two worst in terms of average precision. Aroma's baseline SPT configuration is highlighted in orange. The best two CASS configurations show an average improvement of more than 1% over SPT, while the others degraded performance relative the baseline SPT configuration.
These results illustrate that certain CASS configurations can outperform the SPT on average by a small margin, and can outperform the SPT on specific problem groups by large margin. However, we also note that choosing a good CASS configuration for a domain is essential. We leave automating this configuration selection future work.
1.0
Number Problem Groups
204
SPT
0.493
2,892
Less AP than SPT
500
Average Predision
3.41
0.450
Ld5
0.6
172
0.4
0.2
1000
2-0-0-0-1
0-0-0-0-1
2-2-3-0-0
1-0-1-0-0
1-2-1-0-0
SPT
0-1-3-0-1 2-1-3-0-1 1-0-2-1-0 1-1-1-0-0 1-2-1-0-0
Configuration
Configuration
(a) Breakdown of the Number of Groups with AP Greater or Less than SPT.
(b) Average Precision for the Group Containing the Best Case.
Average Precision
0.211
0.323
0,220
0.7
0.678
0.672
SPT
2-1-3-1-1 2-1-2-1-1 1-2-2-2-0 1-2-1-0-1 1-2-1-0-0
Configuration
(c) Mean of Average Precision Over All Program Groups.
Figure 7: Comparison of CASS and SPT. The blue bars in (a) and (b), and all the bars in (c), from left to right, correspond to the best two, the median, and the worst two CASS configurations, ranked by the metric displayed in each subfigure.
C.5.3 ANALYSIS OF CONFIGURATIONS
Figures Sa Se serve to illustrate the performance variation for individual configurations. Figure Sa shows effect of varying the options for the node prefix label configuration. Applying the node prefix label for the parentheses operator (option 2) results in the best overall performance while annotating every internal node (option 1) results
concrete syntax tree and the worst overall performance. This underscores the trade-offss in incorporating syntax-binding transformations in CASS. Figure 8b we observe that removing all features relevant to compound statements (option 1) leads to the best overall performance when compared with other options. This indicates that adding separate features for compound statements obscures the code's intended semantics when the constituent statements also individually featurized.
Figure shows that removing all features relevant to global variables (option 1) degrades perfor- mance. We also observe that eliminating the global variable identifiers and assigning a label to signal their presence (option 2) performs best overall, possibly because global variables appearing in similar contexts may not use the same variable identifiers. Further, option 2 performs better than the case where global variables are indistinguishable from local variables (option 3). Figure 8d indicates that removing features relevant to identifiers of global functions, but flagging their presence with
20

---

Under review as a conference paper at ICLR 2021
special label done in option 2, generally gives the best performance. This result is consistent with the intuitions for eliminating features of function identifiers in CASS as discussed Section [A.I] Figure shows that capturing the input and output cardinality improves the average performance. This aligns with our assumption that function I/O cardinality may abstract the semantics of certain group of functions.
A Subtle Observation. A more nuanced and subtle observation is that our results seem to indicate that for each CASS configuration the optimal granularity of abstraction detail different. For compound statements the best option seems to corresponds to the coarsest level of abstraction detail, while for node prefix label, global variables, and global functions the best option seems to corresponds to one of the intermediate levels of abstraction detail. Additionally, for function I/O cardinality, the best option has finer level of detail. For our future work, we aim to perform a deeper analysis on this and hopefully learn such configurations, to reduce (or eliminate) the overhead necessary of trying to manually discover such configurations.
Nisan et Aerrage Predeom
0.22
Meen Averaee Prectsen
0.2
0.70
0.70
0.70
0.08
0.88
0.8
Confiumationn
Configuration
Configuationn
(a) Node Prefix Labels.
(b) Compound Statements.
(c) Global Variables.
Mean of Average Poeckon
8.23
Average Precióre
8.72
0.700
6.70
1,00
Configuration
Condiguation
(d) Global Functions.
(e) function cardinality.
Figure 8: The Distributions of Performance for Configurations with Fixed Option Type.
21


Reviews:
[{"review_id": "mgUrXB7EZEH", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 7, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The main contribution of this paper is the CASS structual feature for source code and its high performance. The advantages of CASS are that it doesn't require compliation and even not completement, also the fast building time of the tree and the global attributes table, which lead the MISIM system can work in a live scenario.\n\nThree kinds of deep learning models (GNN, RNN and BoF) follow the CASS representation to demonstrate its performance on two code similarity experiments. The scoring part uses a Siamese network to evaluate the different kinds of distance which is reasonable for solving this task.\n\nQuestions:\n1. How CASS preserve the sementic information? It's more like syntax to me in its current form.\n2. Could the CASS representation be extended to other software engineering jobs, such as code summarization, bug detection/repair or auto completion.\n3. The phase 1 (CASS featurization) can be replaced by other code representation methods, such as CuBERT[1]. The power of CASS should be demonstrated through necessary comparsions.\n4. Last, will the authors release their code to public?\n\n[1] Learning and Evaluating Contextual Embedding of Source Code, ICML 2020\n\n", "extra_scores": {}, "extra_text": {}}, {"review_id": "dMdnYu_VWuj", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper proposed a code similarity system called MISIM, which uses a context-aware semantic structure to represent code and calculate the similarity scores using a neural-based scoring algorithm. The context-aware semantic structure (CASS) is a structural representation of code specifically designed to lift semantic meaning from code syntax and provide an extensible representation that can be augmented. The neural-based similarity scoring part is implemented with deep neural networks, where authors tried 1). bag of features; 2). RNN, and 3). GNN. They found GNN performs the best. The experiment results show that MISIM can outperform code representing models like NCC and code2vec to detect code similarity.  \n\n\nPros:\n1.\tThe context-aware semantic structure for code representation is  interesting. The configurable code structure representation gives the model the flexibility to handle multiple tasks.  \n2.\tThis paper is well organized and easy to follow. The demonstration figures are clear and understandable. \n\nCons:\n1.\tThis paper does not give detailed guidance on what configuration should be used for various circumstances. As shown in the appendix, different configurations will have a considerable influence on the experimental results. It is difficult for other researchers to apply CASS to different scenarios until a detailed configuration explanation is provided. \n\n2.\tAnother weakness is that some important baselines are missing. This paper compares 3 different baselines: code2vec, NCC, and aroma, in which code2vec and NCC are designed for general code representation. Aroma is a code search system that locates code snippets in code corpus and makes code snippets recommendation. None of them are specially proposed for pair-wised code similarity comparison, as in this paper's experimental settings. There are many pieces of research about measuring code similarities such as CCLearner, CDLH, and DeepSim. Without comparison with those approaches, it is hard to convince others of the superiority of MISIM.\n\n\n3.\tThe author emphasizes code similarity is now a first-order problem that must be solved in the introduction part, but the lack of relevant statics and examples makes this argument weak. Besides, the claim that AST representation will mislead code similarity systems into learning too much syntax but semantics also lacks solid evidence to support. Please provide some concrete examples that modeling those \"syntax\" in AST actually HURTS the performance, i.e., state-of-the-art methods learning from ASTs (e.g., the methods listed below) performs worse than the proposed method.\n\nReferences: \n[DeepSim] Zhao, G., & Huang, J. (2018, October). Deepsim: deep learning code functional similarity. In Proceedings of the 2018 26th ACM Joint Meeting on European Software Engineering Conference and Symposium on the Foundations of Software Engineering (pp. 141-151).\n\n[CCLearner] Li, L., Feng, H., Zhuang, W., Meng, N., & Ryder, B. (2017, September). Cclearner: A deep learning-based clone detection approach. In 2017 IEEE International Conference on Software Maintenance and Evolution (ICSME) (pp. 249-260). IEEE.\n\n[CDLH] Wei, H., & Li, M. (2017, August). Supervised Deep Features for Software Functional Clone Detection by Exploiting Lexical and Syntactical Information in Source Code. In IJCAI (pp. 3034-3040).\n", "extra_scores": {}, "extra_text": {}}, {"review_id": "SEQVK4eRnY0", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "### Summary ###\n\nThe paper presents a technique for code similarity. The paper combines two ideas: (1) a new “context-aware” representation for code which is used as a basis for computing code embeddings, (2) code similarity based on cosine similarity between the embeddings.\n\n### Strengths ###\n\n* The experimental results significantly improve previous work.\n\n### Weaknesses ###\n\n* Looks like an extension of AROMA, with a slightly more general approach for feature engineering, but no significant innovation beyond that. \n\n* The code2vec baseline is old and weak (should have used code2seq). \n\n### Questions for Authors ###\n\n* You need to provide specific human customization per programming language. Can't this customization be learned itself? For example by using a Tree RNN to represent different abstractions of subtrees as needed?\n\n* CASS seems to be closely related to an AST. In fact, the mapping of several syntactic elements (folding compound statements) could be viewed as a particular design choice when abstracting a syntax tree to an abstract syntax tree (in your case reflected in \"node lables\").\n\n* The GAT includes a very shallow abstraction of function signatures. As such, I am curious about it's contribution to the final accuracy. Do you have an ablation study of the results obtained when GAT is omitted? I assume it has a rather different contribution across different languages?\n\n* code2vec is a weak baseline. You should at least consider code2seq as it uses a significantly more powerful encoder.\n\n* I wonder what would be the results using a textual sequence encoder as a baseline.\n\n\n### Improving the Paper ###\n\n* It would help to have a comparison of CASS+GAT to a simple AST-based approach. Keep your entire pipeline identical, including the linearization of the tree mentioned in B.2, and show what is the gain from the CASS per-language customization. I am sure that CASS+GAT would do better, but I conjecture that their contribution would be marginal in many mainstream languages.\n\n* The comparison with Code2Vec is not clear to me. What do you mean by \"we feed the AST paths from all function(s) in a program into the neural network and train it using the metric learning task described in Section 2.2.2.\". Is this trained separately per program? \n\n### Minor questions and comments ###\n\n* This was a strange point to emphasize in Definition 1: \"A child node is either an internal node or a leaf node. An internal node has at least one child node while a leaf node has no child nodes.\"\n\n* Page 15, \"pretended\" -> prepended\n\n\n\n\n\n\n", "extra_scores": {}, "extra_text": {}}, {"review_id": "S2kSTQ_a2-g", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper addresses the problem of code similarity detection and proposes an approach called MISIM.\n\nEssentially, this paper proposes a tree representation, called context-aware semantic structure (CASS), which is very similar to the abstract syntax tree (AST). Then, different neural architectures (e.g., GNN or GRU) process the CASS and encode the program as a vector for similarity modeling. \n\nMajor concerns:\n\n1. While the title of this paper claims \"a novel code similarity system,\" I do not feel the model novel at all, as it just follows a standard Siamese structure. \n\nThe proposed CASS is very similar to abstract syntax tree, and there has been many structure variants proposed in previous work, like contextual flow graph, simplified parse tree. CASS is yet another, and I do not feel the need of CASS is well motivated. \n\n\n2. Experimentally, the paper shows performance improvement on GCJ and POJ-104 datasets, compared with a few baseline systems, such as code2vec, NCC, and Aroma. \n\nHowever, there's no controlled experiments on the choice of different structures, such as the AST and contextual flow graph. To be convinced, I expect \n\n1) The authors show the results of controlled experiments with different structures. \n\n2) The authors show the generality of the CASS in different tasks, such as clone detection. \n\n3) The authors have a deeper analysis on CASS. What's the additional information encoded in CASS than a standard AST? \n\nMinor:\n    code2vec, NNC --> code2vec, NCC", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper studies the problem of computing a similarity measure between two pieces of code. The main contributions are a configurable alternative (CASS) to abstract syntax trees (ASTs) for representing code and a model for embedding these structures within a Siamese net-like architecture. While parts of the ICLR community that make use of ASTs would likely find interest in the options provided by CASS and the associated experiments, the contribution is mostly around feature engineering of AST-like structures for one specific application, which is quite niche. The machine learning modeling appears fairly standard. Thus in total, I don’t see enough here to recommend acceptance.
Decision: Reject