NS3: Neuro-symbolic Semantic Code Search
Authors: Shushan Arakelyan
Anna Hakhverdyan
Miltiadis Allamanis
Luis Antonio Garcia
Christophe Hauser
Xiang Ren
Venue: NeurIPS 2022 Accept (2022)
Keywords: deep learning
ml4code
semantic code search

Abstract: Semantic code search is the task of retrieving a code snippet given a textual description of its functionality. Recent work has been focused on using similarity metrics between neural embeddings of text and code. However, current language models are known to struggle with longer, compositional sentences, and multi-step reasoning. To overcome this limitation, we propose supplementing the query sentence with a layout of its semantic structure. The semantic layout is used to break down the final reasoning decision into a series of lower-level decisions. We use a Neural Module Network architecture to implement this idea. We compare our model - $NS^3$  (Neuro-Symbolic Semantic Search) - to a number of baselines, including state-of-the-art semantic code retrieval methods, such as CodeBERT, CuBERT and GraphCodeBERT, and evaluate on two datasets - Code Search Net (CSN) and Code Search and Question Answering (CoSQA). On these datasets, we demonstrate that our approach results in higher performance. We also perform additional studies to show the effectiveness of our modular design when handling compositional queries.  

: Neuro-Symbolic Semantic Code Search
Shushan Arakelyan', Anna Hakhverdyan', Miltiadis Allamanis
Luis Garcia', Christophe Hauser Xiang Ren
University of Southern California, Department of Computer Science
National Polytechnic University of Armenia
Microsoft Research Cambridge
USC Information Sciences Institute
shushanaQusc. edu, aanbhalavardyaa@@@giii.cc com
miltos@allamania com, (lgarcia, Hauser]@isi. edu, xiangrendusc. edu
Abstract
Semantic code search is the task retrieving a code snippet given textual descrip- tion of its functionality. Recent work has been focused on using similarity metrics between neural embeddings text and code. However, current language models are known to struggle with longer, compositional text, and multi-step reasoning. To overcome this limitation, we propose supplementing the query sentence with layout of its semantic structure. The semantic layout is used to break down the final reasoning decision into a series of lower-level decisions. We use a Neural Module Network architecture implement this idea. We compare our model N -Syubo-Symboic Semantic Search) to a number of baselines, including state-of-the-are semantic code retrieval methods, and evaluate NodeSearchNet and Code Search and Question Answering. We demonstrate that approach results in more precise code retrieval, and study the effectiveness of our modular design when handling compositional querie
datasets
Introduction
The increasing scale of software repositories makes retrieving relevant code snippets more challenging. Traditionally, source code retrieval has been limited keyword [3 B0 or regex (7] search. Both rely on the user providing the exact keywords appearing in or around the sought code. However, neural models enabled new approaches for retrieving code from a textual deseription of its functionality, a task known as semantic code search (SCS). A model like Transformer [36] can map database of code snippets and natural language queries to shared high-dimensional space. Relevant code snippets are then retrieved by searching over this embedding space using predefined similarity metric, or learned distance function 266 [2]. Some of the recent works capitalize on the rich structure of the code, and employ graph neural networks for the task
Despite impressive results SCS, current neural approaches far from satisfactory in dealing with a wide range of natural-language queries, especially on ones with compositional language structure. Encoding text into dense vector for retrieval purposes can be problematic because we risk loosing faithfulness of the representation, and missing important details of the query. Not only does this a) affect the performance, but can b) drastically reduce model's value for the users. because compositional queries such as "Check that directory does not exist before creating it" require performing multi multi-step reasoning on code.
"Currently at Google Research Equual supervision Code and data are available at http://twwggg..ascooo///a/aaa/aa
36th Conference on Neural Information Processing Systems (NcurIPS 2022).

---

Parsing into layout
Action module estimates masiedd
Load from
zng
Load
Entiyy das, module messures relevance w/ code
Load from
Query q'
FROM
"all tables"
[v0 y
v
E
[v]]]
v
all tablos
detaned
[c| je- Cyy
[d] p cl
[at pan
Tokenize code
"dataset"
[e] 1.
e'u
(ct al
[at)
Code
[c g c
[c Cly
Mask one arpument
Figure 2: Overview the NS3 approach. illustrate the pipeline of processing for an example query "Load all tables from dataser". Parsed query is used for deciding the positions of entity discovery and action modules in the neural module network layout. Each entity discovery module receives a noun/noun phrase as input, and outputs relatedness scores for code tokens, which are passed input to action module. Action module gets scores for all its children in the parse-tree, except one, which is masked, and the goal predicting, cloze-style, what are the relatedness scores for the missing argument.
We suggest overcoming these challenges by intro- ducing a modular workflow based on the semantic structure of the query. Our approach based on the intuition of how engineer would approach SCS task. For example, in performing search for code that navigates folders in Python they would first only pay attention to code that has cues about operating with paths, directories or folders. Afterwards, they would seek indications of iterating through some of the found objects other entities in the code related to them. In other words, they would perform multiple steps of different nature i.e. finding indications of specific types of data entities, or specific operations. Figure D [illustrates which parts of the code would be important to indicate that they have found the desired code snippet at each step. We attempt to imitate this process in this work. To formalize the decomposition of the query into such steps, we take inspiration from the idea that code is comprised of data, or entities, and transformations, or actions, over data. Thus, a SCS query is also likely to describe the code in terms of data entities and actions.
Navigate folders
3ef 1istRoldar for entry in snar Bi Acw. (
cef 1sstfolder(1): Step
entry 1
ada www.rr..aaaddiee dir
try. (K) yleld entry.
entry.is_ddrdd
yield entry.
Figure 1: Motivating Example. To match query "Navigate folders" on a code snippet, we find all references (token spans) to entity "folders" in code (e.g.. paths and directories) using various linguistic cues (Step 1). Then we look for cues in code that indicate the identified instances of "folders" are being iterated through i.c., "navigate" (Step 2).
We break down the task of matching the query into smaller tasks of matching individual data entities and actions. In particular, we aim to identify parts of the code that indicate the presence of the corresponding data or action. We tackle each part with a distinct type of network a neural module. Using the semantic parse of the query, we construct the layout of how modules' outputs should be linked according to the relationships between data entities and actions, where each data entity represents a noun, or noun phrase, and each action represents a verb, or verbal phrase. Correspondingly, this layout specifies how the modules should be combined into single neural module network (NMN) 4 Evaluating the NMN on the candidate code approximates detecting the corresponding entities and actions in the code by testing whether the neural network can deduce one missing entity from the code and the rest of the query.
This approach has the following advantages. First, semantic parse captures the compositionality of a query. Second, it mitigates the challenges of faithful encoding of text by focusind only small portion of the query at a time. Finally, applying the neural modules in a succession can potentially mimic staged reasoning necessary for SCS.
We evaluate our proposed 3 model on two SCS datasets CodeSearchNet (CSN) 24! and CoSQA/WebQuerye Test P Additionally, we experiment with a limited training set size of CSN of 10K and 5K examples. We find that NS² provides large improvements upon baselines in all cases. Our experiments demonstrate that the resulting model is more sensitive to small, but semantically significant changes in the query, and is more likely to correctly recognize that a modified query no longer matches its code pair.
Our main contributions are: (i) We propose looking at SCS as compositional task that requires multi-step reasoning. (ii) We present implementation of the aforementioned paradigm based on

---

NMNs. (ii) We demonstrate that our proposed model provides large improvement on number of well-established baseline models. (iv) We perform additional studies to evaluate the capacity of our model to handle compositional queries.
2 Background
2.1 Semantic Code Search
Semantic code search (SCS) is the process of retrieving a relevant code snippet based on a textual description of its functionality, also referred to query. Let C. be database of code snippets c².
For each c' E C, there is a textual description of its functionality q'. In the example in Figure 2 the query q' is "Load all tables from dataset". Let be indicator function such that r(q', c²
if
i i= ji and 0 otherwise. Given some query q the goal of SCS is to find c" such that r(q, ) 1. We assume that for each q" there is exactly as input a pair of query and a candidate code snippet: c") and assign the pair a probability pM for being a correct match. Following the common practice in information retrieval, we evaluate the performance of the model based on how high the correct answer c* is ranked among a number of incorrect, distractor instances {c]. This set of distractor instances can be the entire codebase C, or a subset of the codebase obtained through heuristic filtering, or another ranking method.
such c* Here we look to construct model which takes
2.2 Neural Models for Semantic Code Search
Past works handling programs and code have focused on enriching their models with incorporating more semantic and syntactic information from code [1 I
Some prior works have cast the SCS as sequence classification task, where the code is represented as a textual sequence and input pair (q', c²) is concatenated with a special separator symbol into a single sequence, and the output the score g زzو f(q'. e.). The function f performing the classification can be any sequence a
34 47
classification model, e.g. BERT
define separate networks for independently representing the query (f), the code (g) and measuring the similarity between them: أمر = (( This allows one to design the code encoding network g with additional program-specific information, such as abstract syntax trees 3 or control flow graphs 15 455. Separating two modalities of natural language and code also allows further enrichment of code representation by adding contrastive learning objectives (25 In these approaches, the original code snippet e is automatically modified with semantic-preserving transformations, such as variable renaming, introduce versions of the code snippet- c' with the exact same functionality. Code encoder g is then trained with an appropriate contrastive loss, such as Noise Contrastive Estimation (NCE) [9]. or InfoNCE 351
Alternatively,
However, there is also merit in reviewing how we represent and use the textual query to help guide the SCS process. Firstly, existing work derives single embedding for the entire query. This means that specific details or nested subqueries of the query may be omitted or not represented faithfully getting lost in the embedding. Secondly, prior approaches make the decision after single pass over the code snippet. This ignores cases where reasoning about a query requires multiple steps and thus multiple look-ups over the code, as is for example in cases with nested subqueries. Our proposed approach N S attempts to address these issues by breaking down the query into smaller phrases based on its semantic parse and locating each of them in the code snippet. This should allow us to match compositional and longer queries to code more precisely.
Limitations
3 Neural Modular Code Search
We propose to supplement the query with loose structure resembling its semantic parse, as illustrated in Figure We follow the parse structure break down the query into smaller, semantically coherent parts, so that each corresponds to an individual execution step. The steps are taken in succession by a neural module network composed from layout that is determined from the semantic parse of the
"This is not the case in CoSQA dataset. For the sake of consistency, we perform the evaluation repeatedly, leaving only correct code snippet among the candidates time, while removing the others.
3

---

query (Sec. B.I. The neural module network is composed by stacking "modules", or jointly trained networks, of distinct types, each carrying out a different functionality.
Method Overview In this work, we define two types of neural modules entity discovery module (denoted by E; Sec. ).2) and action module (denoted by A; Sec (.3). The entity discovery module estimates semantic relatedness of each code token & in the code snippet e¹ = _ ,xy] to an entity mentioned in the query - e.g. "all tables" or "datasef" as in Figure The action module estimates the likelihood of each code token to be related to an (unseen) entity affected by the action in the query e.g. "dataset" and "load from" correspondingly, conditioned on the rest of the input (seen), e.g. "all tables". The similarity of the predictions of the entity discovery and action modules measures how well the code matches that part of the query. The modules are nested the action modules are taking as input part of the output of another module- and the order of nesting is decided by the semantic parse layout. In the rest of the paper we refer the inputs of a module as its arguments.
Every input instance fed to the model is 3-tuple (q', Sgg,, c²) consisting of a natural language query q'. the query's 's semantic parse Sgt, a candidate code (sequence) c²_ The goal is producing a binary label pty = if the code is a match for the query, and 0 otherwise. The layout of the neural module network, denoted by [(x²,) ). is created from the semantic structure of the query syt. During inference, given (q' sqi, c²) ) as input the model instantiates a network based on the layout, passes q', e and Sn as inputs, and obtains the model prediction f*l This pipeline is illustrated in Figure and details about creating the layout of the neural module network are presented in Section 3.1
During training, we first perform noisy supervision pretraining for both modules. Next, perform end-to-end training, where in addition to the query, its parse, and a code snippet, the model is also provided gold output label r(q', cd) if the code is a match for the query, and r(q', c²) 0
otherwise. These labels provide signal for joint fine-tuning of both modules (Section 3.5
3.1 Module Network Layout
Here present our definition of the structural representation Syt for query q' and introduce how this structural representation is used for dynamically constructing the neural module network, i.e. building its layout L(s,)). ).
Query Parsing To infer the representation So', we pair the query (e.g.. "Load all tables from dataset", as in Figure 2. with simple semantic parse that looks similar to: DO WHAT [ te//from/in/....
WHAT, WHEN, WHERE, HOW, etc]. Following this semantic parse, we break down the query into shorter semantic phrases using the roles of different parts of speech. Nouns and noun phrases correspond to data entities in code, and verbs describe actions or transformations performed on the data entities. Thus, data and transformations are separated and handled by separate neural modules Categorial Grammar- (CCG) semantic parser 43 5 to infer the semantic parse Sy, for the natural language query q'. Parsing is described in further detail in Section 4.1 and Appendix A.2
entity discovery module E and an action module A. We use Combinatory
Specifying Network Layout In the layout L(s,e,, every noun phrase (e.g.. "dataser" in Figure 2 will be passed through the entity discovery module E. Module E then produces a probability score es for every token of in the code snippet el to indicate its semantic relatedness to the noun phrase: E("dataser"" e²) [e1, e2. e.]. Each verb in Sq¹ (e.g.. "load" in Figure 2 will be passed through (0,, 02. av]. Here, p is the span of arguments to the verb
an action module: A("load", p', (action) in query q', consisting of children of the verb in the parse 8,² (e.g. subject and object arguments to the predicate "load"): a1....,ay, ay are estimates of the token scores c for an entity from p'. The top-level of the semantic parse is always an action module. Figure also illustrates preposition FROM used with "dataset", handling which is described in Section 3.3
3.2 Entity Discovery Module
The entity discovery module receives string that references data entity. Its goal is to identify tokens in the code that have high relevance to that string. The architecture of the module is shown in Figure S Given an entity string, "dataser" the example, and a sequence of code tokens (2.... dy., entity module first obtains contextual code token representation using RoBERTa model that is initialized

---

from CodeBERT checkpoint. The resulting embedding is passed through two-layer MLP to obtain a score for every individual code token cj 0 S 6 S Thus, the total output of the module a vector of scores: [e1, 62, ...................... ... To prime the entity discovery module for measuring relevancy between code tokens and input, we fine-tune with noisy supervision, as detailed below.
Noisy Supervision We create noisy supervision for the entity discovery module by using keyword matching and a Python static code analyzer. For the keyword matching, if a code token is an exact match for one or more tokens in the input string, its supervision label set to I, otherwise it Same true if the code token is a substring or a superstring of one or more input string tokens. For some common nouns we include their synonyms (e.g. "map" for "dicf"), the full list of those and further details are presented Appendix B
Ently "datasot"
Sigmold
from query of
RaLü
1ey o2 -.. ey)
Tokanized code c_-c
Linear
Concatonate to compose input
Transformer Enc.
Scores for code takens e-- e
pass
Figure 3: Entity module architecture.
We used the static code analyzer to extract information about statically known data types. We cross- matched this information with the query discover whether the query references any datatypes found in the code snippet. If that is the case, the corresponding code tokens are assigned supervision label 1, and all the other tokens are assigned to 0. the pretraining we learned on equal numbers of (query, code) pairs from the dataset, as well as randomly mismatched pairs of queries and code snippets to avoid creating bias in the entity discovery module.
3.3 Action Module
First, we discuss the case where the action module has only entity module inputs. Figure d provides high-level illustration of the action module. In the example, for the query "Load all tables from dataset", the action module receives only part of the full query "Load all tables from ???". Action module then outputs token scores for the masked argument "dataset". If the code snippet corresponds to the query, then the action module should be able to deduce this missing part from the code and the rest of the query. For consistency, we always mask the last data entity argument. We pre-train the action module using the output scores of the entity discovery module as supervision.
Each data entity argument have multiple entities with prepositions. For that reason, for each data entity argument we create one joint embedding of the action verb and the preposition. Joint embeddings are obtained with a 2-layer MLP model, as illustrated in the left-most part of Figure
be associated with 0 or prepositions, but each action may
"Load all tables from ???"
a data entity does not have a preposition associ- ated with it, the vector corresponding to the preposi- tion is filled with zeros. The joint verb- per-prrepositoon embedding is stacked with the code token embed- ding og and entity discovery module output for that token, this is referenced in the middle part of Fig- ure This vector is passed through transformer encoder model, followed by a 2-layer MLP and a sigmoid layer output token score 0x, illustrated in the right-most part of the Figure Thus, the dimensionality of the input depends on the number of entities. We use a distinct copy of the module with the corresponding dimensionality for different numbers of inputs, from I to 3.
Load
(Linaa
Load
Siamali
RRLLU
Fial moles, d.)
Linsor
[Linse
Load from)
ReeLU
a ag
Joint repe.
Concatanata
tolion emb.
Transformst
Linear
for code token
Score
prapositiion
campose inputt
Forward pass
Figure 4: Action module architecture.
3.4 Model Prediction
The final score أقم = f(q" e²) is computed based on the similarity of action and entity discovery module output scores. Formally, for an action module with verb x and parameters pd ipf... .pf.....p2 the final model prediction is the dot product of respective outputs of action and entity discovery modules: رعر A(r, pf,.. .k ElfE). Since the action module estimates token scores for the entity affected by the verb, if its prediction is far from the truth then either the action is not found in the code, or it not fully corresponding to the query, for example, in the code snippet tables are loaded from web, instead of a dataset. We normalize this score to make it a probability. If this is the
5

---

only action in the query, this probability score will be the output of the entire model for (q', e²) pair: رo otherwise pe will be the product of probability scores of all nested actions in the layout.
Compositional query with nested actions Consider a compositional query "Load all tables from dataset using Lib library". Here action with verb "Load from" has an additional argument "using" also an action with an entity argument "Lib library".. In case of nested actions, we flatten the layout by taking the conjunction of individual action similarity scores. Formally, for two verbs x and y Pp in a layout that looks like: A(x. p", A(y: p))). the output of the model is the conjunction of similarity scores computed E(p() )). This process is repeated until all remaining p* and p are data entities. This design ensures that code snippet is ranked highly if both actions are ranked highly, we leave explorations of alternative handling approaches for nested actions future work.
and their corresponding arguments p" = Ipf...... -P and p" =
for individual action modules; sim(A(r, p` ................................ Pk 1),, E(fE)) sim(A(y, ........P................ pi-1) 1).
3.5 Module Pretraining and Joint Fine-tuning
We train our model through supervised pre-training, as is discussed in Sections 3.2 and 3.3 followed by end-to-end training. End-to-end training objective is binary classification given pair of query q' and code c², the model predicts probability ft that they are related. In the end-to-end training. we use positive examples taken directly from the dataset- (q', c²), as well as negative examples composed through the combination of randomly mismatched queries and code snippets. The goal of end-to-end training is fine-tuning parameters of entity discovery and action modules, including the weights of the RoBERTA models used for code token representation.
Batching is hard to achieve for our model, so for the interest of time efficiency we do not perform inference on all distractor code snippets in the code dataset. Instead, for given query we re-rank top-K highest ranked code snippets as outputted by some baseline model, in our evaluations used CodeBERT. Essentially, we use our model in a re-ranking setup, this is common in information retrieval and is known 1.2 ranking. We interpret the probabilities outputted by the model ranking scores. More details about this procedure are provided in Section 4.1
Experiments
4.1 Experiment Setting
Dataset We conduct experiments on two datasets: Python portion of the CodeSearchNet (CSN) 244. and CoSQA [23. We parse all queries with the CCG parser, as discussed later in this section, excluding unparsable examples from further experiments. This leaves us with approximately 40% of the CSN dataset and 70% of the CoSQA dataset, the exact data statistics are available in Appendix A in Table 3 We believe, that the difference in success rate of the parser between the two datasets can be attributed to the fact that CSN dataset, unlike CoSQA, does not contain real code search queries, but rather consists of docstrings, which are used as approximate queries. More details and examples can be found in Appendix A.3 For our baselines, we use the parsed portion of the dataset for fine-tuning to make the comparison fair. addition, we also experiment with fine-tuning all models even smaller subset of CodeSearchNet dataset, using only 5K and 10K examples for fine-tuning. The goal is testing whether modular design makes NS3 more samplc-efficient.
All experiment and ablation results discussed in Sections 4.2 and are obtained on the test set of CSN for models trained on CSN training data, or WebQuery Test 31 small natural language web query dataset of document-code pairs- for models trained on CoSQA dataset.
Evaluation and Metrics We follow CodeSearchNet's original approach for evaluation for test instance (g, c), comparing the output against outputs over a fixed set of 999 distractor code snippets. We use two evaluation metries: Mean Reciprocal Rank (MRR) and Precision @K (P@ K) for K=1, 3, and 5, see Appendix I for definitions and further details.
Following a common approach in information retrieval, we perform two-step evaluation. In the first step, we obtain CodeBERT's output against 999 distractors. In the second step, we use N to re-rank the top 10 predictions of CodeBERT. This way the evaluation is much faster, since unlike our

---

CSN
CSN-IOK
CSN-5K
Method
MRR
P@1 1
P@3
P@5
MRR
P@1
P@3
P@5
MRR
P@1
P@3
P@5
BM25
0.209
0.144
0.230
0.273
0.209
0.144
0.230
0.273
0.209
0.144
0.230
0.273
RoBERTa (code)
0.842
0.768
0.905
0,933
0.461
0.296
0.545
0.664
0.290
0.146
0.324
0.438
CUBERT
0.225
0.168
0.253
0.294
0.144
0.081
0.166
0.214
0.081
0,030
0,078
0.118
CodeBERT
0.873
0,803
0.939
0.958
0.69
0.55
0.799
0.873
0.680
0.535
0.799
0.870
GraphCodcBERT
0.812
0.725
0.880
0.919
0.786
0.684
0.859
0.901
0.773
0.677
0.852
0.892
GraphhCooBBRRT
0.883
0.820
0.941
0.962
0.780
0.683
0.858
0.904
0.765
0.662
0.846
0.894
NS
0.924
0.884
0.959
0.969
0.826
0.753
0.886
0.908
0.823
0.751
0.881
0.913
Upper-bound
0.979
0.939
0.936
Table 1:
Mean Reciprocal Rank (MRR) and Precision @ 1/@3/@5 (higher is better) for methods trained on different subsets from CodeSearchNet dataset.
modular approach, CodeBERT can be fed examples in batches. And as we will see from the results, we see improvement in final performance in all scenarios.
Compared Methods We compare Ñ with various state-of aart methods, including some
traditional approaches for document retrieval and pretrained large NLP language models. (1) BM25 is a ranking method to estimate the relevance of documents to given query. (2) RoBERTa (code) is
a variant of RoBERTa [99] pretrained on the CodeSearchNet corpus. (3) CuBERT 26 is a BERT
Large model pretrained on 7.4M Python files from GitHub. (4) CodeBERT [ is an encoder-only Transformer model trained on unlabeled source code via masked language modeling (MLM) and replaced token detection objectives. (5) GraphCode
I is a pretrained Transformer model using MLM, data flow edge prediction, and variable alignment between code and the data flow. (6) GraphCodeBELe GraphCodeBERT to re-rank the top-10 predictions of the CodeBERT model.
is re-ranking baseline. We used the same setup as for NS3, but used
Query Parser We started by building vocabulary of predicates for common action verbs and entity nouns, such as "convert", "find". "dict", "map". etc. For those we constructed the lexicon (rules) of the parser. We have also included "catch-all" rules, for parsing sentences with less-common words. To in- crease the ratio of the parsed data, we preprocessed the queries by removing preceding question words, punctuation marks, etc. Full implementation of our parser including the entire lexicon and vocabulary can be found at https://aat //nn::////nnmmouuu.mm 4open. science/ T/ccg__parsee parser- 4BC6, More details are available in Appendix A.2
CoSQA
Method
MRR
P@1
P@3
P@5
BM25
0.103
0.05
0.119
0.142
RoBERTa (code)
0.279
0.159
0.343
0.434
CuBERT
0.127
0.067
0.136
0.187
CodeBERT GraphCodeBERT
0.345
0.175
0.42
0.54
0.435
0.257
0.538
0.628
GraphhodeBBRR
0.462
0.314
0.547
0.632
N
0.551
0.445
0.619
0.668
Upper-bound
0.736
0.724
0.724
0.724
Table 2: Mean Reciprocal Rank(MRR) and Pre- cision@ l/@ e/@@ (higher is better) for different methods trained on CoSQA dataset.
Pretrained Models ules each embed code tokens with RoBERTa model, that has been initialized from checkpoint of pretrained CodeBERT model during final end-to-end training phase.
Action and entity discovery mod-
[ We fine-tune these models during the pretraining phases, as well as
Hyperparameters The MLPs in entity discovery and action modules have 2 layers with input dimension of 768. We use dropout in these networks with rate 0.1. The learning rate for pretraining and end-to-end training phases was chosen from the range of 1e-6 to 6e-5. We use early stopping with evaluation on unseen validation set for model selection during action module pretraining and end- to-end training. For entity discovery model selection we performed manual inspection of produced scores on unseen examples. For fine-tuning the CuBERT, CodeBERT and GraphCodeBERRT baselines we use the hyperparamcters reported in their original papers. For RoBERTa (code), we perform the search for learning rate during fine-tuning stage in the same interval as for our model. For model selection on baselines we also use early stopping.
7

---

0.50
6mphhodeBBRRT
0.85
Proctsiens
0.4
0.5
8.80
0.2
0.75
0.7
Max depth of quary
Calefarctter Number of arguments for Action
CaRDA
(a)
(b)
Figure 5: We report Precision@ @ scores. (a) Performance our proposed method and baselines broken down by average number of arguments per action in a single query. (b) Performance of our proposed method and basclines broken down by number of arguments in queries with single action.
Effect of training
Scgge normaliation
Similarity measure
CoAQA
8.63
CaliA
6.8
0.55
0.4
0.37
0.44
0.43
0.4
0.46
0.43
0.2
0.0
Product
L2 Precesonge1
Weighted Cesine
Precision@1
(a) Effect of training
(b) Score normalization
(c) Similarity measure
dataset with different ablation variants. (a) Skipping one, or both pretraining procedures, and only training end-to-end. (b) Using no normalization on output scores (None), oellyn-only entity discovery-only, and both. (c) Performance with different options for computing action and entity discovery output similarities.
Figure Performance N test portion
4.2 Results
Performance Comparison Tables and present the performance evaluated on testing portion of CodeSearchNet dataset, and WebQuery Test dataset correspondingly. As it can be seen, our proposed model outperforms the baselines.
Our evaluation strategy improves performance only if the correct code snippet was ranked among the top-10 10 results returned by the CodeBERT model, rows labelled "Upper-bound" boundd report best possible performance with this evaluation strategy.
Query Complexity vs. Performance Here we present the breakdown the performance for our method baselines, using two proxies for the complexity and compositionality of the query. The first one the maximum depth of the query. We define the maximum depth as the maximum number of nested action modules in the query. The results for this experiment As we can see, 5 improves over the baseline all scenarios. It interesting to note, that while CodeBERT achieves the best performance at depth 1. We hypothesize that this can be related to the automated parsing procedure, as parsing errors are more likely be propagated in deeper queries. Further studies with carefully curated manual parses are necessary to better understand this phenomenon.
presented in Figure 5a
queries with depth 3+, model's performance peaks
Another proxy for the query complexity consider, is the number of data arguments to single action module. While the previous scenario is breaking down the performance by the depth of the query, here consider its "width". We measure the average number of entity arguments per action module the query. In the parsed portion of our dataset we have queries that range from to textual arguments per action verb. The results for this evaluation presented in Figure As it can be seen, there is no significant difference in performances between the two groups of queries in either CodeBERT our proposed method- S
4.3 Ablation Studies
Effect of Pretraining In an attempt to better understand the individual effect of the two modules as well as the roles of their pretraining and training procedures, we performed two additional ablation studies. the first one, we compare
final performance the original model with two versions where we skipped part of the pretraining. The model noted (N -AP) was trained with pretrained entity discovery module, but no pretraining was done for action module, instead we proceeded the end-to-end training directly. For the model called 5 (AP&EP), we skipped both pretrainings

---

of the entity and action modules, and just performed end-to-end training. Figure ba demonstrates that combined pretraining important for the final performance. Additionally, we wanted measure how effective the setup was without end-to-end training. 'hee results are reported in Figure 6a under the name N F. There is huge performance dip in this scenario, and while the pertormance is better than random, it is obvious that end-to-end training is crucial for NS33.
Score Normalization We wanted to determine the importance output normalization for the modules a proper probability distribution. In Figure 6b demonstrate the performance achieved using no normalization at all, normalizing either action or entity discovery module, or normalizing both. In all cases used L.1 normalization, since output scores non-negative. The version that is not normalized at all performs the worst on both datasets. The performances of the other three versions are close both datasets.
Similarity Metric Additionally, we experimented with replacing the dot product similarity with different similarity metric. In particular, in Figure be we compare the performance achieved using dot product similarity, 1.2 distance, and weighted cosine similarity. The difference in performance among different versions marginal.
4.4 Analysis and Case Study
Appendix C contains additional studies model generalization, such as handling completely unseen actions and entities, as well as the impact of the frequency of observing an action or entity during training has model performance.
Case Study Finally, we demonstrate some examples of the scores produced by our modules at different stages of training. Figure shows module score outputs for two different queries and with their corresponding code snippets. The first column shows the output of the entity discovery module after pretraining, while the second and third columns demonstrate the outputs of entity discovery and action modules after the end-to-end training. We can see that in the first column the model identifies syntactic matches, such as "folder" and a list comprehension, which "elements" could be related too. After fine-tuning we can see there is wider range of both syntactic and some semantic matches present, e.g. "dirlist" and "filelist" are correctly identified as related to "folders".
Perturbed Query Evaluation In this section we study how sensitive the models are to small changes in the query q', so that it no longer correctly describes its corresponding code snip- pet c¹. Our expectation is that evaluating a sensitive model on c' will rate the original query higher than the perturbed one. Whereas model that tends to over-generalize and ig- nore details of the query will likely rate the perturbed query similar to the original. We start from 100 different pairs (q',c'), that both our model and CodeBERT predict correctly.
We limited our study to queries with a single verb and single data entity argument to that verb. For each pair we generated perturbations of two kinds, with 20
1.0
Tertureed/Orggins
CodeBERT
0.96
0.8
perturbed versions for every query. For the first type of
0.6
0.68
perturbations, we replaced query's 'ss data argument with data argument sampled randomly from another query. For the second type, we replaced the verb argument with another randomly sampled verb. To account for calibration of the models, we measure the change in performance through ratio of the perturbed query score over original query score (lower better). The results shown in Figure 7 labelled "V(argı) V(arg2"" and "V, (arg) + V2 (arg)"
0.4
0.82
0.2
0.42
0.0
V(arg))=>V(arg2)
VV(aag)
Figure 7: Ratio of the perturbed query score to original query score (lower better) on CSN dataset.
Discussion One of the main requirements for the application of our proposed method is being able to construct semantic parse of the retrieval query. In general, it is reasonable to expect the users of the SCS to be able to come up with a formal representation of the query, e.g. by representing in form similar to SQL. or CodeQL.. However, due to the lack of such data for training and testing purposes, we implemented our own parser, which understandably does not have perfect performance since we dealing with open-ended sentences.

---

def listfolder(p):
def listfolder(p))
def Listfolder
Navigate folders
for entry in srandir.scandirg
for entry in scandiff.. (c)ndar(p):
for entry
scandir scandai
entry.is_ddr())
entry 1s_dir
entry .erry.is_dde :
yield antry.name
yield entry.name
yield entry.name nane
Remove
def dedup.11st(1):
def dedup l1st(
def dedup_ l1st(1):
redundant elements
dedup set()
Badup set()
dadun set() return x in
return
return Ix for
not (x in dedup
if not (x in dedup
not (8 in dedup
of list
or Bedup.add(xx)
or dedup. axd())))
or dedug ax)
Entity after pretraining
Entity after finetuning
Action after finetuning
Figure 8: Token scores oulpuited by the modules at different stages of training. Darker highlighting means higher The leftmost and middle columns show output scores of the entity discovery module after pretraining, and end-to-end training correspondingly. The rightmost column shows scores of the action module after the end-to-end training.
Related work
Different deep learning models have proved quite efficient when applying to programming languages and code. Prior works have studied and reviewed the uses of deep learning for code analysis in general and code search in particular 39 BI
number of approaches deep code search is based on creating relevance- prreiiccnnn model between text and code. [16] propose using RNNs for embedding both code and text to the same latent space. On the other hand, 27 capitalizes the inherent graph-like structure of programs to formulate code search as graph matching. A few works propose enriching the models handling code embedding by adding additional code analysis information, such as semantic and dependency parses 12 p. variable renaming and statement permutation [14], as well as structures such as abstract syntax tree of the program 220
37 A few other approaches have dual formulations of code In different line of work, Heyman & Cutsem [2] propose considering the code search scenario where short annotative descriptions of code snippets provided. Appendix discusses more related work.
retrieval and code summarization 9 40
6 Conclusion
We presented 3 a symbolic method for semantic code search based on neural module networks. Our method represents the query and code in terms of actions and data entities, and uses the semantic structure of the query to construct neural module network. In contrast to existing code search methods, N more precisely captures the nature of queries. In an extensive evaluation, we show that this method works better than strong but unstructured baselines. We further study model's generalization capacities, robustness, and sensibility of outputs in series of additional experiments.
Acknowledgments and Disclosure of Funding
This research is supported in part by the DARPA ReMath program under Contract No. HR0011219000 the DARPA MCS program under Contract No. 1660011924033 Office of the Director of National Intelligence (ODNI), Intelligence Advanced Research Projects Activity (IARPA), via Contract No. 2019-19900 1600007, Defense Advanced Research Projects Agency with award NSF IIS 2048211, and gift awards from Google, Amazon, JP Morgan and Sony. The views and conclusions contained herein are those of the authors and should not be interpreted necessarily representing the official policies endorsements, either expressed implied, of DARPA, ODNI, IARPA, or the U.S. Government. The U.S. Government is authorized to reproduce and distribute reprints for Govermmental purposes notwithstanding any copyright annotation thereon. We thank all the collaborators in USC INK research lab for their constructive feedback on the work.
W91 1Q1INNF19-20277 19-2027111
10

---

References
[1] Wasi Uddin Ahmad, Saikat Chakraborty, Baishakhi Ray, and Kai-Wei Chang. A transformer- based approach for source code summarization. In Dan Jurafsky. Joyce Chai, Natalie Schluter, and Joel R. Tetreault (eds.), Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, ACL 2020, Online, July 5-10, 2020, pp. 4998-5007. Association URL https:
for Computational Linguistics, 2020. doi:
//doi.or/// 18653/ www.://66//.00200... acl mai.mmainnccmm 449
[2] Shayan A. Akbar and Avinash C. Kak. SCOR: source code retrieval with semantics and order. In Margaret-Anne D. Storey, Bram Adams, and Sonia Haiduc (eds.), Proceedings of the 16th International Conference on Mining Software Repositories, MSR 2019, 26-27 May 2019, Montreal, Canada, pp. 1-12. IEEE/ ACM, 2019. doi: (00009/MSR.211..0000 URL
1109/M199S 2019. 00012
[3] Uri Alon, Meital Zilberstein, Omer Levy, and Eran Yahav. code2vec: learning distributed representations of code. Proc. ACM Program. Lang,, (PPPPP):::0:14000:: 2019. doi: 10. 145// 3290353. URL https: org/ 1145/3290353
[4] Jacob Andreas, Marcus Rohrbach, Trevor Darrell, and Dan Klein. Neural module networks, In 2016 IEEE Conference Computer Vision and Pattern Recognition, CVPR 2016, Las Vegas, NV, USA, June 27-30, 2016, pp. 48. IEEE Computer Society, 2016. doi: 10.1109/CVR.2201..11
URL https:://dot org/ 10. 1 1009/11000 CWB.///CCR.... 2016. 12
[5] Yoav Artzi, Kenton Lee, and Luke Zettlemoyer. Broad-coverage CCG semantic parsing with AMR. In Lluis Märquez, Chris KallisonnBuucche Jian Su, Daniele Pighin, and Yuval Marton (eds.). Proceedings of the 2015 Conference Empirical Methods in Natural Language Processing, EMNLP 2015, Lisbon, Portugal, September 17-21, 2015, pp. 1699-17100 The Association for Computational Linguistics, 2015. doi: 10. 10.18653/v1/dl)) 1/t1//11111111111 1198. URL https
/
[6] Nghi D. Q. Bui, Yijun Yu, and Lingxiao Jiang. Self-supervised contrastive learning for code retrieval and summarization via cemantic-preserving transformations. In Fernando Diaz, Chirag Shah, Torsten Suel, Pablo Castells, Rosie Jones, and Tetsuya Sakai (eds.), SIGIR '21: The 44th International ACM SIGIR Conference on Research and Development in Information Retrieval, Virtual Event, Canada, July 11-15, 2021, pp. 51 1-511. ACM, 2021. doi: 10.1 10.1145/3404855. 3462840. URL U httph:tt///eeec :tdtpp://doi..oo http://oo.orgg///////// 1145/3400835 3462840
[7] R. Ian Bull, Andrew Trevors, Andrew J. Malton, and Michael W. Godfrey. Semantic grep: Regular expressions + relational abstraction. In Arie van Deursen and Elizabeth Burd (eds.), 9th Working Conference on Reverse Engineering (WCRE 2002), 28 October November 2002, Richmond, VA, USA, pp. 267-276. IEEE Computer Society, 2002. doi: 10.1109/WCRE.2000 21020113084
1173084. URL https: org/10. /00
[8] Yitian Chai, Hongyu Zhang, Beijun Shen, and Xiaodong Gu. Cross-domain deep code search with few-shot meta learning. CoRR, abs/2201 .001500,000000 2022. URL httlh:tt///aaaxt org/ abs/
2201.00155
[9] Qingying Chen and Minghui Zhou. A neural framework for retrieval and summarization of source code. In Mariannne Huchard, Christian Kästner, and Gordon Fraser (eds.). Proceedings of the 33rd ACM/IEEE International Conference on Automated Software Engineering, ASE 2018, Montpellier, France, September 3-7, 2018 pp. 826 831. ACM, 2018. doi: 10. 10.1145/3238147, 3240471. URL https:://oi. org/ 10. 1e..11153338177 3240471
[10] YunSeok Choi, Jin Yeong Bak, CheolWon Na, and Jee-Hyong Lee. Learning sequential and structural information for source code summarization. In Chengqing Zong, Fei Xia. Wenjie Li, and Roberto Navigli (eds.), Findings of the Association for Computational Linguisticy: ACI/IJCNLP 2021, Online Event, August 1-6, 2021, volume ACL/IJCNL P. 2021 of Findings of ACI. pp. 2842-2851. Association for Computational Linguistics, 2021. doi: 10. 1/222/ findings-acl.33 1 URL https: http:://ooi..rrg/1.116555///////
11w.11....1111111 find:eiiiiiiinnnnieiie acl. .251

---

[11] Jacob Devlin, Ming- Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. BERT: pre- treiiriining of deep bidirectional transformers language understanding. In Jill Burstein, Christy Doran, and Thamar Solorio (eds.), Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, NAACI-HLT 2019, Minneapolis, MN, USA, June 2-7, 2019, Volume I (Long and Short Papers), pp. 4171- 4186. Association for Computational Linguistics, 2019. doi: 10.18653/v 1011865/v1/n19--1 1423. URL https://doi.co org/ 1tse//10..865/v111 1ww.1/81/11119/// 1423
[12] Lun Du, Xiaozhou Shi, Yanlin Wang, Ensheng Shi, Shi Han, and Dongmei Zhang. Is single model enough? mucos: A multi-model ensemble learning approach for semantic code search. In Gianluca Demartini, Guido Zuccon, J. Shane Culpepper, Zi Huang, and Hanghang Tong (eds.), CIKM '21: The 30th ACM International Conference on Information and Knowledge Management, Virtual Event, Queensland, Australia, November 1 5, 2021, pp. 2994 2998. ACM, 2021. doi: 10. 00.144//459637.3482222 3482127
URL BtLhtttp:://aa http::://doi.or//1//// 10. 100.11145335963
[13] Zhangyin Feng, Daya Guo, Duyu Tang, Nan Duan, Xiaocheng Feng, Ming Gong, Linjun Shou, Bing Qin, Ting Liu, Daxin Jiang, and Ming Zhou. Codebert: A pre-trained model for programming and natural languages. In Trevor Cohn, Yulan He, and Yang Liu (eds.), Findings of the Association for Computational Linguistics: EMNLP 2020, Online Event, 16-20 November 2020, volume EMNLP 2020 of Findings of ACL, pp. 1546-155..440 Association for Computational Linguistics. 2020. doi: 10. 1188653//1222000
http://////2222...nnnnnnnmoo findings-emmlp. 139.. URL https
//doi.org/10.. 100.00..66631111111 http:///..2.......n....... emnlp 139
[14] Wenchao Gu, Zongjie Li, Cuiyun Gao, Chaozheng Wang, Hongyu Zhang, Zenglin Xu, and Michael R. Lyu. Cradle: Deep code retrieval based on semantic dependency learning. CoRR, abs/2012.01028, 2020. URL ltttps:t/tt/iiiiii org/ hts//bb//2001000 01028
[15] Wenchao Gu, Zongjie Li, Cuiyun Gao, Chaozheng Wang, Hongyu Zhang, Zenglin Xu, and Michael R. Lyu. Cradle: Deep code retrieval based on semantic dependency learning. Neural Networks, 141:385-332 394, 2021. doi: 10.1016fj.neumet22022 2021.00.00 .04.019. URL httpp:://ooi org/
10. 10.161 www.:eueeee.... 2021 04.009
[16] Xiaodong Gu, Hongyu Zhang, and Sunghun Kim. Deep code search. In Michel Chaudron, Ivica Crnkovic, Marsha Chechik, and Mark Harman (eds.), Proceedings of the 40th International Conference on Software Engineering, ICSE 2018, Gothenburg, Sweden, May 27 June 03, URL http:: //oii. org/
2018. pp. 933 944, ACM, 2018. doi: 10.1145/180015533180 10. 10.1145/3180155 3180167
[17] Daya Guo, Shuo Ren, Shuai Lu, Zhangyin Feng, Duyu Tang, Shujie Liu, Long Zhou, Nan Duan, Alexey Svyatkovskiy, Shengyu Fu, Michele Tufano, Shao Kun Deng, Colin B. Clement, Dawn Drain, Neel Sundaresan, Jian Yin, Daxin Jiang, and Ming Zhou Graphcodebert: Pre- training code representations with data flow, In 9th International Conference on Learning Representations, ICLR 2021, Virtual Event, Austria, May 3-7, 2021. OpenReview.net, 2021. URL htttss:/
LLC422433P
[18] Daya Guo, Shuai Lu, Nan Duan, Yanlin Wang, Ming Zhou, and Jian Yin. Unixcoder: Unified cross-modal pre- traiirinn for code representation. In Smaranda Muresan, Preslav Nakov, and Aline Villavicencio (eds.), Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume I: Long Papers), ACI. 2022, Dublin, Ireland, May 22- 27, 2022, pp. 7222-7225. Association for Computational Linguistics, 2022. URL https //aclanthology. wrw./o22.........oo 2002202222222222 aon-- 499
[19] Michael Gutmann and Aapo Hyvärinen. Noise- contrestnt estimation: A new estimation principle for unnormalized statistical models. In Yee Whye Teh and D. Mike Titterington (eds.), Proceedings of the Thirteenth International Conference on Artificial Intelligence and Statistics, AISTATS 2010, Chia Laguna Resort, Sardinia, Italy, May 13-15, 2010, volume 9 of JMLR Proceedings, pp. 297-399 JMLR.org, 2010. URL httt:::/prcceedinnnee mll. press/v9/ gutmann10a. html
[20] Rajarshi Haldar, Lingfei Wu, Jinjun Xiong, and Julia Hockenmaier. A multi-perspeetive architecture for semantic code search. In Dan Jurafsky, Joyce Chai, Natalie Schluter, and Joel R.
12

---

Tetreault (eds.), Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, ACL 2020, Online, July 5-10. 2020, pp. 8563- 8568. Association for Computational Linguistics, 2020. doi: 10. 1221//////2002.llllllll 758. URL htt.hhttps//ooooo org/ 1org/10.11 18653/
11/2020.acl1100 main. 758
[21] Geert Heyman and Tom Van Cutsem. Neural code search revisited: Enhancing code snippet retrieval through natural language intent. CoRR, abs/2008. 2020. URL https://arxiv org/abs/2008. 12199
[22] Matthew Honnibal, Ines Montani, Sofie Van Landeghem, and Adriane Boyd. spaCy: Industrial- strength Natural Language Processing Python. 2020. doi: 10.5281/zenodo. 12t.303.
[23] Junjie Huang, Duyu Tang, Linjun Shou, Ming Gong, Ke Xu, Daxin Jiang, Ming Zhou, and Nan Duan. Cosqa: 20,000+ web queries for code search and question answering. In Chengqing Zong, Fei Xia, Wenjie Li, and Roberto Navigli (eds.), Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), ACL 2021, Online, August 1-6, 2021, pp. 5690-5700. Association for Computational Linguistics, 2021. doi: 10. 10.18653/v1/20211 acI-long.40 442. URL https: //doi oto/ 18653/v1/20211 acl. acl-lonll long 442
[24] Hamel Husain, Ho-Hsiang Wu, Tiferet Gazit, Miltiadis Allamanis, and Marc Brockschmidt. Codesearchnet challenge: Evaluating the state of semantic code search. hts/1999.09436 2019.
URL httpt:tttss:xiii/iiee org/ abs/ 1909. 09436
[25] Paras Jain, Ajay Jain, Tianjun Zhang, Pieter Abbeel, Joseph Gonzalez, and Ion Stoica. Con- trastive code representation learning. In Marie-Francine Moens, Xuanjing Huang. Lucia Specia, and Scott Wen-tau Yih (eds.), Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, EMNLP 2021, Virtual Event / Punta Cana, Dominican Re- public, 7-11 November, 2021, pp. 5954-5971, Association for Computational Linguistics. 2021. doi: 10. 10.18653/v1/22 (tt.6//////22...//o/ http:////22....mmmmaaaa... .emmlp-main.482 URL https:// org/ro..///oo w0.........///.............. 2021. emnlp-main 482
[26] Aditya Kanade, Petros Maniatis, Gogul Balakrishnan, and Kensen Shi. Learning and evaluating contextual embedding of source code. Proceedings of the 37th International Conference on Machine Learning. ICML 2020, 13-18 July 2020. Virtual Event, volume 119 of Proceedings of Machine Learning Research, pp. 5110-5121, PMLR, 2020. URL http::/proccedinng mll press/vi v1 v119 2111199kaaad222a. html
[27] Xiang Ling, Lingfei Wu, Saizhuo Wang, Gaoning Pan, Tengfei Ma, Fangli Xu, Alex X. Liu, Chunming Wu, and Shouling Ji. Deep graph matching and searching for semantic code retrieval. ACM Trans. Knowl. Discow. Data 11:5::881188:211 2021. doi: 10.1145/3447571. URL
https://doi.orge 111111..11111/4 1111/14444447
[28] Shangqing Liu, Xiaofei Xie, Lei Ma. Jing Kai Siow, and Yang Liu. Graphsearchnet: Enhancing gnns via capturing global dependency for semantic code search. CoRR, /abb///11.022611 2021.
htttps:////a org/abs.///
[29] Yinhan Liu, Myle Out, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. Roberta: A robustly optimized bert pretraining approach. abs//907. 11692, 2019. URL httpp:://rxxxi wwwpxxiv..rrreeeooo abs/ 11692
[30] Meili Lu, Xiaobing Sun, Shaowei Wang, David Lo, and Yucong Duan. Query expansion via wordnet for effective code search. In Yann-Gaël Guéhéneue, Bram Adams, and Alexander Serebrenik (eds.), 22nd IEEE International Conference on Software Analysis, Evolution, and Reengineering, SANER 2015, Montreal. QC. Canada, March 2-6, 2015, pp. 545-544454 549. IEEE Computer Society, 2015, doi: 10. 10/1109/ANN..221.... 1874. URL htRlshtts:://// htttps://doi.org/11 1109/SANER, 2015. 7081874
[31] Shuai Lu, Daya Guo, Shuo Ren, Junjie Huang, Alexey Svyatkovskiy, Ambrosio Blanco, Colin Clement, Dawn Drain, Daxin Jiang, Dayu Tang, Ge Li, Lidong Zhou, Linjun Shouv, Long Zhou, Michele Tufano, MING GONG, Ming Zhou, Nan Duan, Neel Sundaresan, Shao Kun Deng, Shengyu Fu, and Shujie LIU. CodeXGLUE: A machine learning benchmark dataset for code
13

---

understanding and generation. In Thirty-fftth Conference on Neural Information Processing Systems Datasets and Benchmarks Track (Round 1), Online, Dec 7-10. 2021. OpenReview.net, 2021. URL httpss://
net. wtt.://weettttormmrmdddeaaeaAaad
[32] Shuai Lu,, Nan Duan, Hojae Han, Daya Guo, Seung won Hwang, and Alexey Svyatkovskiy. Reace: A retrieval- -aureveal-ugmenteed code completion framework. In Smaranda Muresan, Preslav Nakov, and Aline Villavicencio (eds.), Proceedings of the 60th Annual Meeting of the Asso- ciation for Computational Linguistics (Volume I: Long Papers), ACI. 2022, Dublin, Ireland, Association for Computational Linguisties, 2022. URL
May 22-27, 2022, pp. Ngtg/2022 http:2222222ccc21 aal.llog..4..4 .431
https: https://aclantholoog org
[33] Steven P. Reiss. Semantics- based code search demonstration proposal. In 25th IEEE Interna- tional Conference on Software Maintenance (ICSM 2009), September 20-26, 2009, Edmonton, Alberta, Canada, pp. 385-386. IEEE Computer Society, 2009. doi: 10. 10.1109/ICSM.2000 5306319, URL https: org/orr.rooo 11w./I09MMICMMM
5306319
[34] Ensheng Shi, Yanlin Wang. Lun Du, Hongyu Zhang, Shi Han, Dongmei Zhang, and Hongbin Sun. CAST: enhancing code summarization with hierarchical splitting and reconstruction of abstract syntax trees. In Marie-Francine Moens, Xuanjing Huang, Lucia Specia, and Scott Wen- tau Yih (eds.), Proceedings of the 202/ Conference Empirical Methods in Natural Language Processing, EMNLP 2021, Virtual Event Punta Cana, Dominican Republic, 7-11 November, 2021, pp. 4053-4062. Association for Computational Linguisties, 2021. doi: 18.13//3/v 1/2021. emnip-main.332. URL https: /actps://doi... org/ 10. 18653/11/222211111
dictive coding. CoRR, abs/1807 0074003748, 2018. URL http:///ww1/222..mmmmmmmmmmaa emnlp-main....mmmmmmmmmmmmmmmm 332
[35] Aäron van den Oord, Yazhe Li, and Oriol Vinyals. Representation leaming with contrastive pre- 1ww.b/////1....0 03748
[36]
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. In Isabelle Guyon, Ulrike von Luxburg, Samy Bengio, Hanna M. Wallach, Rob Fergus, S. V. N. Vishwanathan, and Roman Garnett (eds.). Advances in Neural Information Processing Systems 30: Annual Con- ference on Neural Information Processing Systems 2017, December 4-9, 2017, Long Beach, CA, USA, pp. 5998-6008, 2017. URL https::/proceeddiiiiie wwwrrinnurpppp.. /00ppp////c///p/r///0/////////// www.::055cc144445aaaaaa Abstract html
[37] Yao Wan, Jingdong Shu, Yulei Sui, Guandong Xu, Zhou Zhao, Jian Wu, and Philip S. Yu. Multi-modal attention network learning for semantic source code retrieval. In 34th IEEE/ACM International Conference Automated Software Engineering, ASE 2019, San Diego, CA, URL
USA, November 11-15, 2019, pp. 13-25. IEEE, 2019, doi: 10.1109/ASE.2019000
https://doi.org oroooogg//111 10. 1109/ASE. 2019. 00012
[38] Xin Wang, Yasheng Wang, Yao Wan, Jiawei Wang, Pingyi Zhou, Li Li, Hao Wu, and Jin Liu. CODE-MVP: learning to represent source code from multiple views with contrastive pre-training. CoRR, abs/2205.022 02029, 2022. doi: 10.48550/axxx.2225.02222 URL https
/ /dooi org/1...111..00 wtt.ooo///....////223...... arxiv. 2205.02022
[39] Frank F. Xu, Bogdan Vasilescu, and Graham Neubig, In-ide code generation from natural language: Promise and challenges. CoRR, abs/2101. 149, 2021. URL https://://// org/ abs/ 0bs/2101.1111
[40] Ziyu Yao, Jayavardhan Reddy Peddamail, and Huan Sun. Coacor: Code annotation for code retrieval with reinforcement learning. In Ling Liu, Ryen W. White, Amin Mantrach, Fabrizio Silvestri, Julian J. McAuley, Ricardo Baeza- Yates, and Leila Zia (eds.), The World Wide Web Conference, WWW 2019, San Francisco, CA, USA, May 13-17, 2019, pp. 2203-2214, ACM, 2019. doi: 10. 01114/3308558.3313666 3313622
https //dois org/ 10. 1145/3308558
[41] Wei Ye, Rui Xie, Jinglei Zhang, Tianxiang Hu, Xiaoyin Wang, and Shikun Zhang. Leveraging code generation to improve code retrieval and summarization via dual learning. In Yennun Huang, Irwin King, Tie- Liu, and Maarten van Steen (eds.). WWW '20: The Web Conference 2020, Taipei, Taiwan, April 20-24, 2020, pp. 2309 2319. ACM IW3C2, 2020. doi: 10. 145/ 3366423.338029. URL https: /ooi org/ 10. 1141 31645//336423... 3380295
14

---

[42] Pengcheng Yin, Bowen Deng, Edgar Chen, Bogdan Vasilescu, and Graham Neubig. Learning to mine aligned code and natural language pairs from stack overflow. In International Conference on Mining Software Repositories, MSR, pp. 476-486. ACM, 2018. doi: https://doi.org/ 10. 1145/ 3196398.3196490
[43] Luke S. Zettlemoyr and Michael Collins. Learning to map sentences to logical form: Structured classification with probabilistic categorial grammars. CoRR, abs/1 1420, 2012. URL
http://arxiv.orrg abs/ 1207 1420
[44]
Jian Zhang, Xu Wang, Hongyu Zhang, Hailong Sun, Kaixuan Wang, and Xudong Liu. novel neural source code representation based on abstract syntax tree. In Joanne M. Atlee, Tevfik Bultan, and Jon Whittle (eds.), Proceedings of the 4 International Conference on Software Engineering, ICSE 2019, Montreal, QC, Canada, May 25-31, 2019, pp. 783-794 doi: 10.1 R..1009CCEE.209.000 2019.0006.. URL httpt:://oo.i..... org/ 100 1109/
IEEE/ / ACM, 2019.
[45]
Gang Zhao and Jeff Huang. Deepsim: deep learning code functional similarity. In Gary Leavens, Alessandro Garcia, and Corina S. Pasareanu (eds.), Proceedings of the 2018 ACM Joint Meeting on European Software Engineering Conference and Symposium on the Foundations of Software Engineering, ESEC/SIGSOFT FSE 2018, Lake Buena Vista, FL, USA, November 04-09, 2018, pp. 141-151, ACM, 2018. doi: 10.1145/2360243333666 URL https:///ooi org 10. 1 1 45/ 3236024. 3236068
[46] Renyu Zhu, Lei Yuan, Xiang Li, Ming Gao, and Wenyuan Cai. A neural network architecture for program understanding inspired by human behaviors. In Smaranda Muresan, Preslav Nakov, and Aline Villavicencio (eds.), Proceedings of the 60th Anmual Meeting of the Association for Computational Linguistics (Volume I: Long Papers), ACI. 2022, Dublin, Ireland, May 22-27, 2022, pp. 5142-5153. Association for Computational Linguistics, 2022. URL https /acclatthoooyy.oo org/ 2022. acl- .53
[47] Daniel Zügner, Tobias Kirschstein, Michele Catasta, Jure Leskovec, and Stephan Günnemann. Language-agnostic representation learning of source code from structure and context. In 9th In- ternational Conference on Learning Representations, ICLR 2021, Virtual Event, Austria, May 7, 2021. OpenReview.net, 2021. / http://eww...eeeeeeooooooocaaaaa
Checklist
1. For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope? [Yes]
(b) Did you describe the limitations of your work? [Yes] See the Discussion paragraph under Section 44
(c) Did you discuss any potential negative societal impacts of your work? [N/A]
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them? [Yes]
2. If you including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results? [N/A]
(b) Did you include complete proofs of all theoretical results? [N/A]
3. If you ran experiments...
(a) Did you include the code, data, and instructions needed to reproduce the main experi- mental results (either in the supplemental material or as a URL)? [Yes] Code and data supplemental material
(b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)? [Yes] See paragraph titled Hyperparameters in Section 41
(c) Did you report error bars (e-g., with respect the random seed after running experi- ments multiple times)? [No]
15

---

(d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)? [Yes]
4. If you using existing assets (e.g., code, data, models) curating/releasing assets...
(a) If your work uses existing assets, did you cite the creators? [Yes]
(b) Did you mention the license of the assets? [No]
(c) Did you include any new assets either in the supplemental material or as URL? [No]
(d) Did you discuss whether and how consent was obtained from people whose data you're 're using/curating? [N/A]
(e) Did you discuss whether the data you are using/curating contains personally identifiable information or offensive content? [N/A]
5. If you used crowdsourcing or conducted research with human subjects...
(a) Did you include the full text of instructions given participants and screenshots, applicable? [N/A]
(b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, applicable? [N/A]
(c) Did you include the estimated hourly wage paid to participants and the total amount spent on participant compensation? [N/A]
16


Reviews:
[{"review_id": "ndfG3I7-W_l", "reviewer": "Reviewer_kn2F", "summary": "This paper proposes NS3, Neuro-Symbolic Semantic Code Search. NS3 supplements the query sentence with a layout of its semantic structure, which is then used to break down the final reasoning decision into a series of lower-level decisions. NS3 outperforms baselines on CodeSearchNet and CoSQA.", "questions": "What are the process of engineering parsing rules and NL-action mappings? Are they engineered according to the performance on the dev set or the test set?", "limitations": "N/A", "rating": 5, "confidence": 3, "soundness": 2, "presentation": 4, "contribution": 2, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n1. NS3 is evaluated in two well-established benchmarks, CodeSearchNet and CoSQA, and compared against strong baselines.\n2. The empirical result is very strong. The proposed method, NS3, outperforms state-of-the-art by a large margin.\n3. The paper includes detailed experiments on reduced dataset settings and ablation studies.\n\nWeaknesses:\n1. NS3 requires rule-based parsing of natural languages, while other baselines, such as CodeBERT, only involve natural language models. Rule-based parsing of natural languages is complicated, language-dependent, and less scalable than language models. In `parser_dict.py` of the released parser, there are hundreds of task-specific parsing/synonym rules for natural languages. These rules are not transferrable to different natural languages (e.g. English to Chinese), and they even cannot generalize to different programming languages (e.g. Python to C++).", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "bzEuFDADzfa", "reviewer": "Reviewer_SyPM", "summary": "The paper introduces a novel method for semantic code search using neural module\nnetworks. The layout of the network is produced from a semantic parse of the\nquery. There are two types of modules: entity discovery modules and action\nmodules. Entity discovery modules correspond to the nouns in the semantic parse\nand each such module tries to discover the given entity (noun) of the query in\nthe code (they assign a relevance score to each code token).\n\nAction modules correspond to the verbs in the query. Each action module receives\nonly part of the full query with the last entity argument masked, and tries to\ndiscover the masked entity (estimate its relevance scores). The intuition is\nthat if the code snippet indeed corresponds to the query, then the action module\nshould be able to estimate the relavence scores of the masked entity based on\nthe code and the rest of the query. Nested action modules are flattened and\ntheir scores are multiplied together.\n\nThe final score which measures the relatedness of the code snippet to the query\nis computed by taking the normalized dot product of the relevance scores of an\nentity computed by its entity discovery module and by the action module in which\nit was masked. If there are multiple such scores (because there are multiple\naction modules) they are multiplied together.\n", "questions": "Why is batching hard for the model? (it was mentioned in line 237)\n\nIt was mentioned that the performance of the model peaks at depth=1. Could that\nbe because of how compositional queries are handled (as described in lines\n219-228)?\n\nI found Figure 4 confusing. Why are both \"Load 0\" and \"Load from\" present in the\nfigure? On Figure 2 we just have \"Load from\" for the same action module and\nquery. Also, there is just one unmasked entity and it's without a preposition, so maybe\n\"Load 0\" would be appropriate?\n", "limitations": "I cannot think of any limitations which are not addressed.\n", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "## Strengths\n\nThe paper is an interesting and novel application of neural module networks to\nsemantic code search and improves the state-of-the-art results.\n\n## Weaknesses\n\nI think the main problem with the paper is that some parts are hard to\nunderstand in their current form. The introduction could be more concrete with\ninformation incorporated from the caption of Figure 2 and Section 3. For\nexample, I think that the introduction should mention the intuition behind\naction and entity discovery modules, and how that is used to compute the\nrelatedness score.\n\nI found the second paragraph of Section 3.3 especially hard to understand, I\nthink it should be elaborated. One reason for my confusion was the dual role of\nprepositions: they belong to the entities, but we embed them with the verbs. I\nstill don't understand what happens when we have multiple entities as the input\ndimension of the transformer would change depending on the number of entities.\nAlso, the dimensions, and what is concatenated to what and in which direction is\nnot clear.\n\nLine 208: the code token embedding should be $c^j_k$ and not $t_k$.\n\nLine 246: \", Section 4.1,\" is somewhat confusing, \"later in this Section\" would be better\n\nLines 268-277: some of the methods are not cited.\n\nI find it odd that there is a \"Background\" and a \"Related work\" section, as\nthe Background already discusses the limitations of related work.\n\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "ZNF0I8gzSWU", "reviewer": "Reviewer_dTG5", "summary": "The paper proposes the NS3 (Neuro-Symbolic Semantic Search) model, which breaks down the query into small phrases using the Categorial Grammar-based semantic parse module, which can better understand the compositional and longer queries.\n\nTo identify the similarity of each query and code snippet, the NS3 model uses two types of neural models, the entity discovery module and the action module. The entity discovery module uses a transformer encoder model and a two-layer MLP to identify the entities and their relevances. RoBERTa model initialization and noisy supervision training are applied in its self-supervised pretraining phase. The Action module architecture is similar to the entity discovery module. It estimates the action similarity through the prediction of the masked entity embedding. The module can be pre-trained with a mask-predict process for the masked entity. After pretraining, the model performs end-to-end fine-tuning for two modules.\n\nThe experiments on the CSN and CoSQA datasets show the superiority of the proposed model over the baseline methods on the parsable samples. Furthermore, the ablation study validates the effectiveness of the pretraining and investigates different score normalization methods and similarity metrics.\n", "questions": "1. Could you elaborate on the influence of the parsable sample selection on the dataset? For example, are the long sentences and samples hard-to-understand remained?\n\n2. Is there any sample that can validate the NS3 model mimics the staged reasoning for SCS? \n", "limitations": "In the experiments, according to my understanding, only parsable data are used in the evaluation. I would suggest using all the data in the experiments to verify if the proposed method still helps in improving the SCS performance. In other words, it is not fair to compare with other baselines and models using only the dataset bias towards your method.", "rating": 5, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "**Strengths**:\n\n1. The proposed NS3 model utilizes the Categorial Grammar-based(CCG) semantic parser to comprehend the semantic structure of the query better and combines the Transformer-based neural model to capture semantic information of the query text, which is an interesting idea.\n\n2. The experiment validates the effectiveness of the proposed modules and the pretraining strategy. Furthermore, the proposed model achieves noticeable improvements over the baseline models in the parsable samples.\n\n\n**Weakness**:\n\n1. The proposed model could not operate properly in arbitrary natural language scenarios that 60% of the CSN dataset and 30% of the CoSQA dataset records are not parsable.\n\n2. According to Line 58, the author claims the model mitigates the challenges of encoding long texts and mimics the staged reasoning for SCS. However, according to Figure 5(a), the NS3 model is not significantly improved in the deeper query situation (D=3+). The model performs better on the query with a simple semantic structure (D=1), which is inconsistent with the initial claim.\n\n3. The unparsable issue restricts the quantity and quality of the experimental data. According to Table 1 and [1], the MRR of the GraphCodeBert model is higher on the parsable dataset (0.812 vs. 0.692). The parsable data may be easier to comprehend through the NS3 model, making the experiment comparison unfair.\nReference:\n\n[1] Guo, D., Ren, S., Lu, S., Feng, Z., Tang, D., Liu, S., ... & Zhou, M. (2020). Graphcodebert: Pre-training code representations with data flow. arXiv preprint arXiv:2009.08366.\n\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "REjXBoVvCM", "reviewer": "Reviewer_FqUC", "summary": "The paper studies the problem of retrieving code snippets given textual queries (called semantic code search). The work is motivated by language models’ limitations on encoding longer and compositional text (which I question a bit about, see my comments below). The authors propose a neural module network (called NS3) and introduce a modular workflow according to the semantic structure of the query. More specifically, NS3 contains two types of neural modules, entity discovery module, and action module, to estimate the semantic relatedness of code tokens and entity mentions and actions in a query separately. It decomposes the task into multiple steps of matching each data entity and action in the query. The authors demonstrate the effectiveness of the proposed method on several code search benchmarks and show that their method outperforms some strong baselines in some settings (on which I’m a bit confused).\n", "questions": "Line 246-248: I’m confused about the evaluation data statistics. Is it a common practice to exclude unparsable examples for evaluation? It looks like that the authors did that only because of the limitations of their method (only taking parsable queries). It would be helpful to provide data statistics both before (original) and after parsing in Appendix A.2 (Table 3).\n\nLine 254: Did you randomly select 5k and 10k examples? If they are randomly chosen did you report results on multiple randomly selected examples? \n\nLine 264-266: Why did you focus on the two-step evaluation? Again, do CodeBERT/GraphCodeBERT themselves (not by yourselves) also report results in this setting? It seems that they can be applied in a single-step setting. Why not evaluate your method in that setting too without depending on CodeBERT’s first step predictions (then only applying your method to rerank the top 10 CodeBERT predictions)? As you mentioned in Line 303-305, the highest possible results you could get with this evaluation strategy is kind of low (74% on CoSQA…)...\n\nLine 307- Fig. 5: Do you have any explanations about why GraphCodeBERT performs much worse than CodeBERT in many cases? Is it the strongest baseline you compare with?\n", "limitations": "N/A", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths: \n\nThe problem of semantic code search is very interesting. It is easy to follow the writing of the paper. The authors compare the proposed methods with other works and show strong performance on multiple code search benchmarks. \n\nWeaknesses:\n\nThe authors mention that one of the main motivations of the work is because language models struggle with encoding long and compositional text. I’m a bit suspicious. Text queries as examples shown in the paper (but not just these examples, generally speaking) are not very long and complicated. Some of them are even too simple and ignore some details, which leads to mismatching between the query and code (which could be the real challenge). Language models are able to encode much more complex and longer texts than these examples in the paper….\n\nAccording to what the authors say in lines 265-266, CodeBERT and many other simpler approaches can be fed examples in batches, which make them much faster in retrieval settings. In real-world retrieval settings, efficiency is also a very important consideration. In this case, the proposed method NS3 is less attractive (also depending on other methods such as CodeBERT). This makes NS3 look more like a re-ranking model instead of a fully actionable code retrieval model. \n\nThe query parser implementation seems to be a lot of human work (e.g., building a vocab of action and entity words).\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}]

Author rebuttal:

Decision: Accept