Venue: NeurIPS 2022 Conference

Reviews:

final_score: rating: 5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.
review_content: summary: This paper proposes NS3, Neuro-Symbolic Semantic Code Search. NS3 supplements the query sentence with a layout of its semantic structure, which is then used to break down the final reasoning decision into a series of lower-level decisions. NS3 outperforms baselines on CodeSearchNet and CoSQA.
strengths_and_weaknesses: Strengths:
1. NS3 is evaluated in two well-established benchmarks, CodeSearchNet and CoSQA, and compared against strong baselines.
2. The empirical result is very strong. The proposed method, NS3, outperforms state-of-the-art by a large margin.
3. The paper includes detailed experiments on reduced dataset settings and ablation studies.

Weaknesses:
1. NS3 requires rule-based parsing of natural languages, while other baselines, such as CodeBERT, only involve natural language models. Rule-based parsing of natural languages is complicated, language-dependent, and less scalable than language models. In `parser_dict.py` of the released parser, there are hundreds of task-specific parsing/synonym rules for natural languages. These rules are not transferrable to different natural languages (e.g. English to Chinese), and they even cannot generalize to different programming languages (e.g. Python to C++).
questions: What are the process of engineering parsing rules and NL-action mappings? Are they engineered according to the performance on the dev set or the test set?
limitations: N/A
review_title: null

final_score: rating: 5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.
review_content: summary: The paper studies the problem of retrieving code snippets given textual queries (called semantic code search). The work is motivated by language models’ limitations on encoding longer and compositional text (which I question a bit about, see my comments below). The authors propose a neural module network (called NS3) and introduce a modular workflow according to the semantic structure of the query. More specifically, NS3 contains two types of neural modules, entity discovery module, and action module, to estimate the semantic relatedness of code tokens and entity mentions and actions in a query separately. It decomposes the task into multiple steps of matching each data entity and action in the query. The authors demonstrate the effectiveness of the proposed method on several code search benchmarks and show that their method outperforms some strong baselines in some settings (on which I’m a bit confused).

strengths_and_weaknesses: Strengths: 

The problem of semantic code search is very interesting. It is easy to follow the writing of the paper. The authors compare the proposed methods with other works and show strong performance on multiple code search benchmarks. 

Weaknesses:

The authors mention that one of the main motivations of the work is because language models struggle with encoding long and compositional text. I’m a bit suspicious. Text queries as examples shown in the paper (but not just these examples, generally speaking) are not very long and complicated. Some of them are even too simple and ignore some details, which leads to mismatching between the query and code (which could be the real challenge). Language models are able to encode much more complex and longer texts than these examples in the paper….

According to what the authors say in lines 265-266, CodeBERT and many other simpler approaches can be fed examples in batches, which make them much faster in retrieval settings. In real-world retrieval settings, efficiency is also a very important consideration. In this case, the proposed method NS3 is less attractive (also depending on other methods such as CodeBERT). This makes NS3 look more like a re-ranking model instead of a fully actionable code retrieval model. 

The query parser implementation seems to be a lot of human work (e.g., building a vocab of action and entity words).

questions: Line 246-248: I’m confused about the evaluation data statistics. Is it a common practice to exclude unparsable examples for evaluation? It looks like that the authors did that only because of the limitations of their method (only taking parsable queries). It would be helpful to provide data statistics both before (original) and after parsing in Appendix A.2 (Table 3).

Line 254: Did you randomly select 5k and 10k examples? If they are randomly chosen did you report results on multiple randomly selected examples? 

Line 264-266: Why did you focus on the two-step evaluation? Again, do CodeBERT/GraphCodeBERT themselves (not by yourselves) also report results in this setting? It seems that they can be applied in a single-step setting. Why not evaluate your method in that setting too without depending on CodeBERT’s first step predictions (then only applying your method to rerank the top 10 CodeBERT predictions)? As you mentioned in Line 303-305, the highest possible results you could get with this evaluation strategy is kind of low (74% on CoSQA…)...

Line 307- Fig. 5: Do you have any explanations about why GraphCodeBERT performs much worse than CodeBERT in many cases? Is it the strongest baseline you compare with?

limitations: N/A
review_title: null

final_score: rating: 5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.
review_content: summary: The paper proposes the NS3 (Neuro-Symbolic Semantic Search) model, which breaks down the query into small phrases using the Categorial Grammar-based semantic parse module, which can better understand the compositional and longer queries.

To identify the similarity of each query and code snippet, the NS3 model uses two types of neural models, the entity discovery module and the action module. The entity discovery module uses a transformer encoder model and a two-layer MLP to identify the entities and their relevances. RoBERTa model initialization and noisy supervision training are applied in its self-supervised pretraining phase. The Action module architecture is similar to the entity discovery module. It estimates the action similarity through the prediction of the masked entity embedding. The module can be pre-trained with a mask-predict process for the masked entity. After pretraining, the model performs end-to-end fine-tuning for two modules.

The experiments on the CSN and CoSQA datasets show the superiority of the proposed model over the baseline methods on the parsable samples. Furthermore, the ablation study validates the effectiveness of the pretraining and investigates different score normalization methods and similarity metrics.

strengths_and_weaknesses: **Strengths**:

1. The proposed NS3 model utilizes the Categorial Grammar-based(CCG) semantic parser to comprehend the semantic structure of the query better and combines the Transformer-based neural model to capture semantic information of the query text, which is an interesting idea.

2. The experiment validates the effectiveness of the proposed modules and the pretraining strategy. Furthermore, the proposed model achieves noticeable improvements over the baseline models in the parsable samples.


**Weakness**:

1. The proposed model could not operate properly in arbitrary natural language scenarios that 60% of the CSN dataset and 30% of the CoSQA dataset records are not parsable.

2. According to Line 58, the author claims the model mitigates the challenges of encoding long texts and mimics the staged reasoning for SCS. However, according to Figure 5(a), the NS3 model is not significantly improved in the deeper query situation (D=3+). The model performs better on the query with a simple semantic structure (D=1), which is inconsistent with the initial claim.

3. The unparsable issue restricts the quantity and quality of the experimental data. According to Table 1 and [1], the MRR of the GraphCodeBert model is higher on the parsable dataset (0.812 vs. 0.692). The parsable data may be easier to comprehend through the NS3 model, making the experiment comparison unfair.
Reference:

[1] Guo, D., Ren, S., Lu, S., Feng, Z., Tang, D., Liu, S., ... & Zhou, M. (2020). Graphcodebert: Pre-training code representations with data flow. arXiv preprint arXiv:2009.08366.


questions: 1. Could you elaborate on the influence of the parsable sample selection on the dataset? For example, are the long sentences and samples hard-to-understand remained?

2. Is there any sample that can validate the NS3 model mimics the staged reasoning for SCS? 

limitations: In the experiments, according to my understanding, only parsable data are used in the evaluation. I would suggest using all the data in the experiments to verify if the proposed method still helps in improving the SCS performance. In other words, it is not fair to compare with other baselines and models using only the dataset bias towards your method.
review_title: null

final_score: rating: 7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
review_content: summary: The paper introduces a novel method for semantic code search using neural module
networks. The layout of the network is produced from a semantic parse of the
query. There are two types of modules: entity discovery modules and action
modules. Entity discovery modules correspond to the nouns in the semantic parse
and each such module tries to discover the given entity (noun) of the query in
the code (they assign a relevance score to each code token).

Action modules correspond to the verbs in the query. Each action module receives
only part of the full query with the last entity argument masked, and tries to
discover the masked entity (estimate its relevance scores). The intuition is
that if the code snippet indeed corresponds to the query, then the action module
should be able to estimate the relavence scores of the masked entity based on
the code and the rest of the query. Nested action modules are flattened and
their scores are multiplied together.

The final score which measures the relatedness of the code snippet to the query
is computed by taking the normalized dot product of the relevance scores of an
entity computed by its entity discovery module and by the action module in which
it was masked. If there are multiple such scores (because there are multiple
action modules) they are multiplied together.

strengths_and_weaknesses: ## Strengths

The paper is an interesting and novel application of neural module networks to
semantic code search and improves the state-of-the-art results.

## Weaknesses

I think the main problem with the paper is that some parts are hard to
understand in their current form. The introduction could be more concrete with
information incorporated from the caption of Figure 2 and Section 3. For
example, I think that the introduction should mention the intuition behind
action and entity discovery modules, and how that is used to compute the
relatedness score.

I found the second paragraph of Section 3.3 especially hard to understand, I
think it should be elaborated. One reason for my confusion was the dual role of
prepositions: they belong to the entities, but we embed them with the verbs. I
still don't understand what happens when we have multiple entities as the input
dimension of the transformer would change depending on the number of entities.
Also, the dimensions, and what is concatenated to what and in which direction is
not clear.

Line 208: the code token embedding should be $c^j_k$ and not $t_k$.

Line 246: ", Section 4.1," is somewhat confusing, "later in this Section" would be better

Lines 268-277: some of the methods are not cited.

I find it odd that there is a "Background" and a "Related work" section, as
the Background already discusses the limitations of related work.


questions: Why is batching hard for the model? (it was mentioned in line 237)

It was mentioned that the performance of the model peaks at depth=1. Could that
be because of how compositional queries are handled (as described in lines
219-228)?

I found Figure 4 confusing. Why are both "Load 0" and "Load from" present in the
figure? On Figure 2 we just have "Load from" for the same action module and
query. Also, there is just one unmasked entity and it's without a preposition, so maybe
"Load 0" would be appropriate?

limitations: I cannot think of any limitations which are not addressed.
review_title: null

Meta-review: metareview: The paper studies the problem of retrieving code snippets given textual queries (NS3, Neuro-Symbolic Semantic Code Search).  The work is motivated by language models’ limitations on encoding longer and not providing a faithful explanation of their reasoning on compositional queries.  NS3 supplements the query sentence with a layout of its semantic structure, which is then used to break down the final reasoning decision into a series of lower-level decisions. NS3 outperforms baselines on CodeSearchNet and CoSQA.


Overall, the reviewers liked the motivation of the work. A lot of concerns have been raised with extensive responses from the authors. Some of these concerns remain and have to be acknowledged and clarified in the modified document. **Despite the limitations of the work, I think "accepting" this work outweighs "rejecting" it, assuming that the authors will put due diligence into improving their drafts based on their latest experiments (outlined in the author's response), along with a clear discussion of their limitations.**



Let me start with the strengths: 

 - All reviewers have found the work interesting. 

 - Strong empirical results: NS3 is evaluated on two well-established benchmarks, CodeSearchNet and CoSQA. The empirical result is very strong. The proposed method, NS3, outperforms state-of-the-art by a large margin.

 - The paper includes detailed experiments on reduced dataset settings and ablation studies.


Here are several points of concern that came up in the reviews: 

 - NS3 requires rule-based parsing of natural languages (unlike other LM-based baselines such as CodeBERT). The difficulty of this construction ("roughly two weeks") brings up several concerns: 
 	- These rules might not generalize to different programming languages (e.g. Python to C++): on this point, the authors have reported some evidence of generalization though the reviewer "kn2F" has viewed it as "low (<=41%)" and not convinced. I suggest the authors report these analyses in their main draft. I suspect these numbers need to be compared with the corresponding numbers of their end-to-end baselines. 
 	- These rules might not transferrable to different **natural** languages. The authors have acknowledged this limitation. I suggest the authors be explicit about such limitations in their work. 
 	- These rules might not generalize to longer queries; this is acknowledged in the author's response. I suggest the authors be upfront about these issues in their draft. 

 

confidence: Less certain
Decision: Accept