Venue: ICLR 2020 Conference

Reviews:

final_score: rating: 1: Reject
review_content: This paper presented a crowdsourced dataset for evaluating the semantic relatedness, similarity, and contextual similarity of source code identifiers. Motivated by similar tasks in evaluating the quality of word embeddings in the natural language, authors collected the smalls-scale evaluation datasets (less than three hundreds of code identifier pairs) and further evaluated the performance of several current embeddings techniques for codes or computing the similarity scores of code identifiers. Although I appreciated the efforts to build such evaluation dataset, I think the contributions are limited in terms of scientific contribution. 

Pros:

1) They collected a new small-scale evaluation dataset for evaluating the quality of semantic representations of various existing embedding techniques for code identifiers. 

2) They performed evaluations on these embeddings techniques for code and provided a few interesting findings based on these tasks. 

Cons:

1) The proposed datasets are very small. The total number of code pairs are less than 300 pairs out of total code identifiers 17,000, which is a very small set of the total pairs (17000 x 17000). Therefore, it is hard to fully evaluate the embeddings quality of various methods with high confidence. 

2) The whole paper is mainly about the data collection as well as a few of evaluations of several existing code embedding techniques. The scientific contributions are quite limited. It would be nice to put these efforts to have a competition of code embedding techniques and this paper could be served as a technical report on this direction. 


review_title: Official Blind Review #3

final_score: rating: 3: Weak Reject
review_content: This paper proposed a benchmark dataset for evaluating different embedding methods for identifiers (like variables, functions) in programs. The groudtruth evaluation (similar/related) are labeled via Amazon MTurk. Experiments are carried out on several word embedding methods, and it seems these methods didn’t get good enough correlation with human scores. 

Overall I appreciate the effort paid by the authors and human labors. However I have sevarl concerns:

1) why the identifier embedding is important? As pre-trained word embeddings are useful for many NLP downstream tasks, what is the scenario of identifier embedding usage?

2) To collect the human labels, are there any requirements? e.g., experiences in javascript. Especially, I’m curious why in Table 3, setMinutes and setSeconds get score of 0.22 (which is too high).

3) It would make more sense to compare with state of the art language pretraining methods, like bert, xlnet, etc. People have trained the language model with GPT2 (TabNine) that works well with code. So to make the work more convincing, I would suggest to include these.

review_title: Official Blind Review #1

final_score: rating: 6: Weak Accept
review_content: The paper introduces a new dataset that includes manually labelled identifier name pairs. The labels determine how much the corresponding embeddings are useful for determining their meaning in context of code, a setting that has sufficient differences from a natural language.

A significant part of the paper is devoted on data cleaning and relating the computed metrics to similar efforts in natural language processing. While there is not much novelty in this part of the paper, it is doing a good job at addressing many possible questions on the validity of their dataset. Another important aspect that is covered by the work is different kinds of similarities of identifier names - similarity corresponding to having the same or similar type in a precise type system or similarity corresponding to being synonyms. Having several of these dimensions would make the results applicable for a wide range of applications of identifier name embeddings.

While not introducing new concepts, this paper is important for the community, because it has the potential to change the way embedding computation is done for “Big Code” problems. Right now, most papers either introduce their own embeddings, or use non-optimal ones like Code2Vec.

The paper also has a surprising finding that even techniques designed for code are in some cases not as good as the FastText embeddings. This is an interesting result, because few other works include this kind of embedding in their experiments. Furthermore, the paper deep dives into the strong and weak sides of several solutions and shows that there is a large opportunity to improve on the existing results.

review_title: Official Blind Review #2

Meta-review: This paper presents a dataset to evaluate the quality of embeddings learnt for source code. The dataset consists of three different subtasks: relatedness, similarity, and contextual similarity. The main contribution of the paper is the construction of these datasets which should be useful to the community. However, there are valid concerns raised about the size of the datasets (which is pretty small) and the baselines used to evaluate the embeddings -- there should be a baselines using a contextual embeddings model like BERT which could have been fine-tuned on the source code data. If these comments are addressed, the paper can be a good contribution in an NLP conference. As of now, I recommend a Rejection.
Decision: Reject