Venue: ICLR 2020 Conference

Reviews:

final_score: rating: 8: Accept
review_content: The paper proposes an embedding method for source code tokens, which is based on contextual word representation, particularly is based on the method of ELMo. The learned representation is evaluated on the task of bug detection, with promising performance.

Strengths:
The paper addresses an important and impactful problem. The solution designed for this problem seems very reasonable. Experiments are useful and reasonable and the experimental results are promising and in the favor of the paper.
The paper is well written and clear.

Weaknesses:
- The data used (in particular the method of buggy code generation applied) seems very specific.  It would be interesting to know the performance of the method on real bugs. 
- The paper is a bit low in technicality. 

Decision: Accept
I think this paper is overall a good work and can open direction of research even beyond the scope of the paper, for example  in combining learning and reasoning, or in source code generation with adversarial models.

Minor: 
- Since compilers can spot errors in code completely, it would be useful to motivate the advantage of learning for bug detection
- The table referrals in the body of the paper contains wrong table numbers in Sections 6.1, 6.2, 6.3.
- The incorrect Binary Operator example in Listing 2 does not seem to be a well justified bug. It could be a correct piece of code for a different purpose.
- which use -> which we use
review_title: Official Blind Review #2

final_score: rating: 3: Weak Reject
review_content: This paper leverage recent advances of ELMo in context embedding and apply it in the source code embedding. With the help of ELMo, source embedding can take the three benefits: (1)  Surrounding names provide indirect information about possible values the variable could take; (2) an variable’s value evolves through the program execution can be captured; (3) open a gate for the reuse of the ptr-trained model. To evaluate the effectiveness of the proposed approach, authors conduct experiments on the downstream task of the bug detection. 
Pros:
1. This work study an interesting problem, which is challenging to solve.
2. The application and combination of different techniques in this paper are smart.
3. The experiment results show better performance of contextual embedding based method compared with non-contextual embedding based methods.
Cons:
1. It is a good application of known techniques, but the novelty is limited.
2. It is suggested to evaluate the effectiveness of the proposed approach on various source code analysis task such as variable misuse.
3. It is suggested to compare with other state-of-the-art baseline methods, e.g. BERT.
4. In the end of the introduction section, the authors claim that "we release our implementation and representation...". However, implementation, representation and dataset are missing.
review_title: Official Blind Review #3

final_score: rating: 3: Weak Reject
review_content: The paper proposes to use ELMO embeddings to improve the precision on the first step of the DeepBugs tasks defined by Pradel and Sen (2018). This first step is an artificial problem created by taking real programs (with and without bugs, but assuming almost all of them are correct) and introducing bugs of certain type into the programs. Then, a classifier needs to distinguish between the real and the artificial set. This classifier is then to be used as a checker for anomalies in code and the anomalies are reported as bugs, however the paper skips this second step and only reports results on the first classification problem.

Technically, the paper improve this first step of DeepBugs by using a standard variant of ELMO. The evaluation is detailed, but the results are unsurprising. The paper simply tech-transfers the idea from NLP to Code. If this work is accepted at the conference, I cannot imagine an interesting presentation or a poster that simply cites the changed numbers. Did we expect ELMO to be worse than more naive or random embeddings?

The work and its results heavily peg on the DeepBugs and increases the precision of its first step by a significant margin, but does not show getting any more useful results.  In fact, on one task (Wrong Binary Operator), SCELmo gets to 100% accuracy. This means it will never report any bugs, whereas DeepBugs seems to be performing best on exactly this kind of reports with its weaker model.

I would recommend the authors to either work on showing practical usefulness of the technique, showing something for the full bugfinding task (not merely the first, artificial part), or to investigate if (or how) the idea to add bug-introducing changes to a code dataset is conceptually flawed for bugfinding (as this idea is widely used by several other works like Allamanis et al 2018b or by https://arxiv.org/abs/1904.01720 which also don't get to practical tools ). There seems to be some indication of this by the reported 100% accuracy, but right now this remains completely uninvestigated. 

Minor issues:
Listing 3: Opernad -> Operand
Page 5. There is no Table 6.1

review_title: Official Blind Review #1

Meta-review: This paper improves DeepBugs by borrowing the NLP method ELMo as new representations. The effectiveness of the embedding is investigated using the downstream task of bug detection. 

Two reviewers reject the paper for two main concerns:
1 The novelty of the paper is not strong enough for ICLR as this paper mainly uses a standard context embedding technique from NLP.
2 The experimental results are not convincing enough and more comprehensive evaluation are needed. 

Overall, this novelty of this paper does not meet the standard of ICLR.

Decision: Reject