Venue: NeurIPS 2020 Workshop CAP

Reviews:

final_score: rating: 7: Good paper, accept
review_content: This paper studies the quality of code translation using a blackbox deep neural network (DNN) model based on the same principles as DNNs used for machine translation. The key challenge is that evaluating the performance of these models in terms of the quality of the code generated (i.e., beyond correctness) is challenging.

To address this issue, they perform a user study to gain qualitative insights into the performance of the DNN, as well as a quantitative evaluation using a linter to diagnose both style issues as well as code errors. I thought the user study was particularly interesting, and the quantitative insights highlight some of the key challenges in applying machine learning to the programming domain (and possibly more broadly).

Their key findings include issues such as the presence of “obvious” mistakes, including high confidence ones. They also find that the DNN is not good at producing code in a certain style, though it might be possible to fine-tune it to do so.

Overall, I think the paper studies an important research problem and identifies some interesting insights.

review_title: Interesting insights into a deep learning model for code translation

Meta-review: The review being positive, I am recommending acceptance.
Decision: Accept