Out of the Bleu: How Should We Assess Quality of the Code Generation Models?

Download This Paper Open PDF in Browser Add Paper to My Library Share: Permalink Using these links will ensure access to this page indefinitely Copy URL Out of the Bleu: How Should We Assess Quality of the Code Generation Models? 19 Pages Posted: 26 Aug 2022 See all articles by Mikhail EvtikhievMikhail Evtikhievaffiliation not provided to SSRNEgor Bogomolovaffiliation not provided to SSRNYaroslav Sokolovaffiliation not provided to SSRNTimofey Bryksinaffiliation not provided to SSRN Abstract Recently, researchers developed many code generation models. As human evaluation of every model version is unfeasible, the community adopted evaluation metrics such as BLEU to approximate the results of human judgement. These metrics mostly originate from the NMT domain and there are hardly any studies on their agreement humans in code generation task. Despite that, minimal differences in the metric scores are used to claim superiority of some code generation models over the others. In this work, we study six metrics for evaluation of code generation models on two datasets and use human annotators to assess the quality of the models. The results indicate that for CoNaLa dataset none of the metrics can correctly emulate human judgement with >95% certainty if the difference in model scores is less than 5 points. For HearthStone dataset a difference in model scores of at least 2 points is enough to claim the superiority of one model over the other.Our findings suggest that the ChrF metric is a better fit for the evaluation of code generation models than the commonly used BLEU and CodeBLEU. Yet, finding a metric for code generation that closely agrees with humans requires additional work. Keywords: code generation, metrics, neural networks, code similarity Suggested Citation: Suggested Citation Evtikhiev, Mikhail and Bogomolov, Egor and Sokolov, Yaroslav and Bryksin, Timofey, Out of the Bleu: How Should We Assess Quality of the Code Generation Models?. Available at SSRN: https://ssrn.com/abstract=4201043 Mikhail Evtikhiev affiliation not provided to SSRN ( email ) No Address Available Egor Bogomolov (Contact Author) affiliation not provided to SSRN ( email ) No Address Available Yaroslav Sokolov affiliation not provided to SSRN ( email ) No Address Available Timofey Bryksin affiliation not provided to SSRN ( email ) No Address Available Download This Paper Open PDF in Browser Do you have a job opening that you would like to promote on SSRN? Place Job Opening Paper statistics Downloads 0 Abstract Views 0 PlumX Metrics Related eJournals Information Systems eJournal Follow Information Systems eJournal Subscribe to this fee journal for more curated articles on this topic FOLLOWERS 222 PAPERS 4,961 Feedback Feedback to SSRN Feedback (required) Email (required) Submit If you need immediate assistance, call 877-SSRNHelp (877 777 6435) in the United States, or +1 212 448 2500 outside of the United States, 8:30AM to 6:00PM U.S. Eastern, Monday - Friday. Submit a Paper Section 508 Text Only Pages SSRN Quick Links SSRN Solutions Research Paper Series Conference Papers Partners in Publishing Jobs & Announcements Newsletter Sign Up SSRN Rankings Top Papers Top Authors Top Organizations About SSRN SSRN Objectives Network Directors Presidential Letter Announcements Contact us FAQs Copyright Terms and Conditions Privacy Policy We use cookies to help provide and enhance our service and tailor content. To learn more, visit Cookie Settings. This page was processed by aws-apollo4 in 0.298 seconds