Package: codebleu
CodeBLEUThis repository contains an unofficialCodeBLEUimplementation that supportsLinux,MacOS(incl. M-series) andWindows. It is available throughPyPIand theevaluatelibrary.Available for:Python,C,C#,C++,Java,JavaScript,PHP,Go,Ruby.The code is based on the originalCodeXGLUE/CodeBLEUand updated version byXLCoST/CodeBLEU.  It has been refactored, tested, built for macOS and Windows, and multiple improvements have been made to enhance usability.Metric DescriptionAn ideal evaluation metric should consider the grammatical correctness and the logic correctness.
We propose weighted n-gram match and syntactic AST match to measure grammatical correctness, and introduce semantic data-flow match to calculate logic correctness.[fromCodeXGLUErepo]In a nutshell,CodeBLEUis a weighted combination ofn-gram match (BLEU),weighted n-gram match (BLEU-weighted),AST matchanddata-flow matchscores.The metric has shown higher correlation with human evaluation thanBLEUandaccuracymetrics.InstallationThis library requiressofile compilation with tree-sitter, so it is platform dependent.Currently available forLinux(manylinux),MacOSandWindowswith Python 3.8+.The metrics is available aspip packageand can be installed as indicated above:pipinstallcodebleuor directly from git repo (require internet connection to download tree-sitter):pipinstallgit+https://github.com/k4black/codebleu.gitUsagefromcodebleuimportcalc_codebleuprediction="def add ( a , b ) :\nreturn a + b"reference="def sum ( first , second ) :\nreturn second + first"result=calc_codebleu([reference],[prediction],lang="python",weights=(0.25,0.25,0.25,0.25),tokenizer=None)print(result)# {#   'codebleu': 0.5537,#   'ngram_match_score': 0.1041,#   'weighted_ngram_match_score': 0.1109,#   'syntax_match_score': 1.0,#   'dataflow_match_score': 1.0# }wherecalc_codebleutakes the following arguments:refarences(list[str]orlist[list[str]]): reference codepredictions(list[str]) predicted codelang(str): code language, seecodebleu.AVAILABLE_LANGSfor available languages (python, c_sharp c, cpp, javascript, java, php, go and ruby at the moment)weights(tuple[float,float,float,float]): weights of thengram_match,weighted_ngram_match,syntax_match, anddataflow_matchrespectively, defaults to(0.25, 0.25, 0.25, 0.25)tokenizer(callable): to split code string to tokens, defaults tos.split()and outputs thedict[str, float]with following fields:codebleu: the finalCodeBLEUscorengram_match_score:ngram_matchscore (BLEU)weighted_ngram_match_score:weighted_ngram_matchscore (BLEU-weighted)syntax_match_score:syntax_matchscore (AST match)dataflow_match_score:dataflow_matchscoreAlternatively, you can usek4black/codebleufrom HuggingFace Spaces (codebleupackage required):importevaluatemetric=evaluate.load("dvitel/codebleu")prediction="def add ( a , b ) :\nreturn a + b"reference="def sum ( first , second ) :\nreturn second + first"result=metric.compute([reference],[prediction],lang="python",weights=(0.25,0.25,0.25,0.25))Feel free to check the HF Space with online example:k4black/codebleuContributingContributions are welcome!If you have any questions, suggestions, or bug reports, please open an issue on GitHub.Make your own fork and clone it:gitclonehttps://github.com/k4black/codebleuFor development, you need to install library (for so file to compile) withtestextra:(require internet connection to download tree-sitter)python-mpipinstall-e.[test]python-mpipinstall-e.\[test\]# for macosFor testing just run pytest:python-mpytestTo perform a style check, run:python-misortcodebleu--check
python-mblackcodebleu--check
python-mruffcodebleu
python-mmypycodebleuLicenseThis project is licensed under the terms of the MIT license.CitationOfficialCodeBLEU papercan be cited as follows:@misc{ren2020codebleu,title={CodeBLEU: a Method for Automatic Evaluation of Code Synthesis},author={Shuo Ren and Daya Guo and Shuai Lu and Long Zhou and Shujie Liu and Duyu Tang and Neel Sundaresan and Ming Zhou and Ambrosio Blanco and Shuai Ma},year={2020},eprint={2009.10297},archivePrefix={arXiv},primaryClass={cs.SE}}