Package: code-bert-score
CodeBERTScoreThis is the official implementation of the paper:Shuyan Zhou, Uri Alon, Sumit Agarwal, Graham Neubig,CodeBERTScore: Evaluating Code Generation with Pretrained Models of CodeCodeBERTScore is an Automatic Evaluation Metric for Code, based onBERTScore.
This repository is based on the code ofBERTScore, and we are grateful to the authors for releasing their code.April 2023- CodeBERTScore is now available onpypi, which means that you can simplypip install code-bert-score!ExampleHow does it work?UsageAdditional FeaturesBackend ModelTrainingEvaluationHuman EvaluationFunctional CorrectnessCitationExample:Figure (a) shows a reference code snippet in Java. Figures (b) and (c) show two generated predictions. Among these two candidates and given the reference, BLEU prefers (scores higher) the code in (b), which is not functionally equivalent to the reference, while CodeBERTScore prefers the code in (c), which is functionaly equivalent to the reference.How does it work?As BERTScore, CodeBERTScore leverages the pre-trained contextual embeddings from a model such as CodeBERT and matches
words in candidate and reference sentences by cosine similarity.
Differently from BERTScore, CodeBERTScore also encodes natural language input or other context along with the generated code, but does not use that context to compute cosine similarities.This example shows how CodeBERTScore can compute the similarity between the Python expressionsx ** 0.5andmath.sqrt(x), which are functionally equivalent, even though they have very few overlapping tokens.Usageimport code_bert_score
pred_results = code_bert_score.score(cands=predictions, refs=refs, lang='python')Wherepred_resultsis a 4-tuple of(precision, recall, F1, F3), where each is a 1-D tensor of scores for each prediction-reference pair.F3is similar to the well-knownF1score, that considers recall 3 times as important as precision. See thedefinition on Wikipedia.See ourexample.pyscript. Additional details are shown in the original BERTScoredemo notebook.Huggingface 🤗 ModelsWe fine-tuned themicrosoft/codebert-base-mlmmodel for 1,000,000 steps (withbatch_size=32) on several languages separately.We released the following models to the Huggingface hub:neulab/codebert-python(the default model forlang='python')neulab/codebert-javascript(the default model forlang='javascript'or'js')neulab/codebert-c(the default model forlang='c')neulab/codebert-cpp(the default model forlang='cpp'or'c++')neulab/codebert-java(the default model forlang='java')The appropriate model will be loaded automatically when passing thelangargument to thescore(..)function, for example:lang='python'.
For other uses, these models can be loaded using (for example):fromtransformersimportAutoTokenizer,AutoModelForMaskedLMtokenizer=AutoTokenizer.from_pretrained("neulab/codebert-python")model=AutoModelForMaskedLM.from_pretrained("neulab/codebert-python")Additional FeaturesWe found that in NL->Code problems, more accurate results are achieved by encoding the NLsourceswith the code prediction, but then measuring similarity only for the encoded code:pred_results = code_bert_score.score(cands=predictions, refs=refs, lang='python', sources=sources)We also found that using Inverse Document Frequencies improve the results, similarly to the original BERTScore. We included an example script that shows how to precompute them herecompute_idf.py. Then, the resulting dictionary can be used with the argumentidf=idf_dict.
Our IDF dicts can be found in./idf_dicts/.Tuning the layer that the similarity is computed from is also helpful, usingnum_layers=NwhereNis between 5-10:We found that more accurate results are achieved by encoding theentireinputs, but measures the similarity only between non-punctuation and non-whitespace tokens. To disable the removal of punctuation toksn, useno_punc=False.See also ourexample.pyscript. Additional details are shown in the original BERTScoredemo notebook.TrainingTherun_mlm.pyscript can be used to fine-tune the base modelmicrosoft/codebert-base-mlmon specific languages.EvaluationThe code to reproduce the results in the paper can be found in theevaluation.Human EvaluationWe find that CodeBERTScore is more correlated with human preference compared to a variety of common metrics. See more details in thepaper.Functional CorrectnessWe find that CodeBERTScore is more correlated with functional correctness compared to a variety of common metrics. See more details in thepaper.Citation@article{zhou2023codebertscore,
  url = {https://arxiv.org/abs/2302.05527},
  author = {Zhou, Shuyan and Alon, Uri and Agarwal, Sumit and Neubig, Graham},
  title = {CodeBERTScore: Evaluating Code Generation with Pretrained Models of Code},  
  publisher = {arXiv},
  year = {2023},
}