CodeBERTScore: Evaluating Code Generation with Pretrained Models of Code
Authors: Shuyan Zhou
Uri Alon
Sumit Agarwal
Graham Neubig
Venue: EMNLP 2023 Main (2023)
Keywords: nl2code
code generation
code
evaluation
codebert
bertscore
TL;DR: An extensive evaluation of BERTScore for NL->code evaluation

Abstract: Since the rise of neural natural-language-to-code models (NL$\rightarrow$Code) that can generate long expressions and statements rather than a single next-token, one of the major problems has been reliably evaluating their generated output. 
In this paper, we propose CodeBERTScore: an evaluation metric for code generation, which builds on BERTScore (Zhang et al., 2020).
Instead of encoding only the generated tokens as in BERTScore, CodeBERTScore also encodes the natural language input preceding the generated code, thus modeling the consistency between the generated code and its given natural language context as well.    We perform an extensive evaluation of CodeBERTScore across four programming languages. We find that CodeBERTScore achieves a higher correlation with human preference and with functional correctness than all existing metrics. That is, generated code that receives a higher score by CodeBERTScore is more likely to be preferred by humans, as well as to function correctly when executed.
We release five language-specific pretrained models to use with our publicly available code.
Our language-specific models have been downloaded more than **1,000,000** times from the Huggingface Hub. Our code and data are available at https://github.com/neulab/code-bert-score

CodeBERTScore:
Evaluating Code Generation with Pretrained Models of Code
Shuyan Zhou' Uri Alon"t Sumit Agarwal Graham Neubig
Language Technologies Institute, Carnegie Mellon University
{ (shuyanzh, ualon, sumita, gneubig (écs cmu.eda
Abstract
developers' time when implemented in tools such as GitHub's Copilot. This sharp rise LLMs' us- ability was achieved thanks to their ability to ac- curately generate long completions, which span multiple tokens and even lines, rather than only single next-token in early models (Allama- nis and Sutton, 2013; Movshovitz-Attias and Co- hen, 2013). Nevertheless, evaluating and compar- ing different models has remained challenging problem (Xu al., 2022) that requires an accurate and reliable evaluation metric for the quality of the models' generated outputs, and existing met- rics are sub-optimal.
Since the rise of neural natural-language-to- code models (NL->odde) that can generate long expressions and statements rather than a single next-token, one of the major prob- lems has been reliably evaluating their gen- erated output. In this paper,
propose CodeBERTTcore: an evaluation metric for code generation, which builds on BERTScore (Zhang al., 2020). Instead of encoding only the generated tokens as in BERTScore, CodeBERTScore also encodes the natural lan- guage input preceding the generated code, thus modeling the consistency between the gener- ated code and its given natural language text as well. We perform an extensive eval- uation CodeBERTScore across four pro- gramming languages. We find that Code- BERTScore achieves a higher correlation with human preference and with functional correct- ness than all existing metrics. That is, gener- ated code that receives higher score by Code- BERTScore is more likely to be preferred by humans, well as to function correctly when executed. release five language- pretrained models available code. Our language-specific models have been downloaded more than 1,000,000 times from the Huggingface Hub.!
Existing evaluation approaches The most common evaluation metrics are token-matching methods such as BLEU (Papineni al., 2002),
adopted from natural language processing. These metrics are based on counting overlapping grams the generated code and the reference code. CrystalBLEU (Eghbali and Pradel, 2022) extends BLEU by ignoring the 500 most oc- curring n-grams, arguing that they are trivially shared between the prediction and the reference. Nonetheless, both BLEU and CrystalBLEU rely the lexical exact match of tokens, which does not account for diversity in implementation, variable names, and code conventions. Figure I shows an example: given the reference code in Figure 1(a), both BLEU and CrystalBLEU prefer (rank higher) the non-equivalent code in Figure 1(b) over the functionally equivalent code in Figure 1(c).
with our publicly
1 Introduction
Naturalllanguuag-tooooooo generation (NL-~oode) has seen sharply growing popularity recently due to the emergence of large language models (LLMs) trained on vast amounts of natural lan- guage and code (Chen al., 2021; Fried et al., 2022; Zhou et al., 2023; Austin al., 2021; Al- lal al., 2023). LLMs have reached such high NL-+Code accuracy that they are now useful for the broad programming audience and actually save
CodeBLEU (Ren et al., 2020) attempts to lower the requirement for lexical exact match, by rely- ing data-flow and Abstract Syntax Tree (AST) matching well; nevertheless, valid generations may have different ASTs and data flow from the reference code, which may lead to low CodeBLEU score even when the prediction correct. Further, partial predictions may be useful for a program-
"Equal contribution
Now Google DeepMind
'The code and data are available at https://gthuh..cm/
neulab/cnde-hert

---

Reference:
int [Obbest target) int D;
for (Object elem: this.elements) relem.equal (target)) return 1;
return -1;
(a) The ground truth reference find the index of target in this. cements.
Non-equivalent candidate:
Equivalent candidate:
boolean (Object target)
int = [Object target)
for (Object elem: this. chis.elements) if (elem.equals (target)) return true;
for (int i-0; isthis. sechisseeementtssieee)) i+4) C
object elem eeiiseeements.gert (i);
if (elem.equa (targst)) { return i;
return false;
return -1;
(b) Preferred by BLEU & CrystalBLEU find whether or not rarg is in this.e ss.
(c) Preferred by CodeBERTScore the index of target this.el eleentt.
An intuitive example for the usefulness of CodeBERTScore in measuring generated code: Figure I(a) shows a reference code snippet in Java. Figure 1(b) and Figure I(c) show two generated predictions. Among these candidates and given the reference, both BLEU and CrystalBLEU prefer (score higher) the snippet in Figure 1(b), which is not functionally equivalent to the reference. while our proposed CodeBERTScore prefers the code Figure (c)), which is functionally equivalent code Figure (aa).
Figure I:
mer, but accepting them may lead to partial code that does not parse, and thus cannot be fully eval- uated by CodeBLEU (for example, predicting the first line of for loop, without the loop's body). Execution based evaluation attempts to address these problems by running tests on the generated code to verify its functional correctness (Chen et al., 2021; Athiwaratkun et al., 2022; Li et al., 2022; Wang et al., 2022; Lai et al., 2022). This provides direct measure of the functionality of the generated code while being agnostic to di- versity in implementation and style. However, execution-based evaluation requires datasets that are provided with Hand-written test cases for each example, which is costly and labor-intensive to create; thus, only few such datasets exist. Addi- tionally, executing model-generated code is sus- ceptible to security threats, and thus should be run in an isolated sandbox, which makes it technically cumbersome work with iteratively.
et al.. 2020). First, CodeBERTScore encodes the generated code and the reference code inde- pendently with pretrained models, with the inclu- sion of natural language instructions comments. Then, we compute the cosine similarity between the encoded representations of each token in the generated code and each token the reference code. Finally, the best matching token vector pairs are used to compute precision and recall. Code- BERTScore allows comparing code pairs that are lexically different while taking into account the (1) programmatic- natural- natural-language- waal---annngg-nneeeee if such provided; the (2) contextual information of each token; and (3) implementation diversity. Our approach illustrated in Figure 2.
Example concrete example is shown in Fig- ure I: while BLEU and CrystalBLEU prefer (rank higher) the non-equivalent code in Figure 1(b) given the reference code in Figure 1(a), Code- BERTScore prefers the code in Figure 1(c), which is functionally equivalent to the reference (Fig- ure (a)).. We note that in this example, the vari- able names are identical across all three code snip- pets. When the variable names of the reference are different than the candidate's 's, is even harder
Our approach In this work, introduce Code- BERTScore, an evaluation metric for code genera- tion, leveraging self-super pretrained models code such as CodeBERT (Feng et al., 2020), and adopting best practices BERTScore (Zhang

---

Natural Language Instruction # Find the square of
Pairwise Cosine Similarity (only between nen-panctuation code tokens)
Similarity Matrix
Generated Code
Reference Code
Encode
Reterence Code
Cod:BERTScorc
0.5
Procision
CodcBERT
Cod:BERTScor: Recall
Generated Code
math. sqrt (x)
CodeBERT
0.5
CodeBERTScore F-ADurn
Figure 2: A diagram illustrating CodeBERTScores We use a language-specific CodeBERT model to encode cach (natural language, reference _oteeee_oeeeeeeee and (natural language. generated .. We then compute the pairwise cosine similarity between every encoded token in the reference and every encoded token in the generated code, ignoring the encoded natural language context tokens and encoded punctuation tokens; finally, we take the max across the rows of the resulting matrix compute Precision and across columns compute Recall.
for token-matching approaches such as BLEU and CrystalBLEU to compare the reference with the candidates, while CodeBERTSScore can trivially match variable names according their semantic similarity and their functional role in the code.
didates is more important than the absolute value of (ū. That is, ideally, if a prediction ő1 is more functionally equivalent to y² and more preferable by human programmers over predic- tion Ú2, wish that good metric would rank ý higher than 82. That is, we seek an function such that fūr v²) > f(9..9 )*).
Contributions In summary, our main contribu- tions are: (a) CodeBERTScore: a self-supervised metric for NL-Code evaluation, based on BERTScore, which leverages the benefits of pre- trained models, while not requiring labeling or manually annotated data. (b) An extensive em- pirical evaluation across four programming lan- guages, showing that CodeBERTScore is more correlated with human preference and more cor- related with execution correctness than all pre- vious approaches including BLEU, CodeBLEU, and CrystalBLEU. (c) We pretrain and release five language-specific CodeBERT models to use with publicly available code, for Java, Python, C, C++. and JavaScript. As of the time of this sub- mission, our models have been downloaded from the Huggingface Hub more than 1,000,000 times.
2.2 Background: BERTScore
BERTScore (Zhang et al., 2020) was proposed as method for evaluating mainly machine transla- tion outputs. The idea in BERTScore is to en- code the candidate sentence (the prediction) and the reference sentence (the ground truth) sepa- rately, using BERT-based model, which encodes each sequence of tokens as a sequence of vectors. Then, BERTScore computes the cosine similarity between every vector from the candidate sequence and every vector from the reference sequences.
Given these similarity scores, BERTScore com- putes sentence-level precision by taking the max- imum similarity score for every candidate vec- tor and averaging, and computes recall by tak- ing the average of the maximum similarity scores for every reference vector. Intuitively, a high ScoTTccor--lelll is obtained, for example, if ev- ery vector from the reference sentence has at least one vector from the candidate sentence that is highly sisile- to it; high BERTScore- precision is obtained if every vector from the can- didate sentence is highly cosine-similar one vector from the reference sentence. Ulti- mately, the final score is the F1 score, computed as the harmonic mean of precision and recall.
2 Evaluating Generated Code
2.1 Problem Formulation
Given a context E X' (e.g., a natural language instruction or comment), code generation model M X + y produces a code snippet ý E y by conditioning on the intent specified by x. The quality of the generation is evaluated by compar- ing ÿ E ) with the reference implementation y" Y, using metric function y essentially computing fØ. w.).
R.
least
larger value of f(j 3y") indicates that the gen- crated code is more accurate with respect to the reference code, and the way ff ranks different can-

---

CodeBERTSScoee
max sim (vi 9)) of Ey [m=]
(4)
[
CodeBERRTScoree
max sim (2 Uz)
(5)
8y eD ml
21 Cy/"((m-
CodeBERITccorep CodeBERTSSoore
CodeBERTTycorer,
(6)
CodeBERRTScorep + CodeBERTScoree
10 CodeBERTSSorcpp CodeBERTScorcp
CodeBERTScoreey
(7)
9 SodrBERTScorep + CodeBERTSScorgp
Figure 3: Main equations for CodeBERTScore
2.3 CodeBERTScore
enized sequence, resulting in sequences vectors:
Our approach generally follows BERTScore, with the following main differences:
B (K. w.n)) (0 Uc.....Ek.... Find yn) (2)
B ((11, ......................... ...))...)))) - (k
1. We encode the context (the natural language instruction or comment) along with each of the generated and reference code snippets, but without using the encoded context in the final similarity computation, essentially com- puting fū, y",""" .x) rather than f(i, y²)
Finally, mask out the encoded context tokens ; 2t as well all non-alphamumerie tokens (parentheses, brackets, dots, commas, whites- paces, etc.) except for arithmetic operators, from each of the encoded reference and encoded can- didate. This results in encoded reference tokens y" = ( Fm) encoded candidate tokens ò = (D,,..., Dn). and their corresponding masks m and m. We denote y [mm] as the remaining encoded tokens in y after selecting only alphanumeric to- ken vectors according to the mask
2. Given the precision and recall, instead of computing the F1 score, also compute F3 to weigh recall higher than precision, follow- ing METEOR (Banerjee and Lavie, 2005).
3. As our underlying BERT-like model, we use programming Lunguage-specingg models that we pretrain and release, rather than models that were intended for natural language only.
Similarity Computation We compute the co- sine similarity between the encoded reference and candidate tokens, following Zhang al. (2020):
We use a BERT-like pretrained model to en- code the reference and candidate. In our exper- iments, B is a CodeBERT model that we fur- ther pretrained using the masked language mod- eling objective (Devlin et al., 2019) language- specific corpora, but B be any transformer- based model which we have access to its internal hidden states.
y
yj |1,,1
sim 05)
(3)
13
Although this compares the individual tokens yf and Új. their vector representations yf and Ú contain information about their context, and thus about their semantic role in the code.
We concatenate the con- text r with each of the reference and the candidate, resulting in x y and æ ÿ. We use the tokenizer To provided with the model B:
Token Representation
CodeBERTScore We use the similarity matrix (see Figure 2), formed by the similarity scores be- tween y² and ÿ, to compute precision, recall, and F,, by taking the maximum across the rows and columns of the similarity matrix, and then aging. Following Banerjee and Lavie (2005), we also compute F33 by giving more weight recall, as shown in Figure Additional details regarding token weighting and scaling are provided in Ap- pendix A.
To (x) (Ii, Tk, Vi .... Vmm) To (x (21,-.., Th, 01, Un)
(1)
to get a sequences of tokens. We run a standard "forward pass" with the model B for each tok-

---

3 Experimental Setup
Correlation metries We used three major cor- relation metrics. Following best practices in natu- ral language evaluation, we used Kendall-Tau (T), Pearson (rp) and Spearman (r:) to measure the correlation between each metric's scores and the references, The detailed equations can be found in Appendix C.
We evaluate CodeBERTScore across multiple datasets and programming languages. We first show that CodeBERTScore is more correlated with human preference than previous metrics, us- ing human-rated solutions for the CoNaLa dataset (Yin et al., 2018a; Evtikhiev al., 2022). We then show that CodoBERTScore more correlated with functional correciness, using the HumanEval dataset (Chen et al., 2021). We also show that CodoBERTScore achieves higher newly pro- posed distinguishaaiiltyy than other metrics (Ap- pendix F). Finally, we analyze some of the design decisions and their implications.
Human preference experiments We evaluate different metrics on CoNaLa (Yin al., 2018b), a natural language to Python code generation bench- mark collected from StackOverflow. We use the human annotation of Evtikhiev et al. (2022) to measure the correlation between each metric and human preference. More details are provided in Appendix B.1.
3.1 Training Language-specific CodeBERT models
Functional correctness experiments We evaluate functional correctness using the Hu- manEval (Chen et al., 2021) benchmark. Each example in HumanEval contains a natural language goal, hand-written input-output test cases, and human-written reference solution. While the original HumanEval in Python, Cassano et al. (2022) translated HumanEval to 18 programming languages, and provided the predictions of the Codex model (Chen et al., 2021) socde-davinci--e and their corresponding functional correctness.* We used Java, C++, Python, and JavaScript for these experiments, which some of the most popular programming languages in open-source projects, More details are provided in Appendix B.2.
Training We used CodeBERT (Feng et al., 2020) as our base model (B) and continued its self- supervised pretraining (Gururangan et al., 2020) with the masked language modeling (MLM) ob- jective (Devlin et al., 2019) on Python, Java, C++, C, and JavaScript corpora. We trained sepa- rate model for each programming language, for 1,000,000 steps for each language, using batch size of 32, an initial learning rate of 5e decayed linearly 3c 5_ Our implementation based on the widely used HuggingFace Trans formers library (Wolf et al., 2019) and BERTScore", and it supports any transformer-bast based model available on the HuggingFace hub.
Dataset We trained each model on the language- specific subset of the CodeParrot (Tunstall et al., 2022) dataset?, which consists of overall 115M code files from GitHub, further filtered by keep- ing only files having average line length lower than 100, more than 25% alphanumeric characters, and non-auto-generatee files. Even after 1,000,000 training steps, none of the models have completed even a single epoch, meaning that every training example was seen only once at most.
Hyperparameterss We tuned only the following hyperparameters for CodeBERTScore: whether to F, or F3, and which layer of the underlying model extract the encoded tokens from, which we examine in Section 5. We used Fj in the hu- man preference experiments and F3 in the func- tional correctness experiments. We perform 3-fold cross-validation and report average results across the three folds. As for the layer to extract the to- ken vectors from, we used layer 7 for CoNaLa, and in HumanEval we used layer for Java, 10 for C++, 11 for JavaScript, and 9 for Python.
3.2 Comparing Different Metrics
We compare CodeBERTScore with existing met- rics that are commonly used on code generation evaluation. We use human annotated preference and execution-based results as the ground truth and measure their correlation with these metries.
Results
Correlation with human preference Table 2 shows the correlation between different metrics
http:://ggitubb.oom/TTig//ee/ere withub-ond-cllam
"htpss//huggingloee.ooaaaatt col/attsets/mupp//tuull..EEEEE "https///nctoversc wwthss..gggtuuuccce ctthub.com/2022) top- programming languages languggss

---

Java
C++
Python
JavaScript
Metric
Fs
F
BLEU
.481
.361
.112
.301
.393
.352
.288
.343
CodeBLEU
.496
.324
.175
.201
.366
.326
.261
.299
ROUGE-1
.166
.318
.262
.260
.368
.334
.279
.280
ROUGE-2
.525
.315
.270
.273
.365
.322
.261
.292
ROUGE-L
.508
.344
.258
.288
.338
.350
.271
.293
METEOR
.558
.383
.301
.321
.418
.402
.324
.415
chrF
.532
.319
.399
.321
.394
.379
.302
.374
CrystalBLEU
.471
.273
.046
.095
.391
.309
.1 .118
.059
CodeBERTScore
.553
.369
.327
.393
.422
.415
.319
.402
Table 1: Kendall-Tau (r) and Spearman (r₂)) correlations of each metric with the functional correctness on Hu- manEval in multiple languages. The correlation coefficients are reported as the average across three runs. Standard deviation is provided Table 3.
Metric
Tp
achieves the highest or comparable Kendall- -Tau and Spearman correlation with functional correct- ness across all four languages. METEOR achieves comparable correlation with CodeBERTScore Java and JavaScript, and its correlation surpris- ingly better than other baseline metrics. However, in C++ and Python, CodeBERTScore is strictly better. Overall on average across languages, Code- BERTScore is more correlated with functional correctness than all baselines.
BLEU
.374
.604
.543
CodeBLEU
.350
.539
.495
ROUGE-1
.397
.604
.500
ROUGE-2
.429
.629
.588
ROUGE-L
.420
.619
.574
METEOR
.366
.581
.540
chrF
.470
.635
.623
CrystalBLEU
.411
.598
.576
CodeBertScore
.517
.674
.662
Analysis
Table 2: The Kendall- (r), Pearson (ry)) and Spear- man (r,) correlation with human preference. The best performance bold. The correlation coefficients are reported as the average across three runs. Standard de- viations are provided Table 4.
We conducted series of additional experiments to understand the importance of different design decisions, and to gain insights applying Code- BERTScore
datasets and scenarios.
Can we use CodeBERTScore in a new lan- guage without language-specific CodeBERT? In all experiments in Section 4, we used the language-specific model which continued pretrain on each language. But what if wish to use CodeBERTScore in language in which we don't 't have a language-specific model? We com- pare the language-specific models to CodeBERT. base Figure 4. Generally, CodeBERT-base achieves close performance to a language-speciff model. However, in most HumanEval experi- ments and correlation metries, using the language- specific model is beneficial. These results show that language-specific models are often preferred if such models are available, but the CodeBERT- base can still provide close performance even without Language-specift pretraining.
and human preference. CodeBERTScore achieves the highest correlation with human preference, across correlation metrics. While Evtikhiev et al. (2022) suggested that chrF and ROUGE-L are the most suitable metrics for evaluating code generation models in CoNaLa, CodeBERTScore outperforms these metrics by significant mar- gin. For example, CodeBERTScore achieves Kendall-Tau correlation of 0.517 compared to 0.470 of chrl' and 0.420 of ROUGE-L. These re- sults show that generated code that is preferred by CodeBERTScore. also tends to be preferred by human programmers.
Correlation with functional correctness Ta- ble shows the correlation between different met- rics and functional correctness: codeBERTScore

---

---aaii-ppecific
r-Base model
D
r.-Buse model
0.7
0.9G 0.65
0.6
0.56
0.32
0.52
0.5
0.46
0.4
0.39
0.37
0.37
O
0.33
0.3
0.2
Co
HumanFval- PPythnn
HumanFval-Jaiie
HumanFval-C*+
HaamanFval-JavaSeeee
Figure 4: The Kendall-Tau and Spearman on development set different datasets with the language-specific pretrained model (Lang-specific) and with the base CodeBERT (Base model).
0.45
tion, while higher layers encode deeper semantic meaning in natural language.
Does encoding natural language context help? One major difference between CodeBERTScore and BERTScore is that CodeBERTScore lever- ages the context for the generated code, such the natural language instruction or intent that was given input for generation. We find that us- ing context increases the correlation, for example, the Kendall- of CodeBERTSccrr from 0.50 to 0.52. While this paper mainly focuses on natu- ral language instructions, we believe that Code- BERTScore can thus benefit other programming scenarios well, for example when generating code given the human-written comments, gen- erating code given the preceding code context.
0.4
0.35
Java
0.3
C++
JavaScript
Python
0.25
0
2
12
Layer
Figure The average Kendall-Tau and Spearman the development set of HumanEval when using the embeddings from different layers.
CodeBERTScore allows soft matching of tokens The heatmaps in Figure show the sim- ilarity scores between tokens in CodeBERTScore. For example, both shutil. rmtree and rmdir in Figure 6(a) delete folder; CodeBERTScore aligns each token to respective token in the other expression, spans do not share many identical tokens.
Which transformer layer should we use? We further investigate the impact of using hidden states from different layers of the model the layer which the vectors in Equation (2) come from, in the computation of CodeBERTScore. The re- sults are shown in Figure 5: generally, the deeper the layer - the higher the average correlation be- tween CodeBERTScore and functional correct- ness, across all programming languages. However in almost all languages, performance reaches its maximum before the last layer, and decreases at the following layers. This suggests that higher lay- ers encode the semantic information of each token more accurately, but the final layers may be more task-specific. These observations are consistent with Tenney et al. (2019), who found that lower layers in BERT tend to process shallow informa-
though the two
In Figure 6(b), both code snippets calculate a square root, where one uses math. sqrt (x) and the other uses x 0.5. An exact surface- form-matching metric such as chrF would assign a low similarity score to this code pair, as they only share the token x. However, CodeBERRTScore as- signs non-zero scores to each token with meaning- ful alignments, such as matching [sq, rt] with [_, 5). since square root is the 0.5-th power. Additionally, we study the robustness Code- BERTScore to adversarial perturbations. We found that tosen-based metrics such as chrF are

---

0.8
05
math
sq
0.5
md
0.3
0.2
folder
0.0
sh
util mt
ree
folder
(a)
(b)
Figure 6: Heatmaps of the similarity scores between two pieces of code that achieve the same goal. Figure 6(a) shows the similarity scores between os rmdi (folder) and shutil. retree (folder). Figure 6(b)
shows the similarity scores between math. sqrt (x) and x ** 0.5.
much more prone to matching trivial tokens rather than tokens that preserve the semantic meaning of the code. Examples can be found in Appendix E. Additional discussion and experiments regard- ing the distinguishability of CodeBERTScore are provided in Appendix F. Additional general exam- ples are provided in Appendix G.
correlate well with execution accuracy.
Execution-based Metrics To alleviate previous issues, execution- evaluation counts a gener- ated code snippet correct if it produces the re- quired outputs when with given inputs (Chen al., 2021; Athiwaratkun al., 2022: Li al., 2022; Wang et al., 2022; Lai et al., 2022; Huang et al., 2022). However, execution-basse evalua- tion requires datasets that are provided with man- ually crafted test cases for each example, which is costly and labor-intensive to create; thus, only few such datasets exist. In contrast, CodeBERTScorre is completely unsupervised and does not depend any specific dataset. Further, executing model- generated code is susceptible to security threats, and thus should be run in an isolated sandbox, which makes technically cumbersome to work with iteratively.
6 Related Work
Token-based metrics Metrics such BLEU (Papineni al., 2002) evaluate code generation by counting matching n-grams between generated and reference code. CrystalBLEU (Eghbali and Pradel, 2022) refines this approach by disregard- ing trivially shared n-grams, while ROUGE (Lin, 2004) and METEOR (Banerjee and Lavie, 2005) emphasize recall and balance of precision and call respectively. However, these metrics, relying on exact lexical matches, often fail capture se- mantically equivalent but lexically different code snippets. Unlike these, SodeBERTScore captures the wide, two-sided context of each token, which n-grams cannot capture.
Conclusion
In this paper, we present CodeBERTScore,, sim- ple evaluation metric for code generation, which builds on BERTScore (Zhang et al., 2020), using pretrained language models code, and lever- aging the natural language context of the gen- erated code. We perform an extensive evalua- tion across four programming languages which shows that CodeBERTScore is more correlated with human preference than all prior metrics. Fur- ther, we show that generated code that receives a higher score by CodeBERTScore is more likely to function correctly when executed. Finally, we
Static analysis-based metrics CodeBLEU (Ren et al., 2020) incorporates data-flow and Abstract Syntax Tree (AST) matching, in addition to token-matching. However, valid code may not always align AST' and data-flows. Addition- ally, partial code, although potentially useful, may not parse, thus cannot be fully evaluated by CodeBLEU, Further, as highlighted by subsequent studies (Wang et al., 2022), Co does not

---

release five programming language-specific pre- trained models use with our publicly available code. These models were downloaded more than 1,000,000 times from the HuggingFace Hub. Our code and data are available at https//github.com/ htaibb/cod--bet-sccom
evaluating encoder-decoder or decoder-only the NL-Codee generator models.
Another point to consider is that Code- BERTScore relies on a strong underlying BERT- based model, while methods such as BLEU do not have many "moving parts" or hyperparameters to tune. However, this is mostly an advantage, since CodeBERT'Score can be further improved in the future using stronger base models.
Acknowledgement
We thank Misha Evtikhiev, Egor Bogomolov, and Timofey Bryksin for the discussions, and for the data from their paper (Evtikhiev et al., 2022). We thank anonymous reviewers for the valuable feed- back. We are grateful to Yiwei Qin for the dis- cussions regarding the T5Score paper (Qin et al., 2022); the idea to use functional correctness as a meta-metric was born thanks to the discus- sion with her. We are also grateful to Aryaz Eghbali and Michael Pradel for the discussions about CryStalBLEU (Eghbali and Pradel, 2022). This material is partly based research spon- sored in part by the Air Force Research Labora- tory under agreement number 148750-19-2-0225
References
Loubna Ben Allal, Raymond Li, Denis Kocetkov, Chenghao Mou, Christopher Akiki, Carlos Munoz Ferrandis, Niklas Muennighoff, Mayank Mishra, Alex Gu, Manan Dey, al. 2023. Santa- coder: don't reach for the stars! arXiv preprint arXiv:2301.0398%
Miltiadis Allamanis and Charles Sutton. 2013. Mining source code repositories massive scale using lan- guage modeling. In 2013 10th working conference on mining software repositories (MSR), pages 207- 216. IEEE
The U.S. Government is authorized to reproduce and distribute reprints for Governmental purposes notwithstanding any copyright notation thereon. The views and conclusions contained herein are those of the authors and should not be interpreted necessarily representing the official policies or endorsements, either expressed or implied, of the Air Force Research Laboratory or the U.S. Gov- ernment. This project was also partially supported by a gift from AWS AI.
Ben Athiwaratkun, Sanjay Krishna Gouda, Zijian Wang, Xiaopeng Li, Yuchen Tian, Ming Tan. Wasi Uddin Ahmad, Shiqi Wang. Qing Sun, Mingyue Shang. al. 2022. Multi- lulti-lingual evalu- ation code generation models. ArXiv preprint, abs/2210. 14866..
Jacob Austin, Augustus Odena, Maxwell Nye, Maarten Bosma, Henryk Michalewski, David Dohan, Ellen Jiang, Carrie Cai, Michael Terry, Quoe Le, al. 2021. Program synthesis with large language mod- els. ArXiv preprint, abs/2108.07732.
Limitations
Satanjeev Banerjee and Alon Lavie. 2005. METEOR: An automatic metric for MT evaluation with im- proved correlation with human judgments. In Pro- ceedings of the ACL Workshop trinsic Evaluation Measures for Machine Transla- tion and/or Summarization, pages 65-72, Ann Ar- bor, Michigan. Association for Computational Lin- guistics.
CodeBERTScorr requires a GPU for computing the metric, while traditional metrics such as BLEU require only CPU. This adds hardware re- quirement to the evaluation of models of code, while most previous approaches are computation- ally cheaper (e.g., by counting n-grams). How- ever, since training and testing neural models re- quire GPU anyways, we can safely assume that a GPU is available. Further, BERT-base models are encoder-only and non autoregressive; this means that they require only a single "forward pass", compared to encoder-decoder models (e.g., T5) and decoder-only models (e.g., GPT-3) that need autoregressively generate token after token, us- ing forward pass for each output token. Thus, the additional time consumption by encoder-only models (e.g., BERT) is negligible, especially when
Intrinsic and Ex-
Federico Cassano, John Gouwar, Daniel Nguyen, Syd- ney Nguyen, Luna Phipps-Costin, Donald Pinck- ney, Ming Ho Yee. Yangtian Zi, Carolyn Jane An- derson, Molly Q Feldman, et 2022. A scalable and extensible approach to benchmarking nl2code for 18 programming languages. ArXiv preprint, ahs/2208.08227
Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde, Jared Kaplan, Harri Edwards, Yura Burda, Nicholas Joseph, Greg Brockman, et al. 2021. Evaluating large language models trained on code. ArXiv preprint, 107/2207.033744

---

Jacob Devlin, Ming- Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of deep bidirectional transformers for language under- standing. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume (Long and Short Papers), pages 4171-4186, Minneapolis, Minnesota. Associ- ation for Computational Linguistics.
Dal Lago, et al. 2022. Competition- code gen- eration with alphacode. Science, 378(6624):1092 1097.
Chin- Lin. 2004, Rouge: A package for auto- matic evaluation of summaries. In Text summariza- tion branches oul, pages 74-81.
Dana Movshovitz-Atiaas and William Cohen. 2013. Natural language models for predicting program- ming comments. In Proceedings of the 51st Annual Meeting of the Association for Computational Lin- guistics (Volume 2: Shori Pupers), pages 35-40.
Aryaz Eghbali and Michael Pradel. 2022. Crystalbleu: precisely and efficiently measuring the similarity of code. In 37th IEEF/ACM International Conference on Automated Software Engineering, pages 1-12.
Kishore Papineni, Salim Roukos, Todd Ward, and Wei- Jing Zhu. 2002. Bleu: a method for automatic eval- uation of machine translation. In Proceedings of the 40th Annual Meeting of the Association for Com- putational Linguistics, pages 31 1-318, Philadelphia, Pennsylvania, USA. Association for Computational Linguisties.
Mikhail Evtikhiev, Egor Bogomolov, Yaroslav Sokolov, and Timofey Bryksin. 2022. Out of the bleu: how should we assess quality of the code gen- eration models? ArXiv preprint, abs/2208.03133
Zhangyin Feng, Daya Guo, Duyu Tang, Nan Duan, Xi- aocheng Feng, Ming Gong, Linjun Shou, Bing Qin, Ting Liu, Daxin Jiang, and Ming Zhou. 2020. Code- BERT: A pre-trained model for programming and natural languages. In Findings of the Association for Computational Linguistics: EMNLP 2020, pages 1536-1547, Online. Association for Computational Linguistics.
Yiwei Qin, Weizhe Yuan, Graham Neubig, and Pengfei Liu. 2022. T5score: Discriminative fine-tuning of generative evaluation metrics. arXiv preprint arXiv:2212.05726.
Shuo Ren. Daya Guo, Shuai Lu, Long Zhou, Shujic Liu, Duyu Tang, Neel Sundaresan. Ming Zhou, Am- brosio Blanco, and Shuai Ma. 2020. Codebleu: a method for automatic evaluation of code synthesis. ArXiv preprint, abs/2009. 10297.
Daniel Fried, Armen Aghajanyan, Jessy Lin, Sida Wang, Eric Wallace, Freda Shi, Ruiqi Zhong, Wen- tau Yih, Luke Zettlemoyer. and Mike Lewis. 2022. Incoder: A generative model for code infilling and synthesis. ArXiv preprint, 004299
lan Tenney, Dipanjan Das, and Ellie Pavlick. 2019. BERT rediscovers the classical NL.P pipeline. In Proceedings of the 57th Annual Meeting of the Asso- ciation for Computational Linguistics, pages 4593- 4601, Florence, Italy. Association for Computa- tional Linguistics.
Suchin Gururangan, Ana Marasović, Swabha Swayamdipta, Kyle Lo. Iz Beltagy, Doug Downey, and Noah A. Smith. 2020. Don't stop pretraining: Adapt language models to domains and tasks. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 8342 8360, Online. Association for Computational Linguistics.
Lewis Tunstall, Leandro von Werra, and Thomas Wolf. 2022. Natural Language Processing with Trans- formers. O'Reilly Media, Inc.".
Zhiruo Wang, Shuyan Zhou, Daniel Fried, and Gra- ham Neubig. 2022. Execution-basee evaluation for open-domain code generation. ArXiv preprint, abs/22 22. 10481.
Junjie Huang, Chenglong Wang. Jipeng Zhang, Cong Yan, Haotian Cui, Jeevana Priya Inala, Colin Clement, and Nan Duan. 2022. Execution-based evaluation for data science code generation mod- els. In Proceedings of the Fourth Workshop on Data Science with Human-in-the-Loopp (Language Advances), pages 28-36, Abu Dhabi, United Arab Emirates (Hybrid). Association for Computational Linguistics.
Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pier- Cistac, Tim Rault, Rémi Louf, Morgan Fun- towicz, et al. 2019. Huggingface's transformers: State-of-the-art art natural language processing. ArXiv preprint, abs/1911.03371.
Yuhang Lai, Chengxi Li, Yiming Wang, Tianyi Zhang, Ruiqi Zhong, Luke Zettlemoyer, Scott Wen-tau Yih, Daniel Fried, Sida Wang, and Tao Yu. 2022. Ds-1000: A natural and reliable benchmark for data science code generation. ArXiv preprint. 155/.
Frank Xu. Uri Alon, Graham Neubig, and Vincent J. Hellendoorn. 2022. A systematic evaluation of large language models of code.
Pengcheng Yin, Bowen Deng, Edgar Chen, Bogdan Vasilescu, and Graham Neubig. 2018a. Learning to mine aligned code and natural language pairs from stack overflow. In International Conference on Min- ing Software Repositories, MSR, pages 476 486. ACM.
Yujia Li, David Choi, Junyoung Chung, Nate Kush- man, Julian Schritwieser, Rémi Leblond. Tom Beeles, James Keeling, Felix Gimeno, Agustin

---

Pengcheng Yin. Bowen Deng, Edgar Chen, Bogdan Vasilescu, and Graham Neubig. 2018b. Learning to mine aligned code and natural language pairs from stack overflow. In Proceedings of the 15th Interna- lional Conference on Mining Software Repositories, pages 476-486.
Aohan 7eng. Xiao Liu, Zhengxiao Du, Zihan Wang, Hanyu Lai, Ming Ding. Zhuoyi Yang, Yifan Xu, Wendi Zheng, Xiao Xia, et al. 2022. Glm-130b: An open bilingual pre-trained model. ArXiv preprint, abs/2210.02414.
Tianyi Zhang, Varsha Kishore, Felix Wu, Kilian Q Weinherger, and Yoav Artzi. 2020. Bertscore: Eval- uating text generation with BERT. In 8th Inter- national Conference on Learning Representations, ICLR 2020, Addis Ababa, Ethiopia, April 26-30, 2020. OpenReview.net.
Shuyan Zhou, Uri Alon, Frank F. Xu, Zhengbao Jiang, and Graham Neubig, 2023. Doeprompting: Gener- ating code by retrieving the docs. In International Conference Learning Representations (ICLR), Kigali, Rwanda.

---

A
Additional Details
2860 annotated code snippets generations
472 examples) where each snippet is graded by 4.5 annotators.
F g
The well-known F score is computed as:
precision recall precision + recall
F =
B.2 Functional Correctness
rocall
precision
(4) A more general F score Fg uses positive factor 3. where recall is considered / times as important as precision:
We evaluate functional correctness using the Hu- manEval (Chen et al., 2021) benchmark. Each ex- ample in HumanEval contains a natural language goal, hand-written input-output test cases, and human- written reference solution. On average, each example has 7.7 cases and there are 164 examples in total. While the original HumanEval in Python, Cassano et (2022) translated Hu- EvnEval to 18 programming languages, and pro- vided the predictions of the Codex model (Chen et al., 2021) (code deeteeddviiii---ee corresponding functional correctness." We used Java, C++, Python, and JavaScript for these periments, which some of the most popular programming languages in open-source projects." Notably, Cassano al. (2022) did not translate the reference solutions to the other languages, so, we collected these from HumanEval-X (Zeng et al., 2022). 8 The reference score of every example is either 1 ("correct", if it passes all test cases) or 0 ("incorrect", otherwise).
(1+ 32) precision recall
F3
(5)
82 precision + recall
As found in METEOR (Banerjee and Lavie, 2005), using F3 with 3 = 3, thus preferring call over precision, results in higher correlation with human preference in machine translation. In our experiments, we found that this applies to NL-> as well.
and their
Token Weighting Following Zhang et al. (2020), compute the inverse document fre- quency (idf), according language-specific set, and weigh each token according its negative log frequency.
Scaling Following Zhang et al. (2020), the co- sine similarity scores of hidden states tend to lie in limited range. Thus, we can linearly scale the resulting scores, using an empirical base scalar b:
C Correlation Metrics
Kendall-Tau (T) T measures the ordinal/rank
CodeBERTScore- CodcBERTSccre b 1- b
association between metric such as Code- BERTScore and the reference measurement. It is calculated
(6)
This typically spreads the CodeBERTScore Fi scores to the [0, 1] range, and is merely cos- metical change: this scaling does not change the way CodeBERTScore ranks different prediction, but can be slightly more intuitive and casier to in- terpret. We computed b empirically by sampling random unrelated code pairs and measuring their average similarity score. For Java, the empirical bjava was 0.78 and for C++, bc++ it was 0.76.
|concordant| discordant concordantt + discordant
where |concordant represents the number of pairs where measurements agree their relative rank. That is, if f(11, vi) [ vš). the reference measurement also yields f" vi) ("22, ct). Similarly, |discordant] represents the number pairs where two measurements yield opposite ranks. Notably, in our experiments, we restrict the comparisons of ranks within the gener- ations of the same question.
B Evaluation Details
B.1 Human Preference
For each example, Evtikhiev et al. (2022) asked experienced software developers to grade the gen- erated code snippets from five different models. The grade scales from zero to four, with zero de- noting that the generated code irrelevant and un- helpful, and four meaning that the generated code solves the problem accurately. Overall, there are
Pearson (rp) Tp measures the linear correlation between metric and the reference measurement.
"htps://huggingface "htps//tuuginngaaecoo/taatttttte wow.eeattaetttttt///t/tll....... "hups://octoverse github..co//02222 top- htwprrummiii--unguueeee

---

It is defined as:
has the same variable name but without any func- tion call. Contrarily, chrF assigns higher rank- ing to (folder) which has longer common sequence of characters, although semantically in- equivalent.
rp- EE (10:0
where N is the number of generations in the dataset, f is the mean Scde of the dataset, and f is the mean similarity score cal- culated by the reference measurement.
F Distinguishing Code with Different Semantics
We study how well can CodeBERTScore perform as a generic similarity function that measures the similarity between two arbitrary code snippets yi and 4/-
Spearman (rg) rs measures the Pearson corre- lation coefficient between the ranks produced by a metric and the reference measurement:
F.1 Distinguishability Metric
Jp CO (RU(()))R(M((
We evaluate CodeBERTScore using the distin- guishability metric d proposed by Eghbali and Pradel (2022) which is calculated as follows:
where R returns the ranks of code snippets in a
collection of code snippets Y. cov(-,-) is the co- variance of two variables and (.) is the standard deviation.
Ey.wy EPairs f(yss 91)
(7)
Σ CParssar [yss
where Pairinr defines a set of code pairs from the same semantically equivalent clusters, and Pairinter defines set of code pairs from two clusters of dif- ferent functionality. Formally,
D Standard Deviation
Table 3 shows the same results as in Table 1, but with standard deviations. Table 4 shows the results from Table 2, with standard deviations.
Pairintra = { (()),,;)) 3k such that yı: yj E C
E Robustness to advessarial perturbations
Pairmter =1 (1)),,;)) 3k such that
Yi E Ck. Ck}
where Ck is the k-th cluster with semantically equivalent code snippets. Intuitively, similar- ity function f that distinguish between similar and dissimilar code will produce d larger than meaning that a pair of code snippets from the same semantic cluster has a higher similarity score than a pair of snippets from different clusters. Since the number of intra-class and inter-class pairs grows quadratically with the number of code snippets, in our experiments we followed Eghbali and Pradel (2022) to sample N inter- and N intra-class pairs instead.
Ref: shutil.rmtree
reeree (folder)
Candidate
CodeBERTScore
chrF
68. rnddi (fol der)
1st
1st
os cn.rdd (f)
2nd 3rd
(folder)
3rd 2nd
The similarity rankings of three code snippets given the reference code shutil. reer (folder). BERTScore correctly ranks os. rmdir (E) over the the non-equivalent (folder), chrF prefers just (folder) over rmddi
Figure 7:
While Code-
F.2 Dataset with Semantically equivalent clusters
We conducted a qualitative evaluation of Code- BERTScore under various perturbations. An example is shown in Figure 7, which shows the CodeBERTScore and chrF rankings of three code snippets based on the similarity the reference shutil. rmtree (folder). CodeBERTScore gives a higher ranking to the code snippet that employs the appropriate API (os. rmdir) than the trivial (folder) that
We follow Eghbali and Pradel (2022) evaluate whether ScdeBERTScore can distinguish similar and dissimilar code mined from ShareCode an online coding competition platform. Semantically equivalent code snippets are from the same coding problem, and they all pass the unit tests provided
"https//sharocnde..o/

---

Java
C++
Python
JavaScript
Metric
BLEU
(80
(351
.112(4-089)
(301c).084)
(393
.352
.248 175)
.343ca.052)
CodeBLEU
096(=:03)
0374
021)
.201(2.037)
.366(=009)
.326(=
.261 065)
.299(1.043)
ROUGE-1
516(1:052)
.318:1003)
2.23)1.073)
..00(.0024)
.334:1.00
.279
1.00))
ROUGE:2
029)
031
2730
.273c2006)
037)
(267(tt)77))
.292:+.057)
ROUGE-L
508(+.060)
(344ct038))
.258(t.091)
.288c2.027)
338 103)
.350ct:064)
.271(t.078)
.293(1.016)
METEOR
(58(2.058)
027)
(31((+0061)
(321c22023)
418(2.090)
402(+::49)
.324(+.075)
.415ct.022)
chrF
532(4.067))
(35)
.321(1.020)
.394
.379:10088)
.302
..74(4.044)
CrystalBI.FU
(74
067)
.446+ (09)
.095c 064)
.39 (280)
073)
.118ct 057)
.05922009)
CodeBERTScore
(53()))
.369(t:099)
.327 12772.1800
.393
422
015
.319(:
402c+.090
Table 3: Kendall-Tau (r) and Spearman (r₂)) correlations of each metric with the functional correctness on Hu-
EvnEval in multiple languages. The correlation coefficients are reported as the average across three runs, along with the standard deviation.
Metric
7p
P>
BLEU
025)
.543ct 018)
CodeBLEU
.350
.339(1003)
.495(1.037)
ROUGE-1
(397(=023
.604(1.016)
.580
ROUGE-2
.429(.0055)
.629(1.01S)
.58)(1.022)
ROUGE-I
027)
.619(1.014)
-574(-022)
METEOR
.36)
.581(L.06)
.540(.022)
chrF
029)
.635c4003)
.123(t018))
CrystalBLEU
(30((000)
.58
576(2.034)
CodeRertScore
-517(=04)
.674ct.012)
.662(+-013)
Table 4: The Kendall-Tee Tau (7), Pearson (ry,) and Spearman (r,) correlation with human preference. The best performance is bold. The correlation coefficients are reported as the average across three runs. Numbers inside parentheses indicate the standard deviations.
Metric
Java
C++
9.13 while CrystalBLEU achieves only 6.94. This result confirms that CodeBERTScore assigns higher similarity scores semantically similar code pairs, compared to randomly paired snippets that belong to different semantic classes.
BLEU
2.36
2.51
CodeBLEU
1.44
1.42
CrystalBIFUU
5.96
6.94
CodeBERTScore
9.56
9.13
Table 5: Distinguishability with different metrics the similarity function. CodeBERTScone achieves a higher distinguishability than CrystalBLEU, which proposed this meta-metric, on the same datasets.
Can We Hack the Distinguishability Metric? Despite the encouraging results in Table 5, we also found that distinguishability can be casily manipu- lated since it compares absolute scores across dif- ferent metrics. For example, while CrystalBLEU achieves distinguishability score of 5.96, craft variant of CodeBERTScore that achieves a distinguishability score of 120,000 by simple ex- ponentiation CodeBERTScone'ss output score.
by the platform. The dataset consists 6958 code snippets covering 278 problems in Java and C++ We use CodeBERTScore to calculate the similar- ity score for code pairs that share the same seman- tic class and code pairs that do not. We then mea- sure the distinguishability of CodeBERTScore cording Equation The results are shown in Table
To illustrate this, conducted a distinguisha- bility evaluation with the same configurations before, but with variant of CodeBERT that we call CodeBERTSSoree and defined as the composition of CodeBERT x function, that is: CRd (y1, 1/) = (CodeBERRScoree (y1, v2))
Table 5 shows that CodeBERTScore achieves higher distinguishability than CrystalBLEU, which proposed this meta-metric, in both Java and C++. CodeBERTScore achieves distinguisha- bility scores of 9.56 in Java while CrystalBLEU achieves 5.96; in C++, ScdeBERTScore achieves
with the f (x) =
As Figure shows, distinguishability of CodcBERTScore increases almost exponentially while increasing kt although the base Code-

---

130,000
120,000
CodeBERTScore &
110,000
100,000
90,000
80,000
70,000
60,000
50,000
40,000
30,000
20,000
10,000
20
30
50
Figure 8: Distinguishability by exponentiating the orig- inal CodeBERTScore by k.
BERTScore metric has not changed.
We thus argue that distinguishability is not reliable meta-metric and is substitute for inccutionbbssed or human-rating We further sus- pect that any meta-metric that compares exact, ab- solute, scores across different metrics suscepti- ble to such manipulations, and the reliable way to compare metrics is according to the way they rank different examples, rather than the exact scores. results
The distinguishability CodeBERTScore* with different values of are shown in Figure 8. As Figure 8 shows, the distinguishability increases almost exponentially with the increasing value of k. We thus argue that distinguishability
of
not a reliable meta- metric and is no substitute for bxccution-based- or human-rating. We further suspect that any meta-metric that compares exact, absolute, scores across different metrics is susceptible to such manipulations, and the reliable way to compare metrics is according to the way they rank different examples, rather than the exact scores.
G Additional Examples
In this section, we provide additional examples in which CodeBERTScore prefers the functionally correct prediction, while the best bascline metric in each language ranks higher functionally in- correct prediction, which is inequivalent to the ref- erence. Figure 9 shows an example in Java, and Figure shows C++ example.

---

Natural Language Question:
Reference:
/*a
public static int howkanylimes (String string,
Find how many times given substring be found in the original string. Count overlaping cases. howManyTimes ("*, "a")
String substring)
int times
for (int 0; String.lengte bulsstring.eengtan + 1; i-+)
substring, length )
.equals (substring)) {
>>> howNanyTimes ("aaa", "a")
times 1;
>>>
howNanyTimes ("aaaa", "aa ")
return times;
(a) The natural language question.
(b) The ground truth reference.
Non-equivalent candidate:
Equivalent candidate:
public static int howKanyTimes(
public static int howManyTimes
string string, String substring) int count = 0;
String string, String substring)
int counter = 0;
int index = 0;
int index = 0;
while ((index string. index0f
while (true) index string, (substring, index):
substring, index) !- -1)
count+t;
if (index == -1)
index +- substring. 1ength
break; counter +- 1;
return count;
index +- 1;
return counter;
(c) Preferred by METEOR.
(d) Preferred by CodeBERTScore.
Figure 9: An example in HumanFYvalJava, in which METEOR assigns a higher score to Figure 9() which is not functionally equivalent to the reference (Figure 9(b)), while CodeBERTScore prefers the functionally correct candidate Figure 9d)).

---

Natural Language Question:
Reference:
/x
long specialFilter(
Write a function that takes a vector of numbers as input and returns the number of elements in the vector that are greater than 10 and both first and last digits of number are odd (1, 3, 5, 7, 9) For examplo: spectalFiiter (115, -73, 14, -15))
secctorco nums) {
int num-0;
for (int i-0;i=nums.sa size ; ;1++)
if (nums [i]=10> {
string t--tosstring (nums [1]); if (#[0] and x [ww lww.tength ()- -11] & == 1)
=> 1
num+=1;
specialFiiter (133, -2, -3, 45, 21, 1091)
=> 2
return DUR?
(a) The natural language question.
(b) ground truth reference.
Non-equivalent candidate:
Equivalent candidate:
long specialFilter(
long specialFilter(
std: :vector<long- nums)
std: std::vectr<<oong nums) {
// Write your code here 0;
long count
long
for (int i-0; i<nums size 1-+)
for (int 1=0; 1<nums.4 se (); 1++) {
(nums 10)
std: :string str
int last nums [i] 8 10:
std::to_string (nums] [1]);
int first nums [1];
long firstDigit nums [i]
while(first 10) {
abs (nums [1]);
/= 10;
long lastDigit nums i1100;
[il 10 &
(last $ !- 0 &
(firstDigit $
11
first !- 01 c
lsstDigit
1
sum++;
return sum;
return count;
(c) Preferred by chrF.
(d) Preferred by CodeBERTScore.
Figure 10: An example HumanEval-C++, which chrF assigns a higher score to Figure 10(c) which is not functionally equivalent to the reference (Figure 10()), while CodeBERTScore assigns higher score the func- tionally correct candidate Figure 10(d).


Reviews:
[{"review_id": "MhjNczPQSI", "reviewer": "Reviewer_29mb", "summary": "", "questions": "I believe that your paper is well experimented and well written. But I still have a few questions that I need you to clarify.\n\nA. About Human Preference.\nIn the paper, the authors mention that \"use the human annotation released by Evtikhiev et al. (2022) to measure the correlation between each metric and human preference.\" \nI am confused about this, why don't the authors use the same method to evalute on other datasets (not just CoNaLa)?\n\nB. The choice of base models and pre-training methods.\nWhy do authors focus on CodeBERT and not consider alternatives such as GraphCodeBERT and UniXcoder. Additionally, it is relevant to inquire why pre-training tasks like contrast learning were not taken into account.\n\nC. Robustness. \nDid the authors design CodeBERTScore with its robustness in mind?", "limitations": "", "rating": 5, "confidence": 5, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for taking the time to review our paper and for your kind words!  We were happy to read that you think that this paper is “of great interest to NLP and SE” and that “The paper provides extensive evaluation”.\n\nWe think that all your questions are addressable within this discussion period. Please see our response below.\n \n> ### While I found the performance of CodeBERTScore to be impressive when I downloaded and used it ... CodeBERTScore may lack robustness in accurately understanding the semantics of code ... \n\nThis is a great point. We agree that distinguishing code snippets with nuance differences is challenging for all metrics, including CodeBERTScore. Quantitatively and qualitatively (Figure 6 and Appendix G), we observe that CodeBERTScore is more robust and accurate than all previous methods, including approaches that explicitly take data dependency into account (such as CodeBLEU).\n\nWe agree that further improving the robustness of neural-based metrics is an important direction for future research. To this point, we are planning to include a discussion and analysis of the current limitations in the revised version, specifically adding typical examples where CodeBERTScore still fails at providing good assessments. Thank you for raising this point, we believe this additional analysis will make the paper stronger!\n\n\n\n> ### CodeBERTScore is not customised for the features of the code (e.g., syntax information, data flow information, or even LLVM-IR information obtained through the compiler).\n\nAlthough CodeBERTScore doesn’t explicitly encode syntax or data-flow information, we compare it with metrics that do so, such as CodeBLEU which uses data-flow and Abstract Syntax Trees. However, as shown in Table 1 and Table 3, CodeBLEU does not exhibit a superior correlation with human preference and functional correctness, and is sometimes even lower than BLEU. \n\nWe believe that similarly to the process that NLP research has gone through, from using syntax, dependency, etc. to pure neural modeling, it might be the case for code as well: although syntax and data flow information are provided “for free”, strong neural models can just work better empirically. This is also evident by the rise of generative models of code such as Codex and  CodeLlama, which perform much better than prior models that explicitly generated ASTs and graphs.\n\n> ###  it may not introduce significant novelty compared to applying BERTScore to other languages ... However, it is still valuable to have a dedicated evaluation metric for code generation, as it addresses the unique challenges and requirements of that task.\n\nThe main novelty in our work lies in the empirical results demonstrating the utility of (Code)BERTScore (and the new code BERT models that we train) for the evaluation of both human preferences and functional correctness of code. Such a metric is urgently needed in the NLP-SE community, and the community so far did not have any strong metric to rely on which was extensively tested and evaluated. We hope that our work will inspire future work in this direction.\n\n> ### Q1: the authors mention that they \"use the human annotation released by Evtikhiev et al. (2022) to measure the correlation between each metric and human preference.\" I am confused about this, why don't the authors use the same method to evaluate on other datasets?\n\nThe annotations released by Evtikhiev contain only annotations for the Conala dataset. We thus used functional correctness to evaluate the different metrics on the other dataset (HumanEval).\n\n\n> ###  Q2: Why do authors focus on CodeBERT and not consider alternatives such as GraphCodeBERT and UniXcoder? Additionally, it is relevant to inquire why pre-training tasks like contrast learning were not taken into account \n> ### Q3: Did the authors design CodeBERTScore with its robustness in mind?\n\nSince such an approach with its extensive evaluation was so missing in the current literature and the community, we wanted to show that **even the simplest base model helps**. \nOur approach is certainly not coupled with the specific “CodeBERT” model. The underlying base model can be replaced easily with any model that is available on the Huggingface hub.\nWe believe that our paper presents the first extensive evaluation in this direction, and that stronger base models and better training, such as contrastive learning, can certainly further improve the results.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "A. Relevance: Code generation is an important and challenging task in the field of natural language processing and software enginering. This paper addresses the crucial problem of evaluating the output of code generation models, which is of great interest to the NLP and SE.\n\nB. Evaluation: The paper provides extensive evaluation of CodeBERTScore across multiple programming languages, demonstrating its higher correlation with human preference and functional correctness compared to existing metrics. This evaluation adds to the credibility and significance of the proposed metric.\n\nC. Practicality and Open Access: The authors make their code and pre-trained models publicly available, facilitating reproducibility and remedying the gap in the code generation field where semantic-based evaluation metrics are missing. This practical contribution provides an invaluable resource for researchers and developers in NLP and SE.", "reasons_to_reject": "A. Novelty: While CodeBERTScore builds upon the BERTScore approach and applies it to the code generation domain, it may not introduce significant novelty compared to applying BERTScore to other languages (such as Japanese or Russian languages). However, it is still valuable to have a dedicated evaluation metric for code generation, as it addresses the unique challenges and requirements of that task.\n\nB. Customization: CodeBERTScore is not customised for the features of the code (e.g., syntax information, data flow information, or even LLVM-IR information obtained through the compiler). \n\nC. Robustness: We are aware that existing neural networks and pre-trained models may encounter challenges with robustness. While I found the performance of CodeBERTScore to be impressive when I downloaded and used it, I did come across potential issues.\nFirstly, CodeBERTScore appears to have difficulty accurately recognizing different implementations of the same functional code, such as distinguishing between looping statements with \"for\" and \"while\" constructs.\nSecondly, CodeBERTScore seems to struggle when it comes to code that contains similar tokens but different functions. For example, it may not effectively differentiate between expressions like 'a += b' and 'a -= b'.\nThese observations imply that CodeBERTScore may lack robustness in accurately understanding the semantics of code.", "questions_to_authors": "I believe that your paper is well experimented and well written. But I still have a few questions that I need you to clarify.\n\nA. About Human Preference.\nIn the paper, the authors mention that \"use the human annotation released by Evtikhiev et al. (2022) to measure the correlation between each metric and human preference.\" \nI am confused about this, why don't the authors use the same method to evalute on other datasets (not just CoNaLa)?\n\nB. The choice of base models and pre-training methods.\nWhy do authors focus on CodeBERT and not consider alternatives such as GraphCodeBERT and UniXcoder. Additionally, it is relevant to inquire why pre-training tasks like contrast learning were not taken into account.\n\nC. Robustness. \nDid the authors design CodeBERTScore with its robustness in mind?", "paper_topic_and_main_contributions": "Topic: This paper addresses the problem of evaluating the code generated by neural network, which needs achieve a higher correlation with human preference and with functional correctness than all existing metrics.\n\nContributions:\nA. The introduction of CodeBERTScore. CodeBERTScore builds upon BERTScore, a contextual encoding method, to compute a soft similarity score between the generated code and the reference code. Unlike traditional metrics like BLEU, which rely on exact token matching, CodeBERTScore takes into account the contextual encodings of large pretrained models. Additionally, CodeBERTScore encodes not only the generated tokens but also the programmatic context surrounding the generated code.\nB. Large-scale empirical research. The authors extensively evaluate CodeBERTScore across four programming languages. They demonstrate that CodeBERTScore achieves a higher correlation with human preference and functional correctness compared to existing metrics. Moreover, they conducted a series of additional experiments to understand the importance of different design decisions, and to gain insights on applying CodeBERTScore to new datasets and scenarios.", "excitement": 5, "reproducibility": 5, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Missing_References": "None.", "Typos_Grammar_Style_and_Presentation_Improvements": "None."}, "extra_text": {}}, {"review_id": "KkEB2GOjc9", "reviewer": "Reviewer_HHa9", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 4, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for taking the time to review our paper and for your kind words!  We were happy to read that you think that “The evaluation is thorough and well thought out” and that you “believe many researchers and practitioners would benefit from it”.\n\n> ### I really enjoyed reading this paper, but the contribution is incremental compared to prior work (they straightforwardly extend BERTScore).\n\nThe main novelty in our work lies in the empirical results demonstrating the utility of (Code)BERTScore (and the new code BERT models that we train) for the evaluation of both human preferences and functional correctness of code. Such a metric is urgently needed in the NLP-SE community, and the community so far did not have any strong metric to rely on which was extensively tested and evaluated. We hope that our work will inspire future work in this direction.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. Metrics for code generation are still poor and the authors improve the SoTA in this space. It is very easy to reuse their work and I believe many researchers and practitioners would benefit from it.  \n\n2. The evaluation is thorough and well thought out. They compare to many other metrics and they report sensible estimates for the level of correlation. \n\n3. The paper is well written and easy to follow. ", "reasons_to_reject": "I really enjoyed reading this paper, but the contribution is incremental compared to prior work (they straightforwardly extend BERTScore). ", "questions_to_authors": "", "paper_topic_and_main_contributions": "This paper extends a well know metric for evaluating text generation, BERTScore, to code generation. Instead of encoding only the generated tokens as in BERTScore, CodeBERTScore also encodes the natural language input preceding the generated code. \n\nCodeBERTScore encodes the generated code and the reference code independently with pretrained models, with the inclusion of natural language instructions or comments. Then, it computes the cosine similarity between the encoded representations of each token in the generated code and each token in the reference code. Finally, the best matching token vector pairs are used to compute precision and recall.\n\nThey further pretrain CodeBERT for separate programming languages (Python, Java, C++, C and JS) on CodeParrot and release the models on HuggingFace. \n\nThey evaluate across 4 programming languages. To show correlation with human preference (via Kendall-Tau, Pearson and Spearman scores)  they use CoNaLa, to show correlation with functional correctness they use HumanEval. \n\nThey find that CodeBERTScore achieves a higher correlation with human preference and with functional correctness (via unit tests) than other standard metrics. Results show significant gains for human preference and gains for functional correctness (in particular for C++ and Python). ", "excitement": 4, "reproducibility": 5, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "cEjXGHY0Gg", "reviewer": "Reviewer_Q9Kk", "summary": "", "questions": "", "limitations": "", "rating": 2, "confidence": 4, "soundness": 2, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for taking the time to review our paper!  \nWe think that all your questions are addressable within this discussion period. Please see our response below.\n\n> ### The authors claim that their score is correlated with human human judgement although in fact it is only the correlation with HumanEval labelling and not independent evaluation\n\nWe did perform human judgment experiments, with the annotations taken from Evtikhiev et al. (2022). The setup is explained in L313-L320; the experimental results are in Table 2; and the discussions are in L350-L364. More details are provided in Appendix B.1.\n\n\n\n> ### What is the novelty of this work?\n\nThe main novelty in our work lies in the empirical results demonstrating the utility of (Code)BERTScore (and the new code BERT models that we train) for the evaluation of both human preferences and functional correctness of code. Such a metric is urgently needed in the NLP-SE community, and the community so far did not have any strong metric to rely on which was extensively tested and evaluated. We hope that our work will inspire future work in this direction.\n\n> ### Does mentioning that the model has more than 1M downloads on HuggingFace violate anonymity?\n\nApologies if mentioning that the model has been downloaded on Hugging Face seemed to break anonymity in some way. This was certainly not our intention, and honestly we don’t think it breaks anonymity any more than providing the model name itself (which is standard in many papers). In either case, it would require the reviewers to **actively search** for the model name, which reviewers are encouraged not to do by the ACL reviewing guidelines: https://acl-org.github.io/policies/reviewers", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "The authors claim that their model is pretty popular on HuggingFace.", "reasons_to_reject": "I am not sure if it is not violating anonymity to claim that your model has more than 1M downloads on HuggingFace alongside with providing the name.\nAnother important point is that the authors claim that their score is correlated with human human judgement although in fact it is only the correlation with HumanEval labelling and not independent evaluation.\nAt last but not least there is no scientific novelty in their work. I could consider this paper as a demo one, but not for the main track.", "questions_to_authors": "", "paper_topic_and_main_contributions": "The paper is devoted to presentation of CodeBERTScore model(s). The authors provide some evaluation of the models with comparison with other existing metrics.\n", "excitement": 2, "reproducibility": 4, "ethical_concerns": "Yes", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Justification_for_Ethical_Concerns": "As I mentioned above, there I am not sure if the anonymity is not spoiled by the authors in the paper."}, "extra_text": {}}]

Author rebuttal:

Meta-review: Model generated code evaluation is a really hard problem and this work is a very interesting and useful contribution to that issue. While the authors go to good lengths to justify the value of the model, it is still perhaps a matter of judgement whether the score coming from CodeBERTScore is more meaningful than other metrics. It's my opinion that code execution in environments with tests is the gold standard for code evaluation, but this is a very high burden for many important questions.
Decision: Accept-Main