Beyond Accuracy: Evaluating Self-Consistency of Code Large Language Models with IdentityChain
Authors: Marcus J. Min
Yangruibo Ding
Luca Buratti
Saurabh Pujar
Gail Kaiser
Suman Jana
Baishakhi Ray
Venue: ICLR 2024 poster (2024)
Keywords: Large Language Models
Code Generation
Code Summarization
LLM Evaluation
Self-Consistency
Primary area: generative models
TL;DR: Self-consistency of Code LLMs is overlooked. We design a new framework that evaluates self-consistency and general accuracy simultaneously, showing that self-consistency is a different aspect from accuracy which needs to be improved.

Abstract: Code Large Language Models (Code LLMs) are being increasingly employed in real-life applications, so evaluating them is critical. While the conventional accuracy evaluates the performance of Code LLMs on a set of individual tasks, their self-consistency across different tasks is overlooked. Intuitively, a trustworthy model should be self-consistent when generating natural language specifications for its own code and generating code for its own specifications. Failure to preserve self-consistency reveals a lack of understanding of the shared semantics underlying natural language and programming language, and therefore undermines the trustworthiness of a model. In this paper, we first formally define the self-consistency of Code LLMs and then design a framework, IdentityChain, which effectively and efficiently evaluates the self-consistency and conventional accuracy of a model at the same time. We study eleven Code LLMs and show that they fail to preserve self-consistency, which is indeed a distinct aspect from conventional accuracy. Furthermore, we show that IdentityChain can be used as a model debugging tool to expose weaknesses of Code LLMs by demonstrating three major weaknesses that we identify in current models using IdentityChain. Our code is available at https://github.com/marcusm117/IdentityChain.

Published as a conference paper at ICLR 2024
BEYOND ACCURACY:
EVALUATING SELF-CONSISTENCY OF CODE LARGE
LANGUAGE MODELS WITH IDENTITYCHAIN
Marcus J. Min
Yangruibo Ding²
Luca Buratti
Saurabh Pujar"
Gail Kaiser"
Suman Jana'
Baishakhi Ray!
CColubii University
2BB Research
$m50250columbs edu
(yrbding, kaiser, suman, rayb jccs columbia......mmomo.o..m.c...i.a edu Aluca.buratti2 saurabh. jujrabb..jjaree... pujar] com com
ABSTRACT
Code Large Language Models (Code LLMs) are being increasingly employed real-life applications, so evaluating them is critical. While the conventional accu- racy evaluates the performance of Code I.LMs on a set of individual tasks, their self-consistency across different tasks is overlooked. Intuitively, a trustworthy model should be self-consistent when generating natural language specifications for its own code and generating code for its own specifications. Failure to pre- serve self-consistency reveals lack understanding of the shared semantics un- derlying natural language and programming language, and therefore undermines the trustworthiness of model. this paper, we first formally define the self- consistency of Code LLMs and then design framework, IdentityChain, which effectively and efficiently evaluates the self-consistency and conventional accu- racy of model at the same time. We study eleven Code LLMs and show that they fail to preserve self-consisteney, which is indeed a distinct aspect from conven- tional accuracy. Furthermore, we show that Identity can be used as a model debugging tool to expose weaknesses of Code LLMs by demonstrating three ma- jor weaknesses that we identify in current models using Identity Our code is available at htttts: http://gtttttttteteeee cwm/ cusm 17 1tt.mmmmm1mm/eeetttthhhhchhl wtwt:///w/mmmmmmm/m/occoccoccco/
INTRODUCTION
Code Large Language Models (Code LLMs) are being increasingly employed in real-life applica- tions (GitHub) 2023 OpenAl 2023). Hence, evaluating them rigorously crucial problem. Con-
ventional eval avaluations of Code LLMs focus on the models' accuracy on a wide range of individual tasks (Lu et al. 2021 Zhu al. 2022) primarily the following two:
1) Code Generation i.e. Natural Language to Programming Language (NL-to-PL) Generation: Given a natural language specification, the model tasked generate a corresponding program.
2) Code Summarization i.e. Programming Language to Natural Language (PL-to-NL) Generation: Given a program, the model tasked generate a corresponding natural language specification.
However, evaluating these two tasks in isolation overlooks their symmetric nature. NL-to-PL and PL-to-NL Generation can be thought of as semantic-prens prreantriii-eeeiingg translation and bock-translation be- tween the PI. space and the NI space. Therefore, trustworthy model should be able to correctly perform Pl.to-NI. Generation given programs generated by itself from previous NL.-to-P.. tasks. Similarly, it should correctly perform NL-to-PL Generation given natural language specifications generated by itself from previous PL-to-NL tasks. We call such property conisssssit...""....
Consider real example shown in Figure GPT-3.5 is first instructed to generate program plo according to a specification nlg written in a docstring, and then instructed to summarize its own code pla into new docstring nl1, If we evaluate NL-to-PL and PL-to-NL Generation in isolation, GPT-3.5 3.5 is more than capable as it achieves 100% accuracy on both tasks. However, from the self- consistency perspective, even though the model is self-consistent when generating nl, from pla,

---

Published as a conference paper at ICLR 2024
Code Generation
Code Summarization
Code Generation
HumanEval/1188 38: nlg
Model--Gnnrrate Code: pla
Model-Generated Docstring: Medea-Genearted Code:
....
Evaluate whether the given number n can be written as the sun of exactly positive even numbers unn
for in range(1, m//2 1):
Check if there exist four numbers that their sun is equal to
for in range(), n): for in range n): for k range for range B):
for i in rangeti, n//2 + 1): for in range(j, n//2
j.
range(k, n//2 1):
Accurate
Self-Consistems
Not Self Ceff-Consistent
Test Outputs
Test Outputs
is_equel to Se___SR__eee (4)
Palse
Test Outputs Don't Match
is_equal even(4) False
is_equal__t to Sue (S) True
is_equal_to_sur (ven
False
Figure I: The IdentityChain Framework. Starting from a docstring nlg instruct the model to gener- ate a program pla, summarize plg into new doestring n/1, and generate a new program pl1. If the test outputs of pl1 do not match the ones of pla, then the model is not self-consistent. This chain can be extended to length ones of pl,+1, returning binary result that indicates if the model is self-consistent regarding nlo.
and we compute whether, for all n, the test outputs of pl, match the
it surprisingly fails to preserve self coniisensst when generating pla from its own doestring nl1. Note that self-consistency different from consistency: nl, here is generated by the model itself instead of arbitrarily crafted by humans or synthesized by other algorithms. This example reveals that GPT-3.5 doesn't understand the underlying semantics of the programs and natural language specifications, which raises significant trustworthiness concern.
Unfortunately, current NL-to-PL evaluations (Chen et al.) 2021 Li et al. 2023 Rozière et al. 2023) typically assess if the model-generated programs pass set of test cases, and current PL-to-NL. eval- uations (Ahmad et al. 2021: Li et al. 2023] Rozière et al. 2023) commonly employ token- metrics like BLEU (Papineni et al. 2002), which both fail to take self-consistency into account. Although similar self-consistency properties of LLMS have been prohed through some natural lan- guage tasks (Jiang et al., 2023 Ohmer al. 2023). their evaluations rely Closed-domain QA tasks and cannot be generalized to open-ended generation (Section 2 Therefore, in this paper:
1) We formalize the definition of self-consistency and its evaluation (Section B)
2) We design a novel framework, IdentityChain (Section 4). which effectively and efficiently evalu- ates Code LLM's self-consistency by employing new metric, Test Output Match (TOM) score, and leveraging greedy decoding during inference. Through experiments, we exhibit the effectiveness of the TOM score (Section 6.2 and the efficiency of greedy decoding (Section 6.3
3) We evaluate eleven current Code LLMs including GPT-4, showing that they are not always self- consistent. Furthermore, find that more accurate models are not necessarily more self-consistent, highlighting that self-consistency is different aspect from conventional accuracy (Section
4) We show through experiments that TOM score also an effective metric evaluate PL-to- NL Generation (Section 6.2). thus completing IdentityChain as a holistic framework that evaluates the NL--to-PI. accuracy, PL.to-NL accuracy, and coniistenst of Code L.MMs at the same time. We further discuss three major weaknesses of current models that we identify using IdentityChain, demonstrating the potential of IdentityChain as a debugging tool that helps model developers by exposing weaknesses of models and inspiring potential improvements (Section 6.4)
2 RELATED WORK
Evaluating Code Large Language Models. For NL-to-PL evaluation, token-based metrics like Exact Match (Ding al. (023b). Edit Distance (Zhang et al. (223). Jaccard Similarity (Pei et al. 2023), and BLEU (Iyer et al. 20188) Ahmad et al. (021) are used, but these metrics fail to capture the code-specifie characteristies. To address this issue, CodeBLEEU (Ren al. 2020) takes Key-
2

---

Published as a conference paper at ICLR 2024
words, Abstract Syntax Tree, and Data-Flow Match into account, and CodeBERTScore (Zhou al. 2023) computes a similarity score of code embeddings extracted by pre- Code LLMs. How- ever, static code similarity doesn't 't reflect the dynamic semanties of programs, which gave rise to execution-basel metrics like Pass @ (Chen et al. 2021; Austin et al. 2021; Hendrycks et al. 2021 Li et al. (022). Nonetheless, all existing NI.-to-P PT. metrics focus only the one-time accuracy while overlooking whether they are self-consistent regarding a model's 's own output. For PL-to-NL evaluation, BLEU (Papineni al.) 2002) score has been the automated metrie adopted by most mod- els (Rozière et al. 2023] Li et al. 2023 Wang et al., (023). Metrics like ROGUE (Lin 2004), chrF (Popović 2013). and BERTSscore Zhang et al. 2020) are also reasonable choices. However, these static metrics fail to capture semantics separately from syntax and require ground truth references for comparison. In this paper, we proposed a dynamic metric TOM score for self-consistency evalua- tion, showing that it is not only effective but also compatible with all existing evaluation benchmarks with test cases. We also show that TOM score effectively evaluates PL-to-NL Generation regardless of ground-truth references, outperforming all aforementioned PL-to-NL metrics.
Evaluating Self-Consistency of Large Language Models. Previous studies (Minervini & Riedel, 2018 Li et al. 2019 Asai & Hajishirzi 2020) show that LLMs behave inconsistently when given two semantically -oodddddd inputs. However, measuring those inconsistencies is different from eval- uating model's self-consistency since these inputs, either hand-crafted algorithm-synthesizee are not generated by the model itself. As LLMs become better at multitasking (Brown et al.) 2020) Ouyang al. 2022), their self-consistency across tasks evolves into an important issue. Jiang et al. (2023) asks LLMs to generate the answer for an arithmetic reasoning problem, replace a variable in the original problem with an unknown x. and then instruct the same model to solve for given the answer it previously generated. Ohmer et al. (2023) asks LLMs to translate question from English to another language and instruct the same model to answer the questions in both languages. However, both evaluation settings above rely on tasks with fixed ground truths and cannot be gen- eralized to open-ended generation tasks where there can multiple ground truth answers with arbitrary lengths. In this paper, we evaluate Code LLMs on two major open- open-ended generation tasks: NL-to-PL and PL-to-NL Generation.
FORMALIZATION
3.1
SELF-CONSISTRNCY DEFINITION
Given a model M that is capable of performing both NL-to-PL and PL-to-NL Generation, let n2p and p2n denote two instructions that respectively set M perform NL-to-PL Generation or PL-to- NL Generation. In practice, the instructions n2p and p2n are usually prompts. Therefore, a model instructed to perform one of the two tasks can be defined as two functions:
Mn2p NL -
Mpzn: PC
where PL denotes the space of all valid programs in a specific programming language and NL denotes the space of all semantically valid and unambiguous program specifications in a specific natural language. For example, PL can be the space of all valid Python programs and ALC can be the space of all valid and unambiguous corresponding English specifications of these programs, which is the setting for all experiments later in this paper.
program
Let nly E NL be valid and unambiguous natural language specification, and pla = MM2p(nni be the program generated by the model M for nłg- If the model is accurate, then pla and nly should have the same underlying semantics F If we further instruct the model generate specification nl, = Mp2n (pln) given plo. then the semantics of ph1, n/1, pla should be all identical. We call such a property "self-consistency".. Generally, self-consistent model should be able to perform such translations between NL and PC infinitely many times without changing underlying semantics.
Note that self-consiseency is a different property from accuracy. While accuracy assesses a model's ability to uni-directionally translate from Nom.N/ L to PL or the converse in single step, self-consistency
One input entails, contradicts, or is identical to the other.
NNonsensical and amhiguous text is important in natural languages, but for NI.-II tasks, it makes more sense to only consider subset of the natural language that validly and unambiguously specifies programs. AAside from program semanties f.e. input-output hehavior, nlo and plo should be also aligned regarding pragmatic aspects like complexity, security, and human readability. In this paper, our scope is just the semantics.
3

---

Published as a conference paper at ICLR 2024
assesses the model's ability to bidirectionally translate between the two spaces in infinitely many steps. Therefore, a model can remain self-consistent even when it's inaccurate, as long as consis- tently preserves the same error. Similarly, low self-consistency but high accuracy can also happen.
We can now formalize the above intuitions about the self-consistency of Code LLMs. Assume that given NL and PL, there exists semanties space 'D (we don't assume any specific definition of D) s.f. an interpretation function sem is well-defined as the following:
sem -NCUPL D
which means that for all pl E PC or nl NC, the interpretation function sem maps it uniquely to an element in D. We define the self-consistency property as the following:
Definition 1: Seff-Consistency. Given a valid and unambiguous specification nlo E NL, model M is self-consisten w.r.f ulo if and only if
Vi E N, sem(pl,) sem(nl(+1) semm(li+1)
where
plo (nla( m4,41 Mnan(pli), pla+1 Mnzp(1li+1)
Aligning with the informal intuitions, our definition doesn't I consider the initial generation plo to be semantically the same nlg. As long as, for all N, the three-tuple plu, nli+1, and pl semantically identical, we can say that M is self-consistent w.r.t. nlg. If plo is semantically identical nlo and the model is self-consistent w.r.. nlg, we can say the model is "strong self-consistent", since if model is always accurate, it must be self-consistent. We formally define it as:
Definition 2: Strong Self-Consistency. Given nlg NLC, a model M is strong self- consistent w.r.t. nly if and only if M is self-consistent w.r.f. nlg and sem()lu) sem(plo), where pla = (nly()
Similar to the above definitions, we can further define celf-consistency and strong conissonsst w.r. an arbitrary pla E PL. Note that these two classes of definitions are not equivalent but for simplicity, we adopt the self-consistency and strong self-consistency w.r.f. n/g NL definitions.
3.2 HELF-CONSISTENG EVALUATION
Chain of Identity Transformations. Let model M be self-consistent w.r.t. nlg- Instruct the model to generate pla = (nlp() and iteratively apply the PL-to-NL function to get n/441 = (pl.)pp)) and the NL-to-PL. function to get pla+1 M (nl;+ 1). From the semantics perspective, alternatively applying the PL.-t-NNL and NL-to-PL. functions on pla for n E N+ times is equivalent to applying the identity transformation I in the semantics space D on sem(plo) for 2n times:
semn((Maap (pan)"(plo)) = 12t (semm(p)))
The chain of transformations ple between the language spaces NL and PL corresponds chain of identity transformations on (plo within the semantics space D. In the equation above, the superscript n denotes the length such an "identity chain".
Self-Consistency Scores. To evaluate the self-consistency of a model M, it's impossible to extend the identity chain infinitely long or exhaust all NL, so we approximate by picking a fixed chain length E N+ and reasonably large subset of NL with m E N+ elements as an evaluation set. We index the inputs in the evaluation set by j N 1 1 j < m. For an input nlo the evaluation set, we check its corresponding semantic equalities sem(pp:) = sem(nli+1) = sem(pl:41) for all N. 0 < < n. We use binary output sc,., {0. 1) to indicate whether all semantic equalities true at the same time i.e. whether M is self-consistent w.r.t nlo., within steps. Similarly, use ssc,,, E {0, 1} to denote if M is strong self-consistent w.r.t. nlo.j within n steps. Finally, by aggregating sCn., and ssc,., over all j, can evaluate self-consistency and strong self-consistency of M within n steps by reporting two scores SC,, and SSC,, defined as the following.
SC.. Σ/=1 SCR.f
1= SSCn..
17
SSC,,
Selff-consistency w.t. all nío E NLC doesn't 't imply self consistency w.E. all plo E PC. The converse is also not true. The No-to-PI. function Mn2p simply map all nlg to the exact same plo. where M is strong self-consistent w.r.t. pla- No claim can be made about M's self-consistency w.r.t the entire PL space.
4

---

Published as a conference paper at ICLR 2024
4 THE IDENTITYCHAIN FRAMEWORK
4.1 EFFECTIVE JELF--CONSSSTNCY EVALUATION
Determining the truth value of the semantic equalities sem(pl:) = sem(nl;+1) sem(pl)+1) can be performed by humans. However, it's not feasible to employ human judgment when the evaluation set scales up. Consequently, we need automated metrics as approximations.
Inapplicability of Existing Automated Metries. Ideal automated PL-to-NL and NL-to-PL metrics should map a program and a natural language specification to the semantic space, and directly com- pute their semantic distance. Given such ideal metrics, we can approximate even determine the truth values of sem(pl,) = sem( sem(nl4+1) ) and sem(nl;+1) = scm(pl:+1). However, all existing met- rics gauge the semantic equalities indirectly by computing distance between the model- generated candidate and a predefined ground truth reference. Specifically, all existing NL-to-PL metrics com- pute distance between two programs in the same programming language and all existing PL-to-NL metrics compute a distance between two specifications in the same natural language. Unfortunately, we do not have any predefined ground truth reference for either n44+1 or p4,+1
Relaxation of the Semantic Equalities. Recall that our goal is to approximate the truth value of the semantic equalities spl.) = sem(nl:+1) sem(pl;+1). Although there are no existing metrics to approximate the truth values of sem = sem(nl;+1) or sem(nl5+1) = sem(pl:+1), the third equality sem( = sem(ple+1 is feasible to gauge. We can use existing NL-to-PL metrics to approximate this equality as they directly compute a distance between two programs in the same programming language. In addition, the model summarizes pl, wrongly into semantically different n14+1, then program pl1+1, which is supposed to be semantically identical to n44+1, is highly unlikely to have the exact same semantics as pl, and vice versa. Therefore, any effective NL- to-PL. metric, which approximates the truth value of sem(pl; = sem(pla+1) ), can be also considered as an effective approximation to that of sem sem(pk) = sem( +1) In Table 2 we empirically show that there is a positive correlation between them.
Design of the Test Output Match (TOM) Score. While all NL-to-PL metrics have the potential to be self-consistency evaluation metrics, we want to pick one that best approximates the semantic equality spm(p4) = sem(pl++1). As reviewed in Section 2, execution-bases based dynamic metrics like Pass/Fail can directly, though not complete, gauge the code semantics, and are therefore more preferred than static metrics like CodeBLEEU. In Table 2
we empirically verify this conclusion.
The most widely used dynamic metric, Pass@ K, not directly applicable self-consistency eval- uation. Whether pl, passes or fails the test cases does not imply whether it is semantically identical to pl++ and vice versa, so naturally, we come up with a new metric, the Pass/Fail Match (P/FM) score, which checks if pl, and pl;+1 both pass or both fail at the same time. If both of them pass all test cases, they must be semantically identical. If one passes while the other fails, they must be semantically different. However, P/FM doesn't handle the Fail-Fail situation well since pl, and pl++1 can fail the same test case due to completely different reasons.
We, therefore, propose another new metric, the Test Output Match (TOM) score, which compares the exact output of pl, and pl for each test case, records if the outputs match and 0 if the outputs differ, and finally computes the percentage of matches among all test cases.
Number of Matched Outputs
TOM-
Total Number of Test Cases
For syntax errors and runtime errors like ValueError or IndexError, the TOM score is calculated by comparing the full error message instead of just the error type. By capturing more fine-granular semantic information, TOM score better approximates the truth value of sem(pl;) sem(pl)+1) than the simple P/FM score. In Table 2 we show that TOM indeed better correlates the human- judged truth value, and therefore is an effective metric for self-consistency evaluation.
'Taking nll or pl, as the ground truth reference for ral++1 pl++ is not generally applicable. For example, if pla fails some test cases, then nl1 Mp2n (plo), which is supposed to be semantically identical to pla, must he semantically different from nlo. Therefore, nlo cannot be seen the ground truth for nl1.
5

---

Published as a conference paper at ICLR 2024
4.2 EFFICIENT COLSISSNNSYYN EVALUATION
Efficient Evaluation by Greedy Decoding. To evaluate self-consistency up a certain chain length n, we use greedy decoding for both NI.-to-PL. and PI.to-NI. Generation. Given a starting point nlg- if at some step i in the chain, p4,1 is an exact match of pl,, nl441 is exact match of n4,, then by the deterministic nature of greedy decoding, we know that the model will always generate the same program and specification repeatedly. In such cases, consistent w.r.f. pl, or nl; (not necessarily nly). Therefore, our IdentityChain framework adopts greedy decoding and stops the chain early when exact matches found. We show in Figure that, with greedy decoding and early stopping, self-consistent cases can be quickly determined.
assert that the model is self-
4.3 HOLISTIC EVALUATION OF CODE LLMS
The IdentityChain framework not only effectively and efficiently evaluates the self-consistency of Code LLM,, but also holistically evaluates multiple aspects of a model at the same time.
NL.-2-PL Accuracy. The bootstrapping step from nlg to plo is exactly the canonical NL-to-PL, eval- uation setting, where we can compute the Pass @1 score to evaluate the model's NL-to-PL accuracy.
PL-2-NL Accuracy. Unlike NL-to-PL metrics, existing PL-to-NL metrics are all static and there- fore struggle to capture underlying semantics. As discussed in Section 4.1 by back-translating goner-geeerated natural language specification into another program. we can approximate the se- mantic equality between the original program and the specification. Therefore, the SC₁ score i.e. the averaged TOM score between all pla and pl1, can be an effective metric for the model's PL-to-NL we empirically show that TOM outperforms all existing PL-2-NL metries.
accuracy. In Table 2
Strong Seff-Consistency. An ideal model should be both accurate and self-consistent. An accu- rate but not self-consistent model is not trustworthy, while self-consistent but not accurate model is useless. The strong self-consistency score SSC, takes both accuracy and self-consistency into account, which serves as a comprehensive evaluation of the model's 's overall performance.
Model developers can first check the SSC,, score as a performance summary and then examine the SC,, Pass @ 1, and SC₁ scores to determine whether the model is lacking more accuracy or self- consistency. More importantly, with IdentityChain, it's easy to pinpoint cases where a model is not self-consistent to reveal subtle weaknesses of the model, as we will show in Section 6.4
5 EXPERIMENTS
Benchmarks. We evaluate the self-consistency Code LLMs on two widely adopted benchmarks: HumanEval and MBPP. HumanEval (Chen et al., (021) contains 164 hand-crafted Python prob- lems. Liu et al. (202)) proposes HumanBvvlPPng to augment HumanEval with more test coverage. Specifically, use HumanEvalPlus-Minn Miniiuu---iiii..... 1.6. where each problem has 16.5 test cases aver- age. MBPP et al. (02T) includes 974 crowd-sourees Python problems with 3.0 test cases for each problem average. For more precise evaluations, we the test split of the sanitized version of MBPP, which contains 257 problems manually verified by Austin et al. (202)) In both datasets, all problems have predefined meaningful function names, for example, "has close_elemente model generates an incorrect function body at the initial step, there can be a conflict between the semantics of the function body and the name, which weakens the soundness of self-consistency eval- uation. Therefore, we replace meaningful function names with a generic "func" at all steps except the initial one, so that the model solely relies on the semanties of the function body or docstring instead of taking shortcuts using the function name. See Appendix [for a concrete example.
If the
Models. We evaluate two types of Code LLMs: foundation models and instruction-tunet models. For foundation models, we evaluate two open-source model families, StarCoderBase (Li et al. 2023)
and Code Llama (Rozière et al. 2023). For instruction-tunet models, we evaluate the instruction-
tuned versions of Code Llama and StarCoderBase, Google's Gemini-1.0-Pro-O0 TTem 2023 and three most capable OpenAl models: GPT-3.5-Turbo-061. GPT-4-0613, and GPT-4-0122
We set the temperature to 0.2 for Gemini since the API sometimes returns response due to its recitation safety filtering mechanism. To compare with other models using temperature 0.2, Figure and

---

Published as a conference paper at ICLR 2024
Preview (the latest Gur-o-Turbo snapshot). For models from Google and OpenAI, choose the aurameter-frozen snapshots of them so that the results can be reproduced.
Prompts. We use one-shot prompting for all the models on both benchmarks better guide the model to generate the expected format For instruction-tume models, we formulate the prompt as chats (Ouyang al.) )022), where the "system" role provides general instructions, the "user" role provides the input of the one-shot example, and the "assistant" role provides the output of the one-shot example. For foundation models, the prompt is only the one-shot example. To maximize the capacity of all Code LLMs, we carefully customize the prompt template for each model. See the "examples" folder in our code repository for details of the prompt templates. See Appendix for detailed hardware and software configurations of all experiments.
6 RESULTS
6.1 SELF-CONSISTENCY OF CODE LLMS
Code LLMs Fail to Preserve Self-Consistency. We observe in Table [I] that all models' self- consistency and strong self-consistency decreases as the number of iteration steps increases. For example, all models' SSC, scores, which assess strong celf-connistency within five steps, evidently decline up to 78.0% compared to the initial Pass @ Regardless of the accuracy of the initial gen- eration, all models' SC; scores, which assess self-consistency within five steps, also decline up to 43.8% compared to SC₁, Such performance drop indicates that while the models might be initially (strong) self-consistent, they are not able to preserve it. In Section 6.4 we delve deeper into the some of root-cause errors that trigger violations of (strong) self cofficonsistency.
Human
MBPP Sanitized SSC;
Model
Size
Pass @
SSC₁
SC₁
SC₂
Pass @
SC1
SC₂
Instruction-tuned Models
Gemini-Pro S
N/A
54.6
20.2 J 63.0%
44.2
27.6
37.6%
57.2
30.7
1 46.3%
61.9
44.4
28.35
GPT-4-Turbo
N/A
81.0
59.5 1 26.5%
81.0
68.1
10.9%
73.9
61.5
16.8%
85.6
77.8
9.1%
GPT-4
N/A
74.8
63.8 1 14.8%
84.0
76.1
9.95%
72.8
62.6
J 13.9%
88.7
82.5
7.0%
GPT-3.5
NIA
71.8
40.5 1 43.6%
56.4
50.3
10.9%
68.9
54.9 1 20.0%%
86.4
76.3
11.7%
IodeLlama-Inst
7B
16.d
4.3 1 1%
17.8
14.1
20.756
22.2
11.7 47.45
30.7
25.3
17.7%
13B
30.7
17.8 42.0%
40.5
33.1 1 18.2%
40.5
23.0
1 134363
50.2
42.8
14.7%
StarChat-Beta
15B
25.2
5.5 78.0%
19.6
11.0 43.8%
32.3
7.8 76
14.8
11.3
23.79
Foundation
Models
CodeIJama
7B
23.5
8.0 1 66.7%
22.1
19.0
13.9%
38.9
20.6 1 47 47.0%
45.1
43.6
3.4%
13B
35.6
9.8 72.4%
17.8
14.1
1 20.7%
46.3
23.0
60.4%
47.9
42.0
12.2%
3.7 J 00.7%
12.3
20.0%
28.8
80.8%
34.2
31.5
8.0%
StarCoderBase
3B
17.8
4.9
12.3
11.0
10.0%
37.4
14,4
01
39.3
34,2
12.9%
7B
24.5
8.6
65.0%
19.0
16.0
16.1%
43.6
23,0
47.3%
47.1
43.6
7.4%
15B
27.0
8.0 : 70.5%
20.9
17.2
17.056
44.0
21.0
59.256
44.7
41.2
7.8%
Table 1: Performance of Code LLMs evaluated by IdentityChain. Pass@1 indicates the NL-to-PL accuracy. SC₁ representing self-consistency within I step indicates PL-to-NL accuracy. SC; repre- sents self-consistency within 5 steps and SSCs represents strong self- self-consistency within steps.
Calf-Consistenyy is Different from Conventional Accuracy. Existing evaluations Code LLMs refer to conventional accuracy (e.g. Pass @ K) as the model's overall capacity, which is confirmed by our results in Table [ larger models in the same model families indeed have higher Pass@ I scores. However, results in Table show that stacking more parameters does not necessarily guarantee improvement of self-consistency. For example, the Pass @ score StarChat- (15B), which indicates accuracy, is higher than Code Llama-Instruet-7B for both benchmarks, but the SCş score of the former, which indicates self-consistency. is lower than the latter for both benchmarks. For another example, while StarCoderBase-7B performs worse than StarCoderBase-1 15B in Pass I for both benchmarks, it outperforms the double-sized version of itself in SSC,, which indicates strong self-consistency. for both benchmarks.
Forr MBPP, we use task in the prompt split as the one-shot example. For HumanFyalPlus, since there's no dedicated prompt split, we use HumanEval/0 as the one-shot example and exclude it from experiments. 'For Code Llama- and Code llama 7B, the Pass@1 we measured are noticeably different from those reported by Rozière al. (2023. We conjecture that it might be caused by the models' sensitivity prompts.

---

Published as a conference paper at ICLR 2024
Moreover, conventional accuracy underestimate the capability difference between models, and self-consistency complements the drawback. For example, GPT-4, which is recognized to be sig- nificantly more capable than GPT-3.5, reports Pass @ score of 74.8 on HumanEvalPlus, which only a 4.2% relative improvement compared to GPT-3.5. However, GPT-4 is significantly more self-consistent. It achieves an SCs score of 76.1, which is 51.2% higher than GPT-3.5, highlighting that there is non-trivial capability gap between GPT-4 and GPT-3.5.
6.2 EFFECTIVENESS OF TOM SCORE
To show the effectiveness of TOM, we excerpt a 1-step chain (nlo. pla, nl1, xl1,pp1) from an Identity- Chain experiment of GPT-3.5, and gathered human-judged ground truth of whether nl is semanti- cally identical pla i.e. sem(pla) sem(nn))
Metrie
p
Metric
P
BLEU
.285
.275
.227
EM
.285
.285
.285
ROUGE-L
.254
.239
.196
CodeBLEU
.179
.173
.144
chrF
.271
.256
.210
BERTScore
.381
.389
.319
P/M
.294
.292
.292
TOM
.461
.454
.410
TOM
.500
.482
.445
Table 2:
Pearson (r), Spearman (p), and Kendall-Tau (r) correlations with human-judged ground truth of whether plo is semantically identical to nl1, the mooeel-generaeed docstring for pla-
TOM is Effective for Self-Consistency Evaluation. We compared TOM to two static PL space metrics: Exact Match (EM) and CodeBLEU and the naive dynamic metric Pass/Faill Match (P/FM) using Pearson (r), Spearman (p), and Kendall-Tau (T) correlations with human judgment in Ta- ble 2 Recall our conjectures in Section 4.1 PL space metrics can approximate the truth value of sem)(p)) = sem(nu), ). dynamic metrics are better than static ones, and the fine-grained TOM score better than naive P/FM. All three conjectures are verified in this experiment.
Within the same experiment setting, we compare TOM with four NL space metrics: BLEU, ROUGE-L, chrF, and BERTScore Table Note that for this comparison, the correlations are only computed on 117 out of the total 163 problems where plo passes all test cases. Otherwise, plo is not semantically the same as rlg. we can't use rdo as ground truth reference for nl, to calculate those NL space metrics. We show that TOM outperforms all NL space metrics given that their ground truth references exist, not to mention that TOM works well for the remaining 46 problems, for which the ground truth references absent.
TOM is Effective for PL-to-NL Evaluation.
6.3 EFFICIENCY OF GREEDY DECODING
Greedy Decoding Efficiently Evaluates Self-Consistency. We find that using greedy decoding,
GPT4
CL11SE - Starchat
6PF4
CL- -
StaChat
5P1
+ SLB
80
IdentityChain efficiently reveals most not self-consistent cases
70
within the initial three steps. Fig- ure 2 shows an evident decline
50
60
40
01
of both SC; and SSC; scores within the first three steps. After that, models stabilize or show only minimal decreases in their (strong) self-consistency scores, which underscores the efficiency of IdentityChain as an evaluation tool. Although set the chain length to five in our experiments, for mode developers and researchers with tighter time limits or computing resources, it's reasonable to choose shorter chain length when using greedy decoding.
Step
Step
Figure 2: SSC; and SC, at Computed Each Step i.
Different from the grading policy used in Chen et al. (2021). which ignores incorrect input-output examples the generated docstrings, we consider docstring with correct description but wrong examples still wrong.
8

---

Published as a conference paper at ICLR 2024
Instruction-tuned Models
Foundation Models
Greedy Decoding Results Gen- eralizable to Different Tempera- tures. To show the generalizabil- ity of greedy decoding results, we additionally evaluate the SCs score at four different temperatures. As illustrated in Figure while the SC; scores of all models decrease as the temperature increases, their relative rankings mostly remain i.e. more self-consistent models are ways more self-consistent regardless of temperature, which shows that the greedy decoding results are indeed generalizable. Moreover, it is reasonable that the absolute self-coonistency of all models drops the temperature increases. There is always a balance between exploration, which introduces novel solutions but weakens self- consistency, and exploitation, which ensures self-consistency but may overlook novel solutions.
14
.0
CL-inst-30 CL-30--78
20
SCR-18
4
0.0
Figure SCş Evaluated at Different Temperatures.
From our observations, greedy decoding both more efficient and more appropriate for self- consistency evaluation. See Appendix for the SSC, scores evaluated different temperatures.
6.4 IDENTITYCHAIN AS A MODEL DEBUGGING TOOL.
Evaluating Code LLMs with IdentityChain we can easily pinpoint the cases where the model is not self-consistent. Studying these consself-consistent cases, we identify three major weaknesses of current models in code understanding, which are not captured by accuracy-oriented evaluations.
Code LLMs Have Weak Sense of Data Types. We observe that Code LLMs are not sensitive to data types. all programming languages, data type is a fundamental element that specifies how variables should be stored, manipulated, and interacted with. However, we find that current models tend to overlook data type information. We show such an example in Appendix Figuref6 Inaccurate interpretations of data types will inevitably result in erroneous code usage. In real software develop- ment scenarios, can lead to intricate issues like memory management, performance bottlenecks, unexpected behaviors during code execution (Ding et al. 2022 He & Vechev 2023).
Code LLMs Have Weak Sense of Implicit Code Semantics. We observe that Code LLMs cannot accurately capture the implicit code semantics, which is major root cause of non-seff-consisteney Current models tend to only capture the shallow semantics that are explicitly presented in the pro- gram while overlooking the implicit logic. For example, they tend only summarize the explicit if-checks while ignoring the implicit else-branch. We show two concrete examples in Appendix Figure 7 Ignoring implicit code semantics during PL-to-NL Generation will unavoidably result in misleading or ambiguous documentation in real development scenarios.
Code LLMs Have Weak Sense of Code Execution. We also observe that Code LLMs cannot ac- curately predict the execution outcomes Austin et al. (2021): Ding et al. (2023a). Specifically, when instructed to summarize programs, the models often generate correct natural language specifications but incorrect input-output examples. We show two concrete examples in Appendix Figure [ This weakness is particularly concerning we want to generate test cases guide the entire software development process (Test-Driven Development), which underscores the importance of aligning the models' PL-to-NL Generation ability with their understanding of code execution.
7 CONCLUSION
In conclusion, we reveal that different from accuracy, self-consistency indeed a crucial missing link in current evaluations of Code LLMs, and IdentityChain effectively and efficiently bridges the gap. More importantly, IdentityChain can be used not only as a holistic evaluation tool but also model debugging tool that helps model developers study weaknesses in their models and thus potentially inspire future improvements. See Appendix (A]for future directions that potentially extend the scope of self-consistency evaluation improve current Code LLMs using IdentityChain.

---

Published as a conference paper at ICLR 2024
ACKNOWLEDGEMENT
We would like to thank Qianyu Gu, Nan Jiang, and Sophia Su for valuable discussions. This work was supported in part by an IBM Ph.D. Fellowship. DARPA/NIWC-Pacii N66001-21-C- 4018, NSF NNNS-185595, CNS-2247370, CCF-2221943, SCF-2313055, CCF-1845899 and CCF- 2107405. Any opinions, findings, conclusions, or recommendations expressed herein are those of the authors and do not necessarily reflect those of IBM, DARPA, or NSF.
REFERENCES
Wasi Ahmad, Saikat Chakraborty, Baishakhi Ray, and Kai-Wec Reii Chang. Unified pre-training for program understanding and generation. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguisties: Human Language Tech- nologies, pp. 2655-2668, Online, June 2021. Association for Computational Linguistics. URL https: www www../eb..rr///ttoooooo////////a
naacl main.211 211
Akari Asai and Hannaneh Hajishirzi. Logic-guided data augmentation and regularization for con- sistent question answering, 2020.
Jacob Austin, Augustus Odena, Maxwell I. Nye, Maarten Bosma, Henryk Michalewski, David Dohan, Ellen Jiang, Carrie J. Cai, Michael Terry, Quoc V. Le, and Charles Sutton. Pro- gram synthesis with large language models. CoRR, abs/2108.07733 2021. URL https:
7/arxiv. wwwpv//.rrggbbs//2ss.ooo
Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhari- wal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Jeffrey Wu, Clemens Winter, Christopher Hesse, Mark Chen, Eric Sigler. Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei. Language models are few-shot learners, 2020.
Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Pondé Oliveira Pinto, Jared Kaplan, Harrison Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, Alex Ray, Raul Puri, Gretchen Krueger, Michael Petrov, Heidy Khlaaf, Girish Sastry, Pamela Mishkin, Brooke Chan, Scott Gray, Nick Ryder, Mikhail Pavlov, Alethea Power, Lukasz Kaiser, Mohammad Bavar- ian, Clemens Winter, Philippe Tillet, Felipe Petroski Such, Dave Cummings, Matthias Plap- pert, Fotios Chantzis, Elizabeth Barnes, Ariel Herbert- Voss, William Hebgen Guss, Alex Nichol, Alex Paino, Nikolas Tezak, Jie Tang, Igor Babuschkin, Suchir Balaji. Shantanu Jain, William Saunders, Christopher Hesse, Andrew N. Carr, Jan Leike, Joshua Achiam, Vedant Misra, Evan Morikawa, Alec Radford, Matthew Knight, Miles Brundage, Mira Murati, Katie Mayer, Pe- Welinder, Bob McGrew, Dario Amodei, Sam McCandlish, Ilya Sutskever, and Wojeiech Zaremba. Evaluating large language models trained on code. CoRR, abs/2107.03374 2021. URL https::/taaatt /atp:i//axivvvv/ org/ abs/2107 03374
Yangruibo Ding, Luca Buratti, Saurabh Pujar, Alessandro Morari, Baishakhi Ray, and Saikat Chakraborty. Towards learning (dis)-similarity of source code from program contrasts. In Pro- ceedings of the 60th Anmual Meeting of the Association for Computational Linguistics (Volume J: Long Pupers), pp. 6300-6312, Dublin, Ireland, May 2022. Association for Computational Lin- guistics. doi:
aon.-oon..436... 436. URL / anthology. org/2022.
436
Yangruibo Ding, Ben Steenhoek, Kexin Pei, Gail Kaiser, Wei Le, and Baishakhi Ray. Traced: Execution- aware pre-training for source code, 2023a.
Yangruibo Ding, Zijian Wang, Wasi Uddin Ahmad, Murali Krishna Ramanathan, Ramesh Nallapati, Parminder Bhatia, Dan Roth, and Bing Xiang. Cocomic: Code completion by jointly modeling in-file and cross-file file context, 2023b.
GitHub. Github copilot x: The ai powered developer experience b hwwp:p/wwoo..toooooooooeoeeeeeee
U083.URLLER httpa//eeeooooouattttto///////a
deve exw.rrddddeeeeeepre-eeeeeeeeeeee
10

---

Published as a conference paper at ICLR 2024
Jingxuan He and Martin Vechev. Large language models for code: Security hardening and adver- sarial testing. In Proceedings of the 2023 ACM SIGSAC Conference on Computer and Communi- cations Security (CCS 23). 2023.
Dan Hendrycks, Steven Basart, Saurav Kadavath, Mantas Mazeika, Akul Arora, Ethan Guo, Collin Burns, Samir Puranik, Horace He, Dawn Song, and Jacob Steinhardt. Measuring coding challenge competence with apps, 2021.
Srinivasan Iyer, Ioannis Konstas, Alvin Cheung, and Luke Zettlemoyer. Mapping language to code in programmatic context, 2018.
Weisen Jiang, Han Shi, Longhui Yu, Zhengying Liu, Yu Zhang, Zhenguo Li, and James T. Kwok. Forward-backward reasoning in large language models for verification, 2023.
Raymond Li, Loubna Ben Allal, Yangtian Zi, Niklas Muennighoff, Denis Kocetkov, Chenghao Mou, Marc Marone, Christopher Akiki, Jia Li, Jenny Chim, Qian Liu, Evgenii Zheltonozhskii, Terry Yue Zhuo, Thomas Wang. Olivier Dehaene, Mishig Davaadorj, Joel Lamy-Poirier, João Monteiro, Oleh Shliazhko, Nicolas Gontier, Nicholas Meade, Armel Zebaze, Ming-Ho Yee, Lo- gesh Kumar Umapathi, Jian Zhu, Benjamin Lipkin, Muhtasham Oblokulov. Zhiruo Wang, Rudra Murthy, Jason Stillerman, Siva Sankalp Patel, Dmitry Abulkhanov, Marco Zocca, Manan Dey, Zhihan Zhang. Nour Fahmy, Urvashi Bhattacharyya, Wenhao Yu, Swayam Singh, Sasha Luc- cioni, Paulo Villegas, Maxim Kunakov, Fedor Zhdanov, Manuel Romero, Tony Lee, Nadav Timor, Jennifer Ding, Claire Schlesinger, Hailey Schoelkopf, Jan Ebert, Tri Dao, Mayank Mishra, Alex Gu, Jennifer Robinson, Carolyn Jane Anderson, Brendan Dolan-Gavitt, Danish Contractor, Siva Reddy, Daniel Fried, Dzmitry Bahdanau, Yacine Jernite, Carlos Muñoz Ferrandis, Sean Hughes, Thomas Wolf, Arjun Guha, Leandro von Werra, and Harm de Vries. Starcoder: may the source be with you!, 2023.
Tao Li, Vivek Gupta, Maitrey Mehta, and Vivek Srikumar. A logic-driven framework for consistency of neural models, 2019.
Yujia Li, David Choi, Junyoung Chung, Nate Kushman, Julian Schrittwieser, Ré mi Leblond, Tom Eccles, James Keeling, Felix Gimeno, Agustin Dal Lago, Thomas Hubert, Peter Choy, Cyprien de Masson d' Autume, Igor Babuschkin, Xinyun Chen, Po-Sen Huang, Johannes Welbl, Sven Gowal, Alexey Cherepanov, James Molloy, Daniel J. Mankowitz, Esme Sutherland Robson, Push- meet Kohli, Nando de Freitas, Koray Kavukcuoglu, and Oriol Vinyals. Competition-leveè code generation with AlphaCode. Science, 1786662) dec 2022. doi: 1061/266sc.. abq1158
abq1158. URL
Chin-Yes Yew Lin. ROUGE: package for automatic evaluation of summaries. In Text Summarization Branches Out, pp. 74-81, Barcelona, Spain, July 2004. Association for Computational Linguis- tics. URL https: //as:://claaaa www.elannoooooyooooooooooooooo /w0 1003
Jiawei Liu, Chunqiu Steven Xia, Yuyao Wang, and Lingming Zhang. Is your code generated by chatgpt really correct? rigorous evaluation of large language models for code generation. arXiv preprint ww.iiv:3055002//0 2023.
Shuai Lu, Daya Guo, Shuo Ren, Junjie Huang, Alexey Svyatkovskiy, Ambrosio Blanco, Colin Clement, Dawn Drain, Daxin Jiang, Duyu Tang, Ge Li, Lidong Zhou, Linjun Shou, Long Zhou, Michele Tufano, MING GONG, Ming Zhou, Nan Duan, Neel Sundaresan, Shao Kun Deng, Shengyu Fu, and Shujie LIU, CodeXGLUE: A machine learning benchmark dataset for code understanding and generation. In Thirty-fftth Conference on Neural Information Processing Sys- tems Datasets and Benchmarks Track (Round 1), 2021. URL BBlnttpss///peeeeveeet net/
forum? 61
Pasquale Minervini and Sebastian Riedel. Adversarially regularising neural nli models to integrate
logical background knowledge, 2018.
Xenia Ohmer, Elia Bruni, and Dieuwke Hupkes. Separating form and meaning: Using self- consistency to quantify task understanding across multiple senses, 2023.
OpenAl. Code interpreter: An experimental chatgpt model that can use python, handle uploads and downloads, 2023. URL https: http:///waaa..ooooooo///catttgga

---

Published as a conference paper at ICLR 2024
Long Ouyang, Jeff Wu, Xu Jiang, Diogo Almeida, Carroll L. Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, John Schulman, Jacob Hilton, Fraser Kel- ton, Luke Miller, Maddie Simens, Amanda Askell, Peter Welinder, Paul Christiano, Jan Leike, and Ryan Lowe. Training language models to follow instructions with human feedback, 2022.
Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. Bleu: a method for automatic evaluation of machine translation. Proceedings of the 40th Anmual Meeting of the Associa- tion for Computational Linguistics, pp. 311-318, Philadelphia, Pennsylvania, USA, July 2002. Association for Computational Linguistics. doi: 10.3115/1033033.007333. URL https:
//aclanthol www.ooyyoor//1-11/40/0
Kexin Pei. David Bieber, Kensen Shi, Charles Sutton, and Pengcheng Yin. Can large language models reason about program invariants? In Andreas Krause, Emma Brunskill, Kyunghyun Cho, Barbara Engelhardt, Sivan Sabato, and Jonathan Scarlett (eds.), Proceedings of the 40th International Conference on Machine Learning, volume 202 of Proceedings of Machine Learning Research, pp. 27496-27520 PMLR, 23-29 Jul 2023. URL httpss//ppoceeecng..... .mlr. SIess/v202/per htm
Maja Popović. chrF: character n-gram F-score for automatic MT evaluation. In Proceedings of the Tenth Workshop on Statistical Machine Translation, pp. 392-395, Lisbon, Portugal, September 2015. Association for Computational Linguistics. doi: 10. 10.18653/11/W11-3044 URL https:
//aclannhology.ozz///
3049
Shuo Ren, Daya Guo, Shuai Lu, Long Zhou, Shujie Liu, Duyu Tang, Neel Sundaresan, Ming Zhou, Ambrosio Blanco, and Shuai Ma. Codebleu: method for automatic evaluation of code synthesis, 2020.
Baptiste Rozière, Jonas Gehring, Fabian Gloeckle, Sten Sootla, Itai Gat, Xiaoqing Ellen Tan, Yossi Adi. Jingyu Liu, Tal Remez, Jérémy Rapin, Artyom Kozhevnikov, Ivan Evtimov, Joanna Bitton, Manish Bhatt, Cristian Canton Ferrer, Aaron Grattafiori, Wenhan Xiong, Alexandre Défossez, Jade Copet, Faisal Azhar, Hugo Touvron, Louis Martin, Nicolas Usunier, Thomas Scialom, and Gabriel Synnaeve. Code Ilama: Open foundation models for code, 2023.
Gemini Team. Gemini: A family of highly capable multimodal models, 2023.
Yue Wang, Hung Le, Akhilesh Deepak Gotmare, Nghi D. Q Bui, Junnan Li, and Steven C. H. Hoi. Codet5+: Open code large language models for code understanding and generation, 2023.
Fengji Zhang, Bei Chen, Yue Zhang, Jin Liu, Daoguang Zan, Yi Mao, Jian-Guang Lou, and Weizhu Chen. Repocoder: Repository-leva code completion through iterative retrieval and generation. arXiv preprint arXiv:2303 /2500, 2023.
Tianyi Zhang, Varsha Kishore, Felix Wu, Kilian Q. Weinberger, and Yoav Artzi Bertscore: Evalu- ating text generation with bert, 2020.
Shuyan Zhou, Uri Alon, Sumit Agarwal, and Graham Neubig. Codebertscore: Evaluating code generation with pretrained models of code, 2023.
Ming Zhu, Aneesh Jain, Karthik Suresh. Roshan Ravindran, Sindhu Tipirneni, and Chandan K. Reddy. Xlcost: A benchmark dataset for cross-lingual code intelligence, 2022.
12

---

Published as a conference paper at ICLR 2024
A FUTURE WORK
Introducing PI-to-PL and NL-to-NL Generation. natural to extend the self-consistency definitions large of multiple programming and natural languages by introducing PL-to-PL Generation i.e. Code Translation and NL-to-NL Generation i.e. Machine Translation. In practice IdentityChain can be improved support more programming languages and natural languages.
Studying Weaknesses of Code LLMs. Following the three examples in Section 6.4 future work can further identify and categorize more subtle weaknesses in Code LLMs. More importantly, we encourage future work to investigate the relationship between those weaknesses and the training data. possible that the weaknesses are barely addressed within current training paradigms.
Fine-tuning Code LLMs for Better Self-Consistency. It is not yet clear how we can improve the self-consistency Code LLMs. For a model with imbalanced NL-to-PL and PL-to-NL accuracy, fine-tuning the task that the model performs worse can possibly work. For model with balanced accuracy, we might need to customize a fine-tuning dataset that contains input-output pairs gener- ated by the model itself. Many fair hypotheses be made following this line of reasoning. We encourage future work to raise more and test them accordingly.
B EXPERIMENT CONFIGURATIONS
For all models, we use greedy decoding for our main experiment in Section 6. The closed-source OpenAl models GPT-3.5 and GPT-4 are non-deterministic and there is way set them to perform greedy decoding using APIs. Therefore, we set the temperature to 0 to minimize the randomness.
For open-source models, all model checkpoints formers" from Hugging Face. Note that we downloaded Code Llama and Code Llama-Instruce from https: htw.ggggggggac..eeeo co/ co/cooelll lolelland ama instead of the link provided by Meta AL We run open-source model experiments on NVIDIA RTX A6000 GPUs with CUDA 11.3, cuDNN8- devel, PyTorch 1.12.1, and Python 3.19.9. For efficiency, set the prompt length to be 1,024 tokens, the max generation length to be 512 tokens, and the inference precision to be FP16.
downloaded using the Python library "trans-
13

---

Published as a conference paper at ICLR 2024
C REPLACING MEANINGFUL FUNCTION NAMES
HumanEval/76
ts_simple_povergs n):
""or task is to write function that true if
is simple power simple
of and false in other cases.
if n"rintex
For example: 1s__ttple__owerrL,
1s_simple_power(2, 2) => true
is_simple_power(. 2)
$s_staple_poeer3, 2) >> false
si_sippl__power(3 1) >> false
is s__seppe__ower(SS 3) false
Inaccurate
return True
for i range int(x 0.5) 1); if X 5 =z 0:
Keep Function Name
False
Replace Function Name
return True
def (ttepppp__pooer(,, n):
def Func
if x an 1: return True
if x
1:
for 1 in range(2, int(x 0.5) 1): if 3 return False
For in range(2, int(x 0.5) 1); if X 0: return False
return True
return True
Not ConsiConsitt
Self Consitensttntt
"""Check given number simple power another number. A simple power is defined number that can expressed x"n, whare x positiva intager and positive intager greater than 1. Exasples:
Check given number is prine.
>>> func(5, 2)
>>> func(18, 3)
>>9 is ss_simple__ is_sixple_power(a 2) True
>>> sixxple_oveer27, True
>>2 is_sixple_poverre p1ver____ 2) False
600
Figure 4: Replacing Meaningful Function Names with A Generic "func", Given the docstring with the original function name, GPT-3.5 generates an incorrect program that conflicts with the function name. When further summarizing that program along with the original function name, GPT-3.5 completely ignores the code and generates a new doestring based on the function name. In this case, we will falsely conclude that GPT-3.5 is not self-consistent. However, when summarizing the program along with a generie name "func" replacement, GPT-3.5 correctly captures the code semantics and thus is self-consistent w.rl. the original docstring. Therefore, when generating nl; and pla for 2 1, we replace the original meaningful function name with the generic "func".
14

---

Published as a conference paper at ICLR 2024
D GENERALIZABILLTY OF GREEDY DECODING
Instruction-tuned Models
Foundation Models
40
8
35
30
25
GPT-3.5
SCB-78
CL-Inst-138
SCB-38
CL-Inst-78
SCB-18
2
5
0.0
0.2
0.4
0.6
0.8
0.0
0.2
0.4
0.6
0.8
Temperature
Temperature
Figure 5: SSC, Evaluated Different Temperatures. Similar to the SCs results in Section 6.3 for the strong self-consistency score SSC5, the relative rankings of models mostly remain regardless of temperature i.e. more strong self-consistent models are always more strong self-consistent no matter the temperature, which confirms that greedy results are generalizable to different temperatures.
E EXAMPLES OF CODE LLMs' WEAKNESSES
GPT-3.5 Implementation for HumanEval/92
def func(x, y, 2): if isinstance(x, int) and isinstancely, int) and isinstance(z, int): if x == y + z or = x or =
return True
return False
GPT-3.5 Summarization of the Above Code
"""Check if any two numbers in the given three numbers add up to the third number. ooc func (1, 2, 3)
True
func (4, 5, 6)
False
Fail to Capture The Data Type Checking
Figure 6: Code LLMs Have Weak Sense of Data Types. The implementation checks whether all three inputs are type int the same time, but the summarization only mentions that the inputs are three "numbers" failing to capture the data type information.
15

---

Published as a conference paper at ICLR 2024
GPT-3.5 Implementation for HumanEval/69
def func(1st):
freq_dict = Q
for num lst:
if num in freq_dict:
GPT-3.5 Implementation for HumanEval 102
freq_dict [num]
esse:
def func(x, y):
freq_dict dict [nun]
max_even
max_num nun = =-1
for num range(x, y 1): num 2 == and nun max_even max max_even nun
nun, freq f dict.itens)):
1f freq s= nun and nun max nun: nun
return max_even
return max_nun
GPT-3.5 Summarization of the Above Code
GPT-3.5 Summarization of the Above Code
"*" Find the largest number in the given List that appears at least as many times its value.
nua 'in the largest nunber between two
given numbers.
func(1, 10)
func [1, 2, 3, 4, 4, 41)
10
func(5, 15)
func [1, 2, 3, 3, 3, 4, 4,
14
3
Fail Capture The Implicit Else-Branch
X
Figure 7: Code LLMs Have Weak Sense of Implicit Code Semantics. In the left example, the implementation has an implicit "else" branch that returns -1 when no even number is found. In the right example, the implementation also has an implicit "else" branch that returns -1 when no larger satisfying number found. However, both summarizations fail to capture that implicit logic.
GPT-3.5 Implementation for HumanEval/39
GPT-3.5 Implementation for HumanEval/6
det Tunc(n: int]:
def 15. (nunennnenn)):
fror typing inport List
if range int(nun 0.5)
def func(paren str) - List[int]:
result = for death
in paren_string.splat():
if nun return False
current_depte
return True fib_nuns 10, count
char in group:
char =z '':
surrent_depthh +-
dax_h nax(mox_dophh, current_dapth)
fib_nune aspend fiib_nursss_-_eee [-1] rib_nuns (fib_
elif char ')'s current_depth
if is_prine(fib. B__rinnefib_nuns, nuns [-1]): count += return rib_nuns[-1]
(nallt.appendmmxxcco depth) return result
GPT-3.5 Summarization the Above Code
GPT-3.5 Summarization of the Above Code
"""Given of parentheses, each group parentheses.
list the
-.. Find Fibonacci that also prine
func("((())) () ()")
func
[3, 1,
ase func("() () ()")
SoS [unc(5]
[1, (unc("(((0)))
func (18)
Fall to Predict Execution Output
Figure 8: Code LLMs Have Weak Sense of Code Execution. In both examples, some input-output pairs in the summarization are wrong, which means that the model fails to predict execution.
16


Reviews:
[{"review_id": "xaWhJnumK2", "reviewer": "Reviewer_G8dX", "summary": "This paper presents a concept of self-consistency for Code LLMs and proposes a framework, namely IdentityChain, to evaluate their self-consistency. Using IdentityChain, the paper studies the self-consistency of a series of CLMs beyond the accuracy based on the HumanEval+ and MBPP benchmarks and reveals some inconsistent behaviors of those models.", "questions": "At the end of Page 2, it is said that “the underlying semantics of pl1, nl1, pl0 should all be the same. We call such a behavioral property self-consistency”, but in the next paragraph, the authors write “a model’s accuracy is low, its self-consistency can remain high when the model, despite making initial errors, consistently maintains those errors”. Please clarify.\n\n“it relatively improves GPT-3.5 by 57.5% in FPS-5” What is FPS-5 here?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "Leveraging unique properties of code such as executability and dynamic engine feedback, this paper formalizes a definition of self-consistency for CLMs. The consistency is interpreted on the NL-PL-NL chain, unlike recent work on self-consistency such as Elazar et al. (2021), which defines the consistency as invariance of LLMs to semantic preserving transformations of the prompts.\n\nThe authors propose IdentityChain, a framework for self-consistency evaluation with a new metric, named Test Output Match (TOM) score. TOM is based on matching binary outcome of dynamic test execution as well as syntax and runtime errors.", "weaknesses": "There are some weaknesses in the self-consistency formulation:\n\n1. We could construct a perfectly self-consistent and not-so-useful CLM. For example, a CLM could generate summarization nl_1 exactly the same as pl_0. It is likely that the model generates pl_1 semantically equivalent to the input pl_0 in the next step. The chain goes on and the model is mostly perfectly self-consistent. Based on that, we could have very high self-consistency but that does not say much about the model.\n\n2. An important aspect of consistency is the consistent behavior to transformations of the prompts with equivalent semantics. This paper does not touch upon it.\n\n3. While the concept and the proposed framework sound general to any task, it seems to only apply to code synthesis and summarization. What about other tasks for code such as reasoning about code, fixing bugs?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "Jih3tJjxMq", "reviewer": "Reviewer_d4Cs", "summary": "The paper focuses on evaluating the self-consistency of Code Large Language Models (Code LLMs). The authors stress the importance of self-consistency in these models as they might generate varying outputs for semantically similar inputs. They introduce an evaluation framework, IdentityChain, which helps in understanding the model's vulnerabilities and improves the architecture of Code LLMs. Furthermore, they discuss various existing evaluation metrics for Code LLMs, emphasizing that many of these do not consider the model's self-consistency. By using their framework, they aim to provide better insight into the models and improve the training or architectural design of Code LLMs in the future.", "questions": "How does the IdentityChain framework compare in terms of efficiency and accuracy with other existing evaluation tools for Code LLMs?\nAre there any plans to extend the research to encompass other types of LLMs or machine learning models?\nHow can the IdentityChain framework be integrated into the current training methodologies of Code LLMs for better results?", "limitations": "", "rating": 8, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 4, "strengths": "Originality: The paper addresses the lesser-explored topic of self-consistency in Code LLMs. The introduction of the IdentityChain framework showcases an original approach to understanding the internal workings and vulnerabilities of such models.\n\nQuality: The paper cites multiple sources and provides a comprehensive review of the existing literature, giving it a solid foundation. The work is well-researched and backed by evidence.\n\nClarity: The paper is structured logically, with a clear progression from the introduction of the problem to the presentation of the solution. The content is presented in an organized manner, making it relatively easy to understand for readers.\n\nSignificance: Addressing self-consistency in Code LLMs has potential implications for the future development and deployment of such models, making the research both timely and significant.", "weaknesses": "Generalization: The paper specifically discusses Code LLMs, and it would be useful to understand how the principles and findings apply to other LLMs or different types of models.\n\nComparison with Existing Solutions: While the paper reviews existing evaluation metrics, a direct comparison in terms of performance or effectiveness with the proposed solution might have added more clarity.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "fewkfGg44W", "reviewer": "Reviewer_6ee8", "summary": "The paper proposes to evaluate code LLMs with respect to self-consistency (including documentation for code and code). To this end they perform a formal definition and evaluate multiple models. The paper shows that self-consistency is a concerning aspect.\n\nThis behavior is highly anticipated as it is known for LLM in general. Thus, the research question, while worth asking, is already quite likely to yield only limited insights. The authors would have to motivate much more, why different behavior could be anticipated and then carefully analyze to what extent this reasoning holds (or does not). In the intro the paper does present first steps in this direction though lacking reasoning, i.e., the mere fact that a docstring does not allow to generate code but a natural language completion does is interesting, but per se not surprising. Maybe, it is due to training data? Maybe due to sth else? The authors should carefully think about such questions. In the analysis some findings are not sufficiently explained or might be spurious or follow general known patterns (\"LLMs don't understand (code) semantics\"). A positive example is the lack of awareness of data types, which is specific to code. Finding multiple specific aspects and then generalizing to known model behavior facts is significantly more interesting as it provides deeper insights. In summary, the paper lacks depth and addresses a question that from the outset is of some but not much interest given existing knowledge on LLMs.\n\nDetailed comments:\n*  For example, we observe that stacking more parameters does not necessarily guarantee the improvement of self-consistency: while StarCoderBase-7B performs worse in Pass@1 for both benchmarks than StarCoderBase-13B, the former outperforms the latter after five self-iterations\n This is a single example, no statistical analysis. Also in general large models seem to perform better. So this finding is of very limited relevance. Looking at performance of LLMs there is often some variation.", "questions": "see above", "limitations": "", "rating": 3, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 1, "strengths": "see above", "weaknesses": "see above", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "mF9hZYj0P6", "reviewer": "Reviewer_AByS", "summary": "This paper presents a novel evaluation framework called IdentityChain which is designed mainly to evaluate self-consistency. Basically, given some initial natural language specification ($nl_0$), a model ($M$) will first generate code ($pl_0$). Then, the $M$ generates a natural language summary ($nl_1$), given only ($pl_0$). Now, using only $nl_1$, $M$ generates the corresponding code ($pl_1$). So, there is effectively a chain: $nl_0 \\rightarrow pl_0 \\rightarrow nl_1 \\rightarrow pl_1$. Their framework measures semantic equivalence, where the self-consistency condition is met if $sem(pl_0) = sem(nl_1) = sem(pl_1)$, and the \\textit{strong} self-consistency condition is met if $sem(nl_0) = sem(pl_0) = sem(nl_1) = sem(pl_1)$. This corresponds to a chain of length 1, but this could be extended to an arbitrary length (they use 5 in this paper). For measuring semantic equivalence $sem(pl_i) = sem(pl_{i+1})$, they introduce a metric called Test Output Match (TOM) in which they check whether the exact output of $pl_i$ and $pl_{i+1}$ are the same for each of the tests in the given test suite. They argue that this also implicitly evaluates $sem(pl_i) = sem(nl_{i+1})$. They conduct experiments using HumanEvalPlus and MBPP, using 11 different code LLMs, encompassing both foundational models as well as instruction-tuned models using greedy decoding. Their results show that even though code LLMs can achieve impressive initial accuracy (pass@1), they achieve relatively lower scores on self-consistency, especially as the number of iterations increase, and in fact, accuracy does not always correlate with self-consistency. They demonstrate that IdentityChain can be used for efficiently finding flaws at each step by highlighting a few of their findings from such experiments. They also demonstrate that self-consistency is better correlated with human judgment compared to other common static and dynamic metrics.", "questions": "1) Please comment on the points raised in the section above.\n2) The results shown in Table 2 are unclear to me. Based on the description, it seems that the evaluation is on $sem(pl_0) - sem(nl_1)$. Therefore, it seems that each metric is computed based on $score(pl_0$, $nl_1)$. However, one of these is code and the other is natural language, so computing metrics like exact match on this would not make sense. Could you please clarify the details here?\n3) In this framework, $pl_0$ is generated from a human-written specification $nl_0$. Since $nl_1$ is model-generated and not human-written, is it fair to expect that $nl_1 \\rightarrow pl_1$ generates code that is similar to $pl_0$? Isn't it possible that the distributions are different?\n4) In this paper, you focus on doing NL-to-PL first, which is natural since that is how HumanEval and MBPP are designed. Have you considered starting with PL-to-NL? Would you expect to see similar results?\n\n\nSome minor comments:\n- I would suggest re-considering calling this idea \"self-consistency\" as this term has been used to describe a different principle with LLMs: https://arxiv.org/abs/2203.11171.\n- Section 5.3: \"fin-grained\" should be \"fine-grained\"\n- Last line of page 6: \"StarCoderBase-13B\" is inconsistent with Table 1", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "- This notion of consistency is very interesting and very useful for evaluation. The idea of using the Test Output Match for approximating semantic equivalence is also quite neat, and something particularly useful for evaluating PL-to-NL since existing metrics are not great for this.\n- The analysis in Figure 2 for understanding how performance degrades as the number of iterations increases is quite interesting. The authors have also enumerated a number of findings related to weaknesses of code LLMs from their own use of this framework which can inspire future work in this space.\n- The authors have conducted very thorough experiments and have in-depth evaluation, even demonstrating correlation with human judgment (though some of the details of this are unclear). This underlines the value of the evaluation framework they have introduced.", "weaknesses": "- The IdentityChain framework assumes that the underlying code model performs well at both tasks (NL-to-PL and PL-to-NL). It is possible that the model is good at one and not the other, and it is not possible to decouple these in their framework.\n- The authors introduce the idea in a very general manner, in which the self-consistency idea could be applied between many different tasks. However, it is important to note that this approach can only be applied to a pair of symmetric tasks. More concretely, the two tasks they focus on in this paper are NL-to-PL and PL-to-NL, and these assume that the natural language and code sufficiently and holistically capture one another. These assumptions would not be valid for certain types of tasks, such as summarization tasks, in which a very brief summary with a high-level overview is to be generated for a given input. In such cases, the summary would not sufficiently and holistically capture the input in such a way that something semantically equivalent to the input could be re-generated given only the summary. \n- Based on my previous two points, it seems that the code summarization task (PL-to-NL) may not be symmetric to the (NL-to-PL) task. Namely, in $p_i \\rightarrow n_{i+1}$, the model could generate a high-level natural language summary rather than a holistic description of $p_{i}$ with sufficient details for for generating $n_{i+1} \\rightarrow p_{i+1}$ , with the expectation that $p_{i+1}$ would be semantically equivalent to  $p_{i}$. It is likely that the pretraining data for these code models include a lot of natural language with high-level descriptions of accompanying code (e.g., docstrings, NL StackOverflow test near code snippets). If this is the case, then the model is likely not great at generating holistic natural language descriptions for code. Therefore, it is possible that the model is very good at NL-to-PL (as per the high accuracy scores) but not as good at PL-to-NL, which results in low self-consistency scores. However, since it is not clear how to decouple this in the IdentityChain, it is difficult to clearly see this.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (poster)