Enhancing Large Language Models in Coding Through Multi-Perspective Self-Consistency (2024)

Abstract: Large language models (LLMs) have exhibited remarkable ability in code generation. However, generating the correct solution in a single attempt still remains a challenge. Prior works utilize verification properties in software engineering to verify and re-rank solutions in a majority voting manner. But the assumption behind them that generated verification properties have better qualities than solutions may not always hold. In this paper, we treat them equally as different perspectives of LLMs’ reasoning processes. We propose the Multi-Perspective Self-Consistency (MPSC) framework incorporating both interand intraconsistency across outputs from multiple perspectives. Specifically, we prompt LLMs to generate diverse outputs from three perspectives, Solution, Specification and Test case, constructing a 3-partite graph. With two measure functions of consistency, we embed both interand intra-consistency information into the graph. The optimal choice of solutions is then determined based on analysis in the graph. MPSC significantly boosts performance of foundation models (ChatGPT in this paper) on various benchmarks, including HumanEval (+15.91%), MBPP (+6.43%) and CodeContests (+9.37%), even surpassing GPT-4.1

heading: 1 Introduction
text: In recent years, pre-trained large language models (LLMs) have demonstrated unprecedented proficiency in understanding, generating, and reasoning with human language (Brown et al., 2020; Chowdhery et al., 2022; OpenAI, 2023; Touvron et al., 2023). Among the diverse applications of LLMs, code generation stands out as pivotal task and has been acknowledged as a fundamental task for benchmarking (Liang et al., 2023). This task entails
∗Work done during internship at Microsoft. †Corresponding author.
1 The code is available at https://github.com/skpig/ MPSC.
models to generate source codes from provided natural language intents. Many foundation models have exhibited remarkable zero-shot performance in code generation, such as ChatGPT and GPT4 (OpenAI, 2023), with successful deployments in real-world applications like Github Copilot.
Despite the remarkable abilities, LLMs often struggle to generate the correct code in a single attempt. Therefore, previous works sample diverse codes from LLMs and re-rank them by introducing verification properties from software engineering. For example, CodeT (Chen et al., 2022) generates test cases as a verification property, while ALGO (Zhang et al., 2023) generates oracles, the brute-force version of desired algorithms, as a verification property. These methods are essentially variants of majority voting, making the assumption that the correctness of experts (i.e. the verification properties) is better than that of choices (i.e. the desired code outputs). However, both verification properties and desired code outputs are usually generated by the identical model with respect to the same question, and hence the preference over verification properties is not always correct.
Instead, we believe that both desired code outputs and verification properties should be treated equally, since they are different perspectives of
1429
LLM’s deliberate thinking process in face of identical questions. Aggregating various outputs from different perspectives can lead to a more credible result. To achieve this, we propose the MultiPerspective Self-Consistency (MPSC) framework that incorporates both inter-consistency across multiple perspectives and intra-consistency within a single perspective. In this way, MPSC can fully leverage the consistency information within LLMs and select the model output with the most consistent functionality as the final answer.
In our framework, various verification techniques from software engineering can be included as extended perspectives for better reasoning. Specifically, we prompt the LLM to simultaneously generate diverse outputs from three wellestablished perspectives in software engineering, namely Solution, Specification and Test case (Abrahamsson et al., 2017). Solutions implement the desired functionality, specifications demonstrate the intended properties in formal language, while test cases outline the expected behavior for some specific inputs. Then, we treat these model outputs as vertices in a graph, and establish connections (i.e. edges) based on the pairwise agreement of vertices from different perspectives. Our goal is to identify the most reliable output using a score function, which evaluates all vertices by considering both intra- and inter-consistency information encoded in the graph. Specifically, the intra-consistency information guides the function to favor the most internally consistent output within a single perspective, while inter-consistency ensures that the scores for two outputs from different perspectives are similar if they reach a consensus. We formalize the learning process of the score function as an optimization problem adhering to these two consistency criteria and leverage an iterative algorithm proposed by Zhou et al. (2003b) to achieve this goal.
We evaluate MPSC on four widely used code generation benchmarks, including HumanEval (Chen et al., 2021), HumanEval+ (Liu et al., 2023), MBPP (Austin et al., 2021) and CodeContests (Li et al., 2022). Experimental results show that our method boosts the performance of ChatGPT by a large margin, 15.91% in HumanEval, 15.64% in HumanEval+, 6.43% in MBPP and 9.37% in CodeContests. Our framework even surpasses GPT-4 (OpenAI, 2023) as shown in Figure 1.

heading: 2 Multi-Perspective Self-Consistency
text: A single perspective can often lead to an incomplete understanding of a problem, akin to the parable of “blind men describing an elephant". The reasoning process of LLMs follows a similar pattern. LLMs generally cannot guarantee the correctness of generated output in a single attempt, especially in code generation, which necessitates proficient natural language understanding, deliberate reasoning and rigorous controllability.
However, a key aspect of human intelligence is the ability to think from multiple perspectives, resulting in a more comprehensive understanding of situations and more accurate solutions to problems. Inspired by the human cognition process, we propose a novel code generation method by reasoning from three well-established perspectives, solutions, specifications and test cases. Although noisy outputs may inevitably be included in the generated outputs of every perspective, we can leverage both intra-consistency and inter-consistency among the diverse outputs to distinguish the best ones from the noise. An overview of our proposed MPSC method is illustrated in Figure 2.

heading: 2.1 Solution, Specification and Test Case
text: Given a user intent in natural language, we introduce solution, specification and test case as three perspectives to describe the desired functionality. A solution is the source code implementing the functionality denoted as g : X → Y, which is also the target of code generation. A test case is a pair of valid input and output satisfying the required functionality denoted as (x, y) ∈ X × Y. Specification draws inspiration from Formal Verification in software engineering, which mathematically proves the correctness of one program by ensuring its satisfaction of some certain formal specifications. In the context of software engineering, formal verification is usually written in formal programming languages, e.g. Coq (Team, 2023) and Dafny (Leino, 2010), and conducted by accompanying verification tools. For the generalization of the proposed method in different program language scenarios, we adopt the idea of formal verification and limit the specifications within pre-conditions and post-conditions, which can be written as functions in the same programming language like solutions, without struggling with formal languages. Specifically, a pre-condition constrains the requirements that a valid input should satisfy, while a
post-condition constrains the relationships that a pair of valid inputs and outputs should satisfy. We denote them as hpre : X → {False, True} and hpost : X× Y→ {False, True}. Detailed examples of outputs are shown in Figure 2.

heading: 2.2 Graph Construction
text: We require LLMs to generate diverse outputs from all three perspectives. We employ an 3-partite graph representation to capture the relationships among these generated outputs. Specifically, we represent the generated solutions {g1, g2, ..., gI} with a vertex set V func, the specification set {(hpre1 , hpost1 ), ..., (hpreJ , h post J )} with V spec, the test case set {(x1, y1), ..., (xK , yK)} with V test, and hence construct a vertex set V = V func ∪ V spec ∪ V test. With edges connecting each pair of vertices from two distinct sets, we construct an undirected 3-partite graph G = (V,E). Our goal is to leverage the graph to encode the multiperspective consistency information, and then learn a score function f : V → R (also a vector f , fi = f(vi)) from it to choose the most reliable output among all.

heading: 2.3 Inter-Consistency between Different Perspectives
text: We distinguish between two kinds of consistency based on the perspectives involved. Intraconsistency is defined as the degree to which a given output aligns with others within the same perspective, following the original definition in Wang et al. (2022). Conversely, inter-consistency is defined as the degree of consensus between a pair of outputs from two different perspectives.
With the well-established definitions of these three perspectives in software engineering, each output implicitly describes a latent functionality regardless of whether it is a solution, a specification or a test case. Consequently, we define the inter-consistency between two vertices from different perspectives as the alignment of their latent functionalities. And the most appealing aspect is that we can quantify the alignments with a code interpreter in a deterministic manner2. We formalize the inter-consistency as a measure function ω(·, ·) : V × V → R (also the adjacency matrix
2We provide the Python code snippets implementing the verification in Appendix B.
W , where Wi,j = ω(vi, vj)) to weight different edges in different ways as shown in Table 1.
We then derive an optimization objective based on inter-consistency measurements,
Linter = ∑
(vi,vj)∈E Wi,j(f(vi)− f(vj))2 = fTLf (1)
, where L = D −W is the laplacian matrix of the graph G3. The loss function is the weighted sum of the local variation of each edge on the graph. An underlying assumption is that a pair of outputs exhibiting consistency are either both correct or both incorrect. Therefore, the difference between scores of two connected vertices should be constrained by the penalty corresponding to the degree of consistency, i.e. edge weight.

heading: 2.4 Intra-Consistency within the Same Perspective
text: Following Wang et al. (2022), we define the intraconsistency of one generated output as its similarity to others within the same perspective, which is denoted as a function φ(·) : V → R (also a vector y, yi = φ(vi)).
Wang et al. (2022) limits the consistency to mere equalities in final answers, thereby lacking efficacy when applied to open-form tasks. In the scenario of code generation, we extend the scope of intraconsistency to lexical and semantic similarities.
Lexical intra-consistency by Bayes risk. Minimum Bayes risk decoding (Kumar and Byrne, 2004) selects the hypothesis h ∈ H that minimizes the expected loss R(h) = Ey∼P (y)[L(y, h)] over the distribution of label y. Because of the unavailability of P (y), P (h) is usually used as a proxy distribution in practice. Then the Bayes risk can be rewritten as R(h) = ∑ h′∈H L(h
′, h) ·P (h′), which is in fact measure the consistency of h over the hypothesis space. Specifically, we utilize negative
3In our experiment, we use the symmetric normalized Laplacian Lsym = D− 1 2LD− 1 2 for more robust performance.
BLEU metrics (Papineni et al., 2002) as the loss function L aiming at lexical similarity and assume the hypothesis distribution is uniform, i.e.
φ(vi) = C · ∑
vj∈K(vi) BLEU(vi, vj)
, where C is the normalizing constant so that measures of outputs in one perspective sum up to 1, K(vi) represents the other outputs within the same perspective as vi.
Semantic intra-consistency by structural equivalence. In the realm of graph theory, two vertices are deemed structurally equivalent if they share identical weighted connections with the same thirdparty vertices. Utilizing this equivalence relation, we delineate V func, V spec, and V test into their respective structural equivalence classes. Noted that the weights assigned to edges reflect the alignments of latent functionalities associated with the vertices, and hence outputs within each equivalence class can be regarded as exhibiting consistent functional behavior. Therefore, we define the intraconsistency measure based on the structural equivalence classes within each perspective. Suppose vi belongs to the solution set V func. The structural equivalence class of vi is denoted as S(vi) ⊂ V func, and neighbor sets of vi can be partitioned into two subsets {Nt(vi)|t ∈ {spec, test}} depending on the perspective they belong to. Overall, the lexical intra-consistency measure is defined as the multiplication of these three sets, i.e.
φ(vi) = C · |S(vi)| · ∏
t
|Nt(vi)|
, where C is the normalizing constant. The notation is similar when vi belongs to other perspectives.
Intra-consistency is in fact an estimate of the LLM’s uncertainty (Kuhn et al., 2023; Xiong et al., 2023b) and reflects the confidence of the model on one specific output. Therefore, we can utilize the intra-consistency information as a supervision signal by ensuring the closeness between the score function f and the intra-consistency measure φ with Mean Squared Error (MSE),
Lintra = 1 2
∑
vi∈V |f(vi)− φ(vi)|2 = 1 2 ||f − y||2 (2)

heading: 2.5 Optimization Formulation
text: After all, following the criteria of inter- and intraconsistency, we can then formulate the learning
process of f as an optimization problem that combines both Linter (Eq. 1) and Lintra (Eq. 2):
min f :V →R
{α · Linter + (1− α) · Lintra} (3)
To solve this optimization problem on the graph, we adopt the iterative algorithm proposed by Zhou et al. (2003b). The details of the algorithm can be found in Appendix A.

heading: 3 Experiment

heading: 3.1 Experiment Settings
text: Dataset and metrics. We conduct experiments on four widely used Python code generation benchmarks, including HumanEval, HumanEval+, MBPP and CodeContests. HumanEval (Chen et al., 2021) and MBPP (Austin et al., 2021) are two hand-written Python programming problems. HumanEval+ (Liu et al., 2023) adds more unit tests based on HumanEval. CodeContests (Li et al., 2022) is a much more challenging dataset consisting of competition problems from the Codeforces platform. The evaluation metric is Pass@k (Chen et al., 2021), which is an unbiased estimator of the probability that at least one out of the k solutions generated by the model passes unit tests. Details about the metric can be found in Appendix C.
Implementation and baselines. We compare several baselines from different LLMs for code like ChatGPT4 (i.e. GPT-3.5-Turbo), GPT-4 (OpenAI, 2023), Code Llama (Rozière et al., 2024), WizardCoder (Luo et al., 2023) and Deepseek Coder (Guo et al., 2024), to other post-hoc approaches enhancing LLMs during inference, including Selfconsistency (Wang et al., 2022), MBR-EXEC (Shi et al., 2022), CodeT (Chen et al., 2022) and Selfcollaboration (Dong et al., 2023). For both MPSC and other post-hoc augmentation approaches, we employ GPT-3.5-Turbo as the foundation model to generate 200 solutions. MPSC additionally generates 50 specifications and 100 test cases for each problem. Following the original setting in Chen et al. (2022), we additionally generate 500 test cases for other baselines.
Variants of MPSC. As shown in Eq.2, the intraconsistency measure φ(·) is essentially used as a supervision signal without leveraging the semantics of “consistency". Therefore, we include two variants in our experiments: (1) MPSC-Uniform is
4https://chat.openai.com/
the baseline without any prior intra-consistency information and treats every vertex equally, i.e. φ(vi) = C. (2) MPSC-Label includes the public example test cases in docstrings as silver labels, i.e. φ(vi) = C · 1vi is label5. Further details regarding the implementation of our method and baselines are provided in Appendix E.

heading: 3.2 Main Results
text: The experimental results on the four benchmarks are presented in Table 2. We observe that MPSC consistently enhances the code generation capabilities of the foundation model (i.e. GPT-3.5Turbo) across all benchmarks with a remarkable margin of improvement. Particularly, when k is set to 1, which is the most prevalent scenario in real-world applications, the performance improvement is notably significant (+15.91% on HumanEval, +15.64% on HumanEval+, +6.43% on MBPP and +9.37% on CodeContests). With the foundation model GPT-3.5-Turbo, our MPSC can even outperform GPT-4 in Pass@1 across all benchmarks. Compared to other post-hoc augmentation approaches, even though they utilize more test cases, our MPSC still shows consistent advantages in all benchmarks, excluding the Pass@5 score in MBPP benchmark. MPSC-Uniform serves as the bottom line of MPSC framework and still achieves great gains for the foundation model, which demonstrates that relying on inter-consistency proves to be entirely effective. Moreover, incorporating various types of intra-consistency information leads to further improvements. Specifically, MPSC-Label and MPSC-Semantic exhibit particularly strong results. They are two representative approaches leveraging the external supervision signals or the internal consistency information respectively. Surprisingly, MPSC-Semantic can match or even surpass MPSCLabel in some benchmarks, which further highlights the significance of consistency information in LLMs. Besides, we also note that the performance of MPSC-Semantic and MPSC-Lexical remains largely unchanged as k increases. This phenomenon aligns with the nature of MPSC, which assesses solutions based on their consistency within the foundation model. It implies that top-ranked solutions exhibit semantic similarity and are consistently either correct or incorrect. This reaffirms the capability of our proposed MPSC to effectively
5The number of example test cases is two in average. Noted that MBPP doesn’t provide test cases in docstrings.
aggregate consistency information within LLMs, thereby selecting the most consistent answers. We have a more detailed discussion about this phenomenon in Appendix C.

heading: 3.3 Further Analysis
text: Ablation study. We conduct an ablation study to examine the impact of different perspectives on MPSC. The results are presented in Table 4. Evidently, both the specification and test case perspectives play crucial roles in our framework. Additionally, test cases contribute more to the improvements than specifications. We attribute the observation to the better quality of test cases, as generating an accurate test case is considerably simpler than abstracting a comprehensive and sound specification.
Qualities of three perspectives. We present the accuracy6 of generated solutions, specifications and
6Accuracy is equal to pass@1.
test cases in Table 5. The quality of all three perspectives is insufficient individually. Indeed, the generated verification properties (i.e. specifications and test cases) are even poorer than the generated solutions. It implies that prior works (Zhang et al., 2023; Chen et al., 2022) relying on generated verification properties as experts for majority voting on solutions may fail, as these experts perform worse than the choices (i.e. solutions) themselves. Additionally, it indicates that the significant improvements brought by MPSC do not solely depend on the high quality of verification properties. The improvements come from the consistency information within LLMs, which helps to distinguish noise from high-quality solutions.
Generalization over different LLMs. MPSC is designed as a model-agnostic framework that assumes black-box access to the underlying foundation model. In assessing the extent of MPSC’s
generalization, we employ many other LLMs in addition to ChatGPT as foundation models. In specific, we consider the strongest model, GPT4, and three highly proficient open-source coding LLMs, Code Llama, WizardCoder and DeepSeek Coder in Python. The experimental results presented in Table 3 show that MPSC consistently yields significant improvements across all models, which demonstrates the robust generalization capabilities embedded in our proposed framework.
Impact of edge sparsity. Our framework significantly depends on the inter-consistency information between model outputs, which is represented as edges within the constructed graph. A critical question arises concerning the impact of edge spar-
sity on the framework’s efficacy. To address this, we categorize all queries in the dataset into distinct bins based on the total edge weights in their corresponding graphs and compute the perfect performance ratio (i.e. Pass@1=100) for each bin. In this experiment, we employ the MPSC-Uniform configuration7. Figure 3 illustrates the correlation between edge density and performance. The results clearly demonstrate a positive correlation between the number of edges and the overall performance of our framework.
Impact of sampling number. To examine the effect of the sampling number of different perspectives, we conduct an analysis experiment by vary-
7The Uniform configuration is utilized to eliminate the influence of intra-consistency information.
ing the numbers of generated specifications and test cases. As shown in Table 6, MPSC constantly brings significant performance gains with varied specifications and test cases. We note that MPSC generally suffers a slight degradation in performance when fewer specifications or test cases are used, which is consistent with our intuition. But the performance decline is relatively small (about 4% with only 10% of specifications and test cases retained). The observation indicts the remarkable performance and efficiency of MPSC, suggesting the potential for real-world application with reduced computational costs.

heading: 4 Related Work
text: Prompting techniques on consistency. Based on Chain-of-thought mechanism (Wei et al., 2022), previous works have adopted various prompting techniques and decoding strategies to reveal the consistency of LLM outputs and further enhance the capabilities of LLMs. One line of approaches decodes multiple times from the same perspective and aggregate the results (Wang et al., 2022; Zhou et al., 2022; Jung et al., 2022; Sun et al., 2022; Chen et al., 2023a). For example, Wang et al. (2022) targets tasks with fixed answer sets and scores each answer based on the output frequency. Building on this, Sun et al. (2022) introduces recitation as context for augmentation. While Jung et al. (2022) focus on the two-value entailment relations (True or False) between statements and explanations. They treat the inference process as a weighted MAXSAT problem and utilize a logistic solver to solve it. Another line draws inspiration from the “Dual Process" in cognitive science, which posits that human reasoning is dominated by System 1 and System 2 (Daniel, 2017; Sloman, 1996). As a result, these approaches require LLMs to play different roles like generator (i.e. System 1) and verifier (i.e. System 2), and optimize the result iteratively by a conversational way (Madaan et al., 2023; Shinn et al., 2023; Zhu et al., 2023). Xiong et al. (2023a) also proposes the concept of "inter-consistency". Instead of
referring to the consistency within the same LLM, they focus to tackle the inter-inconsistency problem between different models with a formal debate framework.
LLM for code. LLMs pretrained on large-scale code data have demonstrated strong capabilities in the field of code generation, such as Codex (Chen et al., 2021), AlphaCode (Li et al., 2022), CodeGen (Nijkamp et al., 2023), InCoder (Fried et al., 2022), StarCoder (Li et al., 2023), Code Llama (Rozière et al., 2023), WizardCoder (Luo et al., 2023), DeepSeekCoder (Guo et al., 2024). However, they remain unreliable, particularly in scenarios involving complex input-output mappings. Because of the low tolerance of compilers and operating systems for bugs, the instability makes LLMs hard to deploy into real-world applications. Several methods have been proposed to mitigate the phenomenon (Shi et al., 2022; Chen et al., 2022; Zhang et al., 2022; Key et al., 2022; Ni et al., 2023; Dong et al., 2023; Olausson et al., 2023; Chen et al., 2023b; Zhang et al., 2023). The line of work with the most direct relevance to ours is to re-rank generated solutions in a post-hoc manner. For example, Shi et al. (2022) matches the execution results of generated solutions for minimum Bayes risk selection. Zhang et al. (2022) prompts another model as a reviewer to check whether generated programs satisfy the given language instruction by measuring p(instruction|program). CodeT (Chen et al., 2022) additionally generates test cases to verify the generated solutions. Similarly, ALGO (Zhang et al., 2023) additionally generates exhaustive search algorithms as oracle programs to generate high quality test cases for verification.
Ranking on graph. In our framework, the final stage is in fact a ranking problem in graph. There exists some renowned graph ranking algorithms like PageRank (Page et al., 1998) and HITS (Kleinberg, 1999). While our approach is inspired by manifold ranking (Zhou et al., 2003b), which is built upon a regularization framework on discrete spaces (i.e. graphs in this scenario) (Zhou et al., 2003a; Zhou and Schölkopf, 2004, 2005).

heading: 5 Conclusion
text: In this paper, we present a novel code generation method, Multi-Perspective Self-Consistency (MPSC), aimed at enhancing the performance of LLMs in complex code generation tasks where a
single attempt may not suffice to ensure the accuracy of the output. Our proposed MPSC strategy capitalizes on both intra- and inter-consistency across three perspectives, solutions, specifications and test cases, to identify the most reliable answer. We systematically validate the effectiveness of MPSC through comprehensive experiments conducted on four widely used code generation benchmarks. The evaluation results demonstrate that MPSC outperforms existing methods and achieves the state-of-the-art performance on all of them.
Limitations
Evaluation in the wild. Even though MPSC has shown remarkable performance on most widely used code generation benchmarks, its effectiveness in real-world scenarios remains largely unexplored. Existing code generation benchmarks often present simplified code generation tasks compared to the intricacies encountered in actual code developments, where user intents are harder to understand and the desired functionalities are more complex.
Generalization to other tasks. Since our proposed MPSC framework is designed to be modelagnostic and task-agnostic. We only conduct experiments in the code generation task in this paper. Actually, MPSC can be applied to other textual generation tasks like math problem solving and question answering. However, unlike code generation, where code interpreter can measure the agreement between outputs in a deterministic way, assessing the agreement between natural language outputs is non-trivial. A general task-agnostic interconsistency measure is to solely rely on LLMs, whose evaluation ability for arbitrary textual inputs has been demonstrated recently. We leave it for future works to discuss.

heading: Acknowledgments
text: This work was supported by National Key R&D Program of China (2021YFF0901502), Beijing Science and Technology Program (Z231100007423011) and Key Laboratory of Science, Technology and Standard in Press Industry (Key Laboratory of Intelligent Press Media Technology). We sincerely thank Xinyu Hu, Mingqi Gao, Xiao Pu and Xiang Chen for providing insightful advice about this work. We also appreciate the anonymous reviewers for their helpful comments.

heading: A Details of the Iterative Algorithm
text: Description of algorithm. The iterative algorithm is shown in Algorithm 1.
Algorithm 1: Iterative Optimization Input: degree matrix
D = diag(d1, ..., dN ), initialization score vector y, weighted adjacency matrix W , threshold ϵ
Output: optimal confidence score vector f∗ begin f (0) ←− y T ←−D− 12WD− 12 i←− 0 Do
f (i+1) ←− αTf (i) + (1− α)y i←− i+ 1
While ||f (i) − f (i−1)|| ≤ ϵ f∗ ←− f (i) return f∗
Proof of Convergence We first expand the expression of f (n) according to the recursive formula
f (n) = αTf (n−1) + (1− α)y
= (αT )n−1f (0) + (1− α) n−1∑
i=0
(αT )iy
Notice that T is similar to a stochastic matrix WD−1 = D 1 2 (D− 1 2WD− 1 2 )D− 1 2 = D 1 2TD− 1 2 . Therefore the eigenvalues of αT are in [−α, α]. With α ∈ (0, 1), we have
lim n→∞
(αT )n = 0
lim n→∞
n−1∑
i=0
(αT )i = (I − αT )−1
Therefore
f∗ = lim n→∞ f (n) = (1− α)(1− αT )−1y
Proof of Equivalence Denote the optimization function as
F = αfT (I −D− 12WD 12 )f + (1− α) 2 (f − y)2
= αfT (I − T )f + (1− α) 2 (f − y)2
Differentiate F with respect to f , we have
∂F ∂f = α(I − T )f + (1− α)(f − y)
Let the derivatives to 0, the solution f ′ = (1 − α)(I − αT )−1y = f∗. Therefore, the iterative algorithm is actually optimizing the objective function.
Results of closed-form solution. Despite the existence of a closed-form solution for the optimization problem, the required matrix inversion operation is computationally expensive. Conversely, the iterative algorithm exhibits rapid convergence and demonstrates strong empirical performance. Consequently, we employ the iterative algorithm in our experiments. Additionally, we provide the performance of the closed-form solution in Table 7. Our results indicate that the iterative algorithm achieves a performance on par with that of the direct computation of the closed-form solution.
B Implementation of Inter-Consistency
We present the code snippets measuring the interconsistency between each pair of perspectives in Listing 1, 2, 3. After execution with Python interpreter, the final_result is acquired as ω(vi, vj).
1 """ Generated specifications """ 2 # Pre -conditions 3 def preconditions(input): 4 ... 5 6 # Post -conditions 7 def postconditions(input , output): 8 ... 9
10 """ Generated test cases """ 11 test_case = {'input ': ..., 12 'output ': ...} 13 14 """ Check inter -consistency """ 15 def check (): 16 pass_result = None 17 try: 18 preconditions(test_case['input ' ]) 19 postconditions(test_case['input ' ], test_case['output ']) 20 pass_result = True 21 except Exception as e: 22 pass_result = False 23 return pass_result 24 global final_result 25 final_result = check()
Listing 1: Inter-consistency between specifications and test cases.
1 """ Generated solutions """ 2 def entry_point(input): 3 ... 4 5 """ Generated specifications """ 6 # Pre -conditions 7 def preconditions(input): 8 ... 9
10 # Post -conditions 11 def postconditions(input , output): 12 ... 13 14 """ Generated casual inputs """ 15 casual_inputs = [...] 16 17 """ Check inter -consistency """ 18 def check (): 19 pass_result = [] 20 for ci in casual_inputs: 21 try: 22 output = entry_point(ci) 23 postconditions(ci, output) 24 pass_result.append(True) 25 except Exception as e: 26 pass_result.append(False) 27 return sum(pass_result) / len( pass_result) 28 global final_result
29 final_result = check()
Listing 2: Inter-consistency between solutions and specifications.
1 """ Generated solutions """ 2 def entry_point(input): 3 ... 4 5 """ Generated test cases """ 6 test_case = {'input ': ..., 7 'output ': ...} 8
9 """ Check inter -consistency """ 10 def check(): 11 try: 12 output = entry_point(test_case[' input ']) 13 pass_result = (output == test_case['output ']) 14 except Exception as e: 15 pass_result = False 16 return pass_result 17 global final_result 18 final_result = check()
Listing 3: Inter-consistency between solutions and test cases.
C Discussion about Pass@k In this section, we discuss the flaws of Pass@k in Chen et al. (2021) and propose a variant for evaluating methods involved selection and filtering.
Chen et al. (2021) propose an unbiased estimator called Pass@k, which estimates the probability of a model passing unit tests within k attempts. In specific, Chen et al. (2021) first samples a total of n solutions with c of them are correct, randomly samples k solutions for testing, and use the probability of passing tests for estimation,
Pass@k = 1− ( n−c k ) ( n k )
Although their implementation serves as an effective measure of the code generation ability of different foundation models (referred to as the first category of methods in the following), it is not suitable to evaluate methods involving filtering or selection during the inference stage (Li et al., 2022; Chen et al., 2022) (referred to as the second category of methods in the following), as the n generated solutions are identical.
To address the limitation, we implement a variant of Pass@k. We assume each method provides a score function over the n generated solutions, which provides a unified view for the two method categories and hence enables a fair comparison. The first category can be regarded as assigning an identical score to each solution. Similar to the original definition of Pass@k, we evaluate the method by testing the top-k solutions with the highest scores. As the score function may assign the same score to multiple solutions, the test result of the top-k is not deterministic but an expected value.
Mathematically, let’s assume that a method sequentially arranges outputs into an ordered list {s1, ..., sn}, such that ∀i > j, si ⪯ sj according to their scores. We define a set of solutions Sk = {si|sk ⪯ si}, which represents the solution set selected by the method. Suppose the cardinalily of Sk is n̂, the number of correct solutions within Sk is ĉ. Noted that n̂ ≥ k, and thus we uniformly sample k solutions {s′1, ..., s′k} from Sk for estimation,
Pass@k (Ours) = Es′1,...,s′k [1∪ki=1s′i is correct]
= Pr(∪ki=1s′i is correct) = 1− Pr(∩ki=1s′i is incorrect) = 1− ( n̂−ĉ k
) ( n̂ k )
For a the first category of methods, n̂ equals n since it treats each solution equally. As a result, our implementation of Pass@k is identical to the original implementation in Chen et al. (2021).
Pass@k of MPSC when k is large As shown in Table 2, the performance of MPSC-Semantic and MPSC-Lexical remains largely unchanged as k increases. This phenomenon aligns with the nature of MPSC, which assesses solutions based on their consistency, hence assigning similar scores to solutions with similar semantics. Consequently, all solutions will aggregate into many clusters depending on whether the assigned scores are identical, resulting in a lack of diversity within each cluster. Therefore, the pass rate of solutions selected by MPSC will remains constant even in more attempts (i.e. varying k), if the number of attempts is still less than the top-ranked cluster size. A trivial method to address the problem is to increase the diversity by adopting a round-robin selection from all clusters, rather than selecting solutions according to scores from highest to lowest. The corresponding results are presented in Table 8. We can see the performance of Pass@5 is improved compared with that of Table 2.
Method Pass@1 Pass@2 Pass@5
HumanEval
GPT-3.5-Turbo 68.38 76.24 83.15 MPSC-Lexical 82.32 84.76 86.59 MPSC-Semantic 83.35 86.08 89.75
HumanEval+
GPT-3.5-Turbo 58.75 66.58 73.96 MPSC-Lexical 74.39 75.0 77.44 MPSC-Semantic 73.08 77.89 83.20
MBPP
C.1 Analysis of Other Perspectives
MPSC not only selects the optimal output from the target perspective but also chooses outputs from auxiliary perspectives, thereby generating corresponding by-products. In the context of code generation, which is the primary focus of this paper, MPSC can additionally identify more reliable test cases and specifications. We evaluate the quality of these by-products and present the results in Table 11. The experimental results demonstrate that MPSC is proficient in selecting high-quality outputs across all relevant perspectives.

heading: D More Analysis
text: D.1 MPSC with User-provided Test Cases
In practical applications of code generation, users often provide a limited number of test cases to outline the desired functionality, thereby assisting the model in generating code that aligns with the requirements. In Table 2, MPSC-Label has shown remarkable performance. In this study, we investigate the potential performance improvements of the method in such scenarios by incorporating various quantities of golden test cases. These golden test cases are generated and then validated using canonical solutions provided in the benchmarks. We conduct experiments on the HumanEval and MBPP
dataset 8 and present the results in Table 9. The substantial performance enhancements achieved with the inclusion of merely five golden test cases underscore the feasibility of implementing MPSC in user-interactive application scenarios.
D.2 Time Overhead
The additional time overhead imposed by MPSC mainly comes from inter-consistency measurements and the iterative algorithm (Alg. 1). For the former, we need to process a total of (200× 50 + 50×100+100×200) = 35000 edges with a time limit of 0.001 seconds per edge. In contrast, CodeT (Chen et al., 2022), the strongest baseline, requires to process a total of 200 × 500 = 100000 edges. Moreover, this process can be fully parallelized. For the latter, the iterative algorithm converges rapidly, typically within an average of about 50 rounds, requiring less than 0.1 second. Overall, the time overhead of MPSC is acceptable in most code development scenarios, laying the groundwork for real-world deployments.
D.3 Stability of MPSC
We explore the stability of MPSC with respect to the sampling process. We conduct the sampling process of WizardCoder with three random seeds and then assess the performance of MPSC on the generated solutions. The average results are shown
8CodeContests doesn’t provide canonical solutions. Therefore, we cannot conduct evaluation of test cases.
in Table 10. It is evident that the improvement brought by MPSC is very stable.
D.4 MPSC with Limited API Calls
We here discuss another setting, where MPSC utilizes 100 solutions, 50 specifications and 50 test cases, requiring identical API calls to the foundation model baselines. The results shown in Table 13 again prove the supreme performance of MPSC over baselines under fair comparison.
D.5 Comparison on Other Foundation Models
Table 3 demonstrates the generalization of MPSC on other foundation models. We also conduct an experiment to compare MPSC with CodeT, the strongest baseline, under this setting. The results are presented in Table 12.

heading: E Experiment Settings and Baselines
text: We incorporate various baselines in code generation. First of all, we include many strong large language models like ChatGPT (gpt-3.5turbo 0614 version), GPT-4 (gpt4-0614 version), Code Llama-Instruct-34B, WizardCoder-Python34B and DeepSeekCoder-7B-Instruct. The specific hyper-parameters for inference of ChatGPT and GPT4 are shown in Table 14. MPSC requires an additional hyper-parameter α, which controls the balance between inter-consistency and intraconsistency in the algorithm. Given that the quality of inter-consistency significantly depends on the edge density of the graph, we utilize the mean edge weight to determine the value of α. Empirically, we assign a relatively small value of α (0.01) when the edges are sparse on the graph, indicated by the
mean edge weight less than 0.16. Other, we assign a large value of α (0.95) otherwise to better leverage inter-consistency.
We also include several baselines like SelfConsistency MBR-EXEC, CodeT and Selfcollaboration, which enhance the inference capability of LLMs in a post-hoc manner.
• CodeT: This baseline first uses generated test cases to verify each solution by code execution. Then it utilizes RANSAC algorithm to create consensus sets based on execution results. The size of consensus set is then used to rank solutions. We generate 500 test cases for CodeT following the original implementation in Chen et al. (2022).
• Self-Consistency: We implement this baseline following Chen et al. (2022). If two solution pass the same set of generated test cases and specifications, we regard them “consistent”. Then we take a majority voting to rank solutions following Wang et al. (2022).
• MBR-EXEC: This baseline ranks solutions by minimum Bayes risk decoding based on the execution results in the 500 generated test cases.
For a fair comparison between our proposed MPSC and these baselines, we employ the same solutions generated by ChatGPT for them to rerank. In specific, we sample 200 solutions following the conventional setting. Since some methods leverage generated test cases and specifications as well, we use the same set of test cases and specifications generated by ChatGPT for both MPSC and these baselines.
We don’t include ALGO (Zhang et al., 2023) as baseline, because it requires to keep generating
oracle programs until they pass all public example test cases, whose complexity is unlimited.

heading: F Prompt for MPSC
text: Prompt for Generating Specifications
I want you to act as a python programmer. Given a docstring about a python method, you need to write its pre-conditions in one test function “def preconditions(input)" and post-conditions in another test function “def postconditions(input, output):". You should ensure invalid input or output of the method will raise error in the two test functions.
```Python {Demonstration Docstrings 1} pass #Pre-conditions {Demonstration Preconditions 1} #Post-conditions {Demonstration Postconditions 1} ```
```Python {Demonstration Docstrings 2} pass #Pre-conditions {Demonstration Preconditions 2} #Post-conditions {Demonstration Postconditions 2} ```
```Python {Docstrings} pass
Prompt for Generating Solutions
I want you to act like a Python programmer. I will give you the declaration of a function and comments about its property. You need to implement the body of the function in the code block. Do not modify any code I provide. Do not provide any explanations.
Here is the question. ```Python {Docstrings} ```
Prompt for Generating Test Cases
```Python # Given a docstring, continue to write the following code with 10 valid assertion statements to check the correctness of the function. Provide diverse test cases. {Docstrings} pass
# check the correctness of with 10 different valid assertion statements in the form of “assert {entry point}(...) == ..." assert

heading: G Case Study
text: We present some cases from HumanEval, MBPP, CodeContests in the following pages.
1 """ User Intent """ 2 def get_row(lst , x): 3 """ 4 You are given a 2 dimensional data , as a nested lists , 5 which is similar to matrix , however , unlike matrices , 6 each row may contain a different number of columns. 7 Given lst , and integer x, find integers x in the list , 8 and return list of tuples , [(x1, y1), (x2, y2) ...] such that 9 each tuple is a coordinate - (row , columns), starting with 0.
10 Sort coordinates initially by rows in ascending order. 11 Also , sort coordinates of the row by columns in descending order. 12 13 Examples: 14 get_row ([ 15 [1,2,3,4,5,6], 16 [1,2,3,4,1,6], 17 [1,2,3,4,5,1] 18 ], 1) == [(0, 0), (1, 4), (1, 0), (2, 5), (2, 0)] 19 get_row ([], 1) == [] 20 get_row ([[], [1], [1, 2, 3]], 3) == [(2, 2)] 21 """ 22 23 24 """ Solution selected by MPSC """ 25 def get_row(lst , x): 26 result = [] 27 for row_index , row in enumerate(lst): 28 for col_index , val in reversed(list(enumerate(row))): 29 if val == x: 30 result.append ((row_index , col_index)) 31 result.sort(key=lambda x: x[0]) 32 return result 33 34 35 """ Solution selected by CodeT """ 36 def get_row(lst , x): 37 result = [] 38 for i, row in enumerate(lst): 39 if x in row: 40 for j, val in sorted(enumerate(row), key=lambda x: -x[1]): 41 if val == x: 42 result.append ((i, j)) 43 return result
Listing 4: Cases from HumanEval.
1 """ User Intent """ 2 def count_first_elements(test_tup): 3 ''' 4 Write a function to find the number of elements that occurs before the tuple element in the given tuple. 5 ''' 6 7 8 """ Solution selected by MPSC """ 9 def count_first_elements(test_tup):
10 count = 0 11 for i in range(len(test_tup)): 12 if isinstance(test_tup[i], tuple): 13 break 14 count += 1 15 return count 16 17 18 """ Solution selected by CodeT """ 19 def count_first_elements(test_tup): 20 count = 0 21 for i in range(len(test_tup)): 22 if test_tup[i] == test_tup [0]: 23 break 24 count += 1 25 return count
Listing 5: Cases from MBPP.
1 """ User Intent """ 2 def solution(inputs: str) -> str: 3 ''' 4 You are given an array a of length n. 5 Let's define the eversion operation. Let x = a_n. Then array a is partitioned
into two parts: left and right. The left part contains the elements of a that are not greater than x (<= x). The right part contains the elements of a that are strictly greater than x (> x). The order of elements in each part is kept the same as before the operation , i. e. the partition is stable. Then the array is replaced with the concatenation of the left and the right parts.
6 For example , if the array a is [2, 4, 1, 5, 3], the eversion goes like this: [2, 4, 1, 5, 3] -> [2, 1, 3], [4, 5] -> [2, 1, 3, 4, 5]. 7 We start with the array a and perform eversions on this array. We can prove that after several eversions the array a stops changing. Output the minimum number k such that the array stops changing after k eversions. 8
9 Input: 10 Each test contains multiple test cases. The first line contains the number of test cases t (1 <= t <= 100). Description of the test cases follows. 11 The first line contains a single integer n (1 <= n <= 2 * 10^5). 12 The second line contains n integers a_1 , a_2 , ..., a_n (1 <= a_i <= 10^9). 13 It is guaranteed that the sum of n over all test cases does not exceed 2 * 10^5. 14 15 Output: 16 For each test case print a single integer k - the number of eversions after which the array stops changing. 17 Example 18 assert solution ('3\n5\n2 4 1 5 3\n5\n5 3 2 4 1\n4\n1 1 1 1') == '1\n2\n0' 19 ''' 20 """ Solution selected by MPSC """ 21 def solution(inputs: str) -> str: 22 def eversion(a): 23 n = len(a) 24 l, r = [], [] 25 x = a[-1] 26 for i in range(n - 1): 27 if a[i] <= x: 28 l.append(a[i]) 29 else: 30 r.append(a[i]) 31 return l + [x] + r 32 33 def is_same(a, b): 34 return len(a) == len(b) and all(ai == bi for ai, bi in zip(a, b)) 35 36 res = [] 37 inputs = inputs.strip().split('\n') 38 t = int(inputs [0]) 39 idx = 1 40 for _ in range(t): 41 n = int(inputs[idx]) 42 a = list(map(int , inputs[idx +1]. split())) 43 idx += 2 44 cnt = 0 45 while True: 46 new_a = eversion(a) 47 cnt += 1 48 if is_same(a, new_a): 49 break 50 a = new_a 51 res.append(str(cnt - 1)) 52 return '\n'.join(res)
53 """ Solution selected by CodeT """ 54 def solution(inputs: str) -> str: 55 # split the input into list of test cases 56 test_cases = inputs.strip().split('\n')[1:] 57 result = [] 58 59 # loop through each test case 60 for i in range(0, len(test_cases), 2): 61 n = int(test_cases[i]) 62 a = list(map(int , test_cases[i+1]. split())) 63 k = 0 64 65 # perform eversion operation until array stops changing 66 while True: 67 x = a[-1] 68 left , right = [], [] 69 for j in range(n-1): 70 if a[j] <= x: 71 left.append(a[j]) 72 else: 73 right.append(a[j]) 74 new_a = left + [x] + right 75 if new_a == a: 76 break 77 a = new_a 78 k += 1 79 80 result.append(str(k)) 81 82 # join the results and return 83 return '\n'.join(result)
Listing 6: Cases from CodeContests.

heading: Future Work
text: We leave it for future works to discuss