Functional Overlap Reranking for Neural Code Generation (2024)

Abstract: Code Large Language Models (CodeLLMs) have ushered in a new era in code generation advancements. However, selecting the best code solutions from all possible CodeLLM outputs remains a challenge. Previous methods often overlooked the intricate functional similarities and interactions between solution clusters. We introduce SRank, a novel reranking strategy for selecting the best solutions from code generation, focusing on modeling the relationships between clusters of solutions. By quantifying the functional overlap between solution clusters, our approach provides a better ranking strategy for code solutions. Empirical results show that our method achieves remarkable results on the pass@1 score. For instance, on the Human-Eval benchmark, we achieve 69.66% in pass@1 with Codex002, 75.31% with WizardCoder, 53.99% with StarCoder, and 60.55% with CodeGen, surpassing state-of-the-art code generation reranking methods such as CodeT and Coder-Reviewer on the same CodeLLM by a significant margin (≈ 6.1% improvement on average). Even in scenarios with a limited number of sampled solutions and test cases, our approach demonstrates robustness and superiority, marking a new benchmark in code generation reranking. Our implementation can be found at https://github.com/ FSoft-AI4Code/SRank-CodeRanker.

heading: 1 Introduction
text: Recent advancements in language models tailored for code, known as Code Large Language Models (CodeLLMs) (Luo et al., 2023; Wang et al., 2023; Nijkamp et al., 2023b; Rozière et al., 2023; Wei et al., 2023; Lozhkov et al., 2024; Pinnaparaju et al., 2024; Guo et al., 2024; Bui et al., 2023), have garnered significant interest, particularly due to the expansion of large-scale language models and the volume of pre-training data (Kaplan et al., 2020; Zhao et al., 2023). A primary utility of CodeLLMs is their capacity to generate code given natural lan-
guage descriptions written by humans (Chen et al., 2021; Fried et al., 2023; Chowdhery et al., 2022; Nijkamp et al., 2023b). However, prior studies (Holtzman et al., 2020; Austin et al., 2021) have highlighted that the sequences generated by these models can be prone to errors, especially when likelihood-based decoding techniques like greedy search and beam search are employed. Alternatively, sampling-based decoding techniques (Fan et al., 2018; Holtzman et al., 2020) extract multiple solutions from the model’s multinomial distribution. This method generates a wide range of code solutions, many of which are correct (Austin et al., 2021; Ni et al., 2023). As a result, there is a growing interest in developing reranking strategies for code generation (Li et al., 2022; Inala et al., 2022; Chen et al., 2023; Zhang et al., 2023; Ni et al., 2023), with the goal of sorting through an abundance of sampled solutions to identify high-quality and accurate ones.
The goal of reranking is to organize the set of candidate programs so that accurate programs are prioritized. Li et al. (2022), Chen et al. (2023), and Ni et al. (2023) have clustered code solutions based on their functionality, then used cluster-specific data to determine ranking scores. Given that language models frequently produce code solutions that differ syntactically but are semantically analogous, functional clustering narrows the candidate pool. The emphasis then shifts from ranking individual solutions to ranking clusters themselves. Previous ranking strategies, such as AlphaCode (Li et al., 2022) and CodeT (Chen et al., 2023), provide approaches to clustering and reranking code solutions. While AlphaCode (Li et al., 2022) focuses on identical outputs from model-generated test inputs, CodeT (Chen et al., 2023) focuses on solutions that pass model-generated test cases. The CoderReviewer approach (Zhang et al., 2023), inspired by collaborative software development, uses a dualmodel system to cross-check generated programs
3686
against language instructions. However, by treating clusters in isolation, they fail to model potentially informative functional similarities and interactions across clusters.
To address this limitation, we propose SRank, a novel reranking approach emphasizing modeling inter-cluster relationships. Specifically, we introduce a new metric called functional overlap to quantify the similarity between clusters based on their execution outputs. This allows for identifying the most representative cluster that exhibits maximal overlap with all other clusters. As inconsistencies often indicate incorrect functionality, the cluster interacting most comprehensively likely represents the optimal solution. By incorporating these inter-cluster relationships into the ranking pipeline, we can better identify the most promising solutions. Through extensive evaluation, we demonstrate that modeling inter-cluster relationships and functional overlap provides significant and consistent improvements over prior state-of-the-art solution ranking methods (Li et al., 2022; Chen et al., 2023; Zhang et al., 2023) on a wide range of state-of-the-art CodeLLMs, including Codex, WizardCoder, StarCoder, and CodeGen. For instance, on the HumanEval benchmark, our method achieved a pass@1 score of 75.31% with WizardCoder34B, outperforming the Coder-Reviewer’s score of 66.9%. Similarly, on the MBPP-S benchmark, our method improved the pass@1 score for WizardCoder from 50.3% with Coder-Reviewer to 51.03% with our approach. Similar improvements are applied for other CodeLLMs, including StarCoder, CodeGen, and Codex002. If we compare SRankwith a simple random sampling method to get code solutions, we observe massive improvements across the models, with average improvements of 23.07% and 17.64% for HumanEval and MBPP-S, respectively. Our evaluation is more comprehensive because we include many SOTA CodeLLMs of varying sizes, whereas CodeT and Coder-Reviewer did not. This provides compelling evidence of our approach’s robustness across a wide range of models.
We also conducted an extensive analysis to demonstrate some of our advantages, such as our approach’s remarkable robustness even with limited solutions and test cases. In summary, by moving from isolated clusters to interacting clusters with quantified functional overlap, our novel reranking strategy aims to address the limitations
of prior ranking techniques for code generation. To summarize our contributions, they are as follows:
• We introduce a novel reranking strategy for CodeLLMs that emphasizes the inter-cluster relationships and leverages the functional overlap between them, providing a more robust and accurate approach to pick the best solutions.
• Through extensive and comprehensive evaluations, we demonstrate that our approach consistently outperforms existing state-of-the-art methods in code generation. For instance, our method achieved superior results on both the HumanEval and MBPP-S benchmarks across various CodeLLMs.
• We perform extensive analysis to evaluate the robustness of our method, highlighting its effectiveness even with a limited number of sampled solutions and test cases, and its ability to capture intricate interactions between clusters, setting it apart from previous ranking techniques.

heading: 2 Background & Motivation

heading: 2.1 Code Generation
text: Code generation involves generating code solutions to programming problems based on a given context c. The context includes a natural language description and a code snippet containing statements such as imports and a function signature. Additionally, a predefined set of test cases, denoted as T , is provided to evaluate the correctness of the generated code solutions. Using c as the input on CodeLLM, we obtain a set of solutions S = {s1, s2, ..., sN}, where N is a hyperparameter defining the number of return sequences from the CodeLLM execution. A solution s is considered valid if it successfully passes the predefined set of test cases T.

heading: 2.2 Solution Clustering and Reranking
text: The reranking task aims to prioritize correct programs in the candidate list S. Previously, solutions were clustered by functionality, simplifying the task due to the language models’ tendency to produce syntactically varied but semantically similar solutions. Thus, the focus shifts from ranking individual solutions to ranking these functional clusters.
For instance, AlphaCode (Li et al., 2022) uses a distinct model to create test inputs. Solutions are then executed against these inputs, and those with
matching outputs are clustered. The reranking strategy is based on the understanding that while there can be numerous incorrect program variations, correct ones often show similar patterns, leading to clusters being ranked by their solution count.
Conversely, CodeT (Chen et al., 2023) clusters solutions that pass the same model-generated test cases. However, this can group functionally diverse solutions. If a cluster’s solutions only pass some test cases, it’s uncertain if the outputs for the failed cases are consistent across solutions, potentially compromising cluster functionality and the confidence in selecting from these ranked clusters. We show a concrete example to analyze this problem in Appendix G.

heading: 2.3 Modeling Inter-Cluster Relationships
text: Existing clustering and reranking methods analyze clusters independently without considering intercluster relationships (Li et al., 2022; Chen et al., 2023). However, modeling these interactions can better indicate cluster correctness. As such, we propose a new metric called “functional overlap” to quantify cluster similarity based on execution outputs, as shown in Figure 1. We can execute code solutions from each cluster on the same test inputs and compare their outputs. The level of output match indicates the extent of functional overlap between two clusters.
The intuition is that clusters with high overlap
exhibit greater shared functionality. By modeling the extent to which a cluster overlaps with others, functional overlap identifies the most “representative” cluster. A cluster with maximal cumulative overlap has outputs most consistent with all other clusters. As inconsistencies often indicate incorrect functionality, the cluster interacting most comprehensively is likely the optimal solution. This is similar to the assumptions of Fischler and Bolles (1981), where incorrect solutions are diverse and there is a low probability of having a functional agreement among incorrect solutions.

heading: 3 Approach Details

heading: 3.1 Overview
text: Figure 2 provides an overview of our end-to-end approach. First, given a well-trained CodeLLM, e.g., Codex, and three inputs: (1) Task description, (2) Code generation prompt, (3) Test case generation prompt, we instruct the CodeLLM to generate a set of code solutions as well as test cases. Specifically, we prompt the CodeLLM to produce a collection of code solutions S = {s1, s2, ..., sN} and a set of test cases T = {t1, t2, ..., tM}, where N and M are hyperparameters defining the number of solutions and test cases.
Each test case ti consists of two components: the test input zi and the expected output ôi based on the context (e.g., assert add(1,2) == 3, where (1,2) is the input and 3 is the output). We can then execute the test inputs Z = {z1, z2, ..., zM} on the set of solutions S to generate the execution outputs O = {o11, o12, ..., oNM}. Next, we cluster the solutions S into groups C = {C1, C2, ..., CK} based on their execution outputs, where K is the number of unique clusters. We then compute an interaction matrix I to quantify the functional overlap between clusters. Finally, we multiply the interaction matrix I by a validation score vector V to obtain final ranking scores R for selecting the optimal solutions. The validation scores in V represent features of each cluster, such as the number of solutions.
In the following sections, we elaborate on the key steps of our algorithm.

heading: 3.2 Clustering Solutions by Execution Outputs
text: We first execute each solution si ∈ S on the test inputs Z to produce execution outputs O. Solutions that exhibit identical execution outputs are grouped
into the same cluster:
F (si) = F (sj) ⇐⇒ Osi = Osj .
Here, F represents the clustering function that maps a solution s to a cluster identifier Ck. The above equation indicates that two solutions si and sj are assigned to the same cluster if and only if their output sets Osi and Osj are exactly equal.

heading: 3.3 Computing Interaction Matrix
text: After obtaining execution outputs oij for each cluster Ci on test input zj , we define an interaction matrix I ∈ RK×K to quantify functional overlap:
Iij = 1
M
M∑
k=1
δ(oik = ojk), (1)
where oik and ojk refer directly to the execution outputs of clusters Ci and Cj , respectively, on the kth test input. δ is the indicator function that returns 1 if the condition inside is true and 0 otherwise.

heading: 3.4 Computing Final Ranking Scores
text: In addition to modeling inter-cluster interactions via I, we also consider an extra validation dimension V ∈ RK×1 containing cluster features. For instance, Vi could represent the number of solutions in cluster Ci (abbreviated as cluster sizes) or the number of test cases that the solutions in cluster Ci passed (abbreviated as pass rates), providing a notion of cluster confidence. The final ranking vector R ∈ RK×1 can then be computed as R = I ·V. Here, Ri aggregates information about both the inter-cluster interactions of Ci (via I) and its cluster features (via V). Clusters with higher ranking scores in R are those with significant functional overlap to other clusters and high validity according to V. By considering inter-cluster relationships and functional similarity in a principled manner, we believe our ranking approach can effectively identify the most promising solutions. We validate
our method through extensive experiments in the following sections.

heading: 4 Experimental Setup
text: Models We evaluate our method on several stateof-the-art CodeLLMs, including Codex, WizardCoder, StarCoder, and CodeGen. Each model family has different model sizes (e.g., WizardCoder 15B and 34B) and different training methods (e.g., base model and instruction fine-tuned model). As a result, we chose a diverse set of models, ranging from small to large scale, and from base models to instruction fine-tuned models, to demonstrate the efficacy of our ranking method. In total, we demonstrate our approach on 6 models ranging from 6B to 34B parameters.
Metrics We use pass@k (Chen et al., 2021), which is often employed to evaluate the functional correctness of code solutions based on code execution instead of similarity-based metrics.
Baselines We compare SRank with recent methods for solution reranking, including CoderReviewer (Zhang et al., 2023) and CodeT (Chen et al., 2023). Coder-Reviewer (Zhang et al., 2023) is the state-of-the-art method. On the other hand, CodeT (Chen et al., 2023) shares a similar clustering-reranking approach to our method.
Benchmarks We use two popular benchmarks in code generation: HumanEval (Chen et al., 2021) and MBPP-S (sanitized version) (Austin et al., 2021). For a more challenging assessment, we evaluate SRank on APPS (Hendrycks et al., 2021). To avoid exposing real test cases to the language model, we follow the prompt design in (Chen et al., 2021) by removing all example input-output cases from context before generating code solutions and test cases.
Implementation Details For Codex002 and CodeGen16B, we refer to the artifacts, including
both solutions and test cases, provided by Chen et al. (2023). Regarding the remaining models, we use the HuggingFace library (Wolf et al., 2019) and load models in half-precision. We set the temperature to 0.8, the top p to 0.95, the maximum new tokens to 2048, and the timeout for executing solutions to 5 seconds. For each problem, we sample 100 code solutions and 100 sequences of test cases, each sequence containing multiple test cases. The prompts we used for sampling code solutions and test cases from each model can be found in Appendix B. For post-processing code solutions and test cases, we follow Chen et al. (2023) to truncate the generated sequences by the five-stop words: "\nclass", "\ndef", "\n#", "\nif", and "\nprint".

heading: 5 Experimental Results
text: Table 1 presents the pass@1 results on the HumanEval and MBPP-S benchmarks for various CodeLLMs. Our method, SRank, consistently outperforms other techniques across most models and benchmarks. For instance, on the HumanEval benchmark, SRank achieves average improvements over CodeT and Coder-Reviewer of about 3.63% and 8.81% in pass@1, respectively.
Additionally, when comparing Coder-Reviewer with the random sampling method, it is unstable across the models. Specifically, using WizardCoder15B and StarCoder as examples, CoderReviewer brings modest increases of 4.17% and 6.16%, compared to our improvements of 14.79% and 21.44%. On the MBPP-S benchmark, SRank still achieves outstanding performance, although the magnitude of improvements is slightly less than that of HumanEval. Our comprehensive experiments demonstrate the effectiveness of SRank over CodeT and Coder-Reviewer.
To assess the effectiveness of our proposed method on a wide range of coding problem difficulties, we evaluate SRank on APPS using the Codex002. Results are shown in Table 3. We observe that SRank consistently outperforms all other baselines by a significant margin. Compared with CodeT, adding inter-cluster modeling significantly improves reranking results. On the other hand, due to the difficulty of the tasks, the improvement is less significant when the level of difficulty increases. The results show that SRank is robust and scales well with different difficulty levels.
In addition, to validate our method on closed source and high capability models, we select An-
thropic Claude 3 Opus (Anthropic, 2024) as a representative LLM in our experiment. Please refer to Appendix F for results.

heading: 6 Analysis
text: Assumption Validation We provide a comprehensive analysis to validate our assumption that incorrect solutions are diverse and there is a low probability of functional agreement among incorrect solutions. Formally, let S denote the set of solutions sampled from a certain CodeLLM, si be the i-th solution in S, Ck be the k-th cluster by our clustering algorithm, Ci be the cluster including the solution si, and |Ck| and |Ck|∗ be the number of solutions and quantity of incorrect solutions in the k-th cluster, respectively. The function f(si, sj) is defined as the computation of the functional overlap between si and sj , similar to Eq. 1. We then compute the probability of incorrect solutions with varying levels of functional overlap.
p(l ≤ f(si, sj) < h, si and sj are incorrect)
= ∑ (i,j)∈M |Ci|∗|Cj |∗(|S|
2
) (2)
Here, M is comprised of pairs of (si, sj) where l ≤ f(si, sj) < h, and l and h are the two hyperparameters. We consider two range values, (l1, h1) and (l2, h2) with the same length, where l1 < l2 and h1 < h2. According to our assumption, we anticipate that the following inequality holds:
p(l1 ≤ f(si, sj) < h1, si and sj are incorrect) > p(l2 ≤ f(si, sj) < h2, si and sj are incorrect)
The left term denotes the probability of lower functional agreement among incorrect solutions, while the right term signifies the corresponding probability of higher functional agreement among incorrect solutions. The observed relationship indicates that the probability of lower functional agreement surpasses that of higher functional agreement, substantiating our assumption. Results in Figure 3 show a general decline in probability with increasing values of l and h. Particularly for the range (l, h) = (0, 0.1), the probability is significantly higher compared to those of other ranges, and at the next range, (l, h) = (0.1, 0.2), the probability experiences a notable decrease. Moreover, when incorrect solutions exhibit a high functional overlap, exceeding 0.7, the probability is low at around 3%. These findings are consistent with our assumption. More results can be found in Appendix A.
HumanEval
Impact of Cluster Features We aim to evaluate reranking performance solely with cluster features, i.e., only cluster sizes, pass rates, or a combination of both. Table 2 shows SRank’s performance when these features are added or removed from the ranking pipeline. When ranking code solutions using only one of the mentioned features, we can see that cluster sizes are more important than pass rates.
However, this does not mean that pass rates are unimportant. When both features are combined, the results can be improved even further. Finally, we achieve the best SRank performance by combining both of these features or one of them with the Interaction matrix.
It’s worth noting that the cluster features possess a general and adaptable nature, allowing for potential extensions to incorporate additional information such as likelihood. We also report results of the combination of clusters’ likelihood and CoderReviewer ranking criteria as cluster features with functional overlap in Appendix D.
Impact of Interaction Matrix We conduct additional experiments to demonstrate the effectiveness of the interaction matrix I. From the results in Table 2, it is obvious that the interaction matrix
I helps to boost performance for any cluster features. Importantly, integrating cluster sizes with I achieves results on par with CodeT, highlighting the significance of interactions among clusters.
Scaling Number of Generated Test Cases We conducted an ablation study to assess how the number of generated test cases influences code reranking performance. Figure 4 shows the pass@1 performance as a function of the number of test cases generated, ranging from 2 to 50, on the HumanEval benchmark. Please refer to Appendix C for the results of MBPP-S.
Comparing each pair of solid line (with interaction matrix) and dashed line (w/o interaction matrix), cluster interaction consistently enhances performance over solely ranking by cluster features. The performance gap of reranking with and without interaction increases as the number of generated test cases increases, showcasing our method’s scalability. However, with limited test cases, SRanksometimes underperforms due to the potentially negative impact of the cluster features when integrated with cluster interaction. For an optimal balance between effectiveness and efficiency, we suggest generating at least 30 test cases to fully benefit from cluster interaction, feasible within 1 to 2 sampling rounds given many test cases are generated within a single sampled sequence.
Additionally, comparing the performance of our method and CodeT reveals disparities in some CodeLLMs. This discrepancy arises from the clustering quality of CodeT, such that limited, lowquality test cases can lead to semantically inconsistent clusters, making ranking them less meaningful. In contrast, our method clusters solutions by matching execution outputs, which ensures functional
consistency, enhancing ranking reliability.
Scaling Number of Sampled Solutions In a realworld setting, it is practical to have a limit on the number of both solutions and test cases generated, since sampling multiple sequences from LLM is costly and time inefficient for low resource computing. In this section, we examine the efficiency of our SRankby scaling down the number of sampled solutions to be less than or equal to 50 samples.
For practical reasons, we only run each experiment with 50 test cases generated by each model, rather than all test cases extracted from 100 sequences of test cases. We prompt WizardCoder34B, WizardCoder15B, and CodeGen2.5Instruct to generate 50 different test cases in a single sequence. This prompting technique improves the overall pipeline efficiency by significantly lowering the computational overhead of sampling from CodeLLM. Figure 5 shows the pass@1 performance with the number of sampled solutions ranging from 2 to 50. Results of the MBPP-S benchmark can be found in Appendix C. The figure shows a similar trend to when we scale the number of generated test cases. Indeed, adding cluster interaction along with cluster features brings a performance boost, even when the number of sampled code solutions is small.
Compared to CodeT, as explained earlier, due to the semantically consistent clustering process in our method, both reranking with or without interaction outperform CodeT.
Moreover, with a limited number of sampled solutions and test cases, the results bypass greedy search (represented by the black lines). This proves both the effectiveness and the efficiency of our approach.

heading: 7 Related Work
text: Code Large Language Models The emergence of large language models (LLMs) has transformed code understanding and generation tasks (Rozière et al., 2023; Li et al., 2023a; Nijkamp et al., 2023a; Wang et al., 2023; Nijkamp et al., 2023b; Fried et al., 2023; Li et al., 2023b). Recent research has employed natural language processing models for code-related tasks, using pretraining strategies akin to those for natural languages (Feng et al., 2020; Wang et al., 2021; Guo et al., 2020; Ahmad et al., 2021; Elnaggar et al., 2021; Peng et al., 2021; Kanade et al., 2020; Chakraborty et al., 2022; Ahmed and Devanbu, 2022; Niu et al., 2022). Among various CodeLLMs, those with larger capacities tend to perform better in code generation tasks. For instance, StarCoder (Li et al., 2023a) and CodeLlama (Rozière et al., 2023) handle contexts with up to 8,000 and 100,000 tokens, respectively, while WizardCoder (Luo et al., 2023) excels in evolving instructions. Additionally, models like CodeGen2.5-Instruct (Nijkamp et al., 2023a), CodeT5+Instruct (Wang et al., 2023), Codex002 (Chen et al., 2021), CodeGen-Mono 16B (Nijkamp et al., 2023b), and InCoder 6B (Fried et al., 2023) have also shown promise in this domain.
Reranking Methods for Code Generation Several studies have explored reranking code generated by language models (Chen et al., 2021; Zhang et al., 2023; Ni et al., 2023; Chen et al., 2023; Inala et al., 2022; To et al.), prioritizing solutions
from these models. Chen et al. (2021) showed empirically that selecting solutions based on the mean log probability of tokens improves performance. Coder-Reviewer (Zhang et al., 2023) proposed a mutual information-based ranking method for natural language instructions and generated solutions. Reranking has also been approached using execution-based metrics. MBR-exec (Shi et al., 2022) minimizes a loss function across all solutions, while AlphaCode (Li et al., 2022) clusters solutions based on execution outputs. LEVER (Ni et al., 2023) uses a verifier to assess program correctness, and CodeT (Chen et al., 2023) generates high-quality test cases. Our approach stands out as it does not require model training or fine-tuning and can complement methods like LEVER (Ni et al., 2023).

heading: 8 Conclusion
text: We propose SRank, a novel reranking strategy designed to extract optimal code generation solutions from CodeLLMs. SRankfocuses on modeling intercluster relationships to identify clusters with the highest functional overlap with others. By prioritizing the cluster with the most comprehensive interaction, often indicating correct functionality, we can pinpoint the most promising solution. Incorporating these inter-cluster relationships into the ranking pipeline enhances solution identification.
We showcase the state-of-the-art performance of SRank on pass@1 across various well-known CodeLLMs, surpassing other ranking methods like
CodeT and Coder-Reviewer in extensive evaluations. Our thorough analysis further highlights the method’s effectiveness in realistic scenarios with a limited number of solutions and test cases. These findings are crucial as they address the challenges of code generation in real-world applications, illuminating strategies for selecting superior solutions within constrained coding environments.
Limitations
Our approach presents several opportunities for further development and refinement. While our method demonstrates superiority over others in our empirical evaluation, which focuses on Python, extending SRank to a multilingual benchmark would provide a more comprehensive assessment of its effectiveness across different programming languages.
SRank has shown promising results in its current scope, which primarily focuses on functions with clear output metrics, such as arithmetic or algorithmic problems. To extend SRank to more complex coding tasks, such as system architecture, memory management, and process management, we would need to validate that our underlying assumption of diverse incorrect solutions holds true in these domains and adapt the function δ to effectively capture the intricacies of these tasks. This presents an exciting opportunity to expand the applicability of our approach.
While implementing SRank could introduce computational overhead due to the need to gen-
erate a large number of candidate solutions and test cases, our ablation study demonstrates that SRank can achieve superior performance with a limited number of sampled solutions and test cases. To further optimize the efficiency of our method, particularly in resource-constrained environments or when rapid solution generation is crucial, we plan to conduct additional analyses and explore potential optimizations. Currently, SRank focuses on assessing functional correctness by modeling the functional overlap among solutions. While functional correctness is a critical aspect of code quality, we acknowledge the importance of other factors such as efficiency and robustness. Integrating these factors into our framework presents an exciting direction for future work, as it would enhance the comprehensiveness and robustness of SRank in meeting the diverse requirements of real-world coding tasks. By addressing these aspects, we aim to make SRank an even more valuable tool for the coding community.

heading: B Generation Prompts
text: In this section, we list all the prompts feeding to CodeLLMs for generation process of code solutions and test cases. For concrete demonstration, we use the same problem ‘HumanEval/8‘ throughout this section.
B.1 Code generation Demonstrated prompts for code generation includes Figure 7, Figure 8, Figure 9, and Figure 10.
B.2 Test Generation We give example test case generation prompts in Figure 11, Figure 12, Figure 13, and Figure 14.
B.3 CoderReviewer Prompts In CoderReviewer, Coder generates code solution given the task description and Reviewer checks the generated solution by measuring the sequence probability of the description given the solution.
For generating code solutions, we use the same prompt as in the Code Generation section. For Reviewer, to adapt to instruction-tuned models, here we list the prompt we manually design to calculate the probability of task description given the code implementation with respect to CodeLLMs.
Below is an instruction that describes a task. Write a response that
approriately completes the requests.
### Instruction: Create a Python script for this problem: from typing import List, Tuple
def sum_product(numbers: List[int]) -> Tuple[int, int]: """ For a given list of integers,
return a tuple consisting of a sum and a product of all the integers in a list. Empty sum should be equal to 0 and empty product should be equal to 1."""
### Response:
Figure 7: Code generation prompt used with WizardCoder34B, WizardCoder15B, and CodeGen2.5-Instruct
The prompt templates we used including Figure 15, Figure 16, and Figure 17.

heading: C Scaling Number of Generated Test Cases and Sampled Solutions on MBPP-S
text: This section complements the results on MBPP-S dataset from Section Scaling Number of Generated Test Cases and Section Scaling Number of Sampled Solutions. The setting for MBPP-S is the same as HumanEval mentioned in each above section. Figure 18 and Figure 19 shows pass@1 as function of the number of generated test cases and sampled solutions respectively on MBPP-S benchmark.

heading: D CoderReviewer Ranking Criteria as Cluster Features
text: We investigate adding cluster interaction with CoderReviewer ranking criteria as cluster features, where Coder represents the likelihood PCodeLLM (x|y) and Reviewer represents PCodeLLM (y|x). CoderReviewer is the multiplication PCodeLLM (x|y)PCodeLLM (x|y), with x as the task description and y as generated code solution. The prefixed N. before each term indicates that the likelihood is normalized by the number of sequence tokens, which was shown
to improve performance in previous work. The cluster feature in these cases is the criteria score of the solution with maximum score within that cluster.
Experimental results in Table 5 show consistent improvement when adding cluster interaction over CoderReviewer. This demonstrates that when the interaction matrix is combined with CoderReviewer, the interaction matrix can improve CoderReviewer’s performance even further, indicating that our method is adaptable to different reranking methods.

heading: E Reranking with Interaction Only
text: To further validate that modeling inter-cluster interactions aids in ranking clusters, we present the results of reranking solely by matrix I without any cluster features V in Table 6. In this scenario, V is a column vector with 1 at every entry. Consequently, the ith entry in R is the sum of functional overlap between the ith cluster and all other clusters.
The results demonstrate that incorporating interaction modeling consistently elevated performance beyond random. Furthermore, reranking with interaction alone significantly outperforms the performance of greedy decoding in some cases, such as 51.13 vs 28.49 for CodeGen2.5, 50.01 vs 39.63 for StarCoder, 62.75 vs 47.00 for CodeX002 in HumanEval. This result futher confirms the effectiveness of our method SRank.

heading: F Additional Results with Closed-Source LLM
text: In the previous section, we presented code generation ranking results with both open-source and closed-source models (Codex002), which is not the strongest and up-to-date model for code. To examine our method on contemporary and highcapability LLMs, we selected Anthropic Claude 3
HumanEval
Opus (Anthropic, 2024) as a representative model for our experiments. Table 7 shows the results of code generation on HumanEval using different decoding strategies, such as greedy search and random sampling, as well as when reranking methods like SRankand CodeT are employed. Even with such a large and powerful LLM, SRankstill achieves impressive performance compared to other baselines, while CodeT degrades performance of baselines.
In this experiment, we followed similar hyperparameter settings as with other models, setting the temperature to 0.8 and top-p to 0.95. The results indicate that random sampling and greedy search yield roughly the same performance, suggesting that the model is very confident in its responses and the randomness in output sequences is minimal. This behavior is further confirmed by observing the generated code solutions and test cases, where there is little variance between the samples and a higher rate of duplication compared to other models we experimented with. For such large and high-capability models, the benefits from reranking strategies like SRankare most fully realized when sufficient randomness is introduced in the sampling process by adjusting hyperparameters, such as temperature, top-k, and top-p.
Notably, the highest reported score for Claude 3 Opus from Anthropic on HumanEval is 84.9, which is higher than the numbers presented in this work. The explanation for this discrepancy is that the test cases included in the HumanEval prompt were excluded in our settings, following previous work in code generation reranking.

heading: G Case Study
text: In this section, we aim to compare the quality of cluster samples from CodeT and our method. The top 1 clusters ranked by CodeT and SRank are shown in Figure 20. It is clear that functional inconsistency among solutions plagues CodeT’s clusters. For example, in the problem HumanEval 45, solution #2 is not semantically equivalent to solutions #1 and #3, and in the problem HumanEval 106, solution #1 differs from solution #2. This phenomenon is explained by the fact that CodeT groups solutions that pass the same set of test cases into clusters. As a result, when computing the pass@k, inconsistency in functionalities among top cluster solutions can degrade performance. SRank, on the other hand, considers execution outputs for the clustering process, ensuring functional consistency among solutions within a cluster.

heading: Future Work
text: Integrating these factors into our framework presents an exciting direction for future work, as it would enhance the comprehensiveness and robustness of SRank in meeting the diverse requirements of real-world coding tasks. By addressing these aspects, we aim to make SRank an even more valuable tool for the coding community