Iterative Refinement of Project-Level Code Context for Precise Code Generation with Compiler Feedback (2024)

Abstract: Large Language Models (LLMs) have shown remarkable progress in automated code generation. Yet, LLM-generated code may contain errors in API usage, class, data structure, or missing project-specific information. As much of this project-specific context cannot fit into the prompts of LLMs, we must find ways to allow the model to explore the project-level code context. We present COCOGEN, a new code generation approach that uses compiler feedback to improve the LLM-generated code. COCOGEN first leverages static analysis to identify mismatches between the generated code and the project’s context. It then iteratively aligns and fixes the identified errors using information extracted from the code repository. We integrate COCOGEN with two representative LLMs, i.e., GPT-3.5-Turbo and Code Llama (13B), and apply it to Python code generation. Experimental results show that COCOGEN significantly improves the vanilla LLMs by over 80% in generating code dependent on the project context and consistently outperforms the existing retrieval-based code generation baselines.

heading: 1 Introduction
text: Large Language Models (LLMs), especially those pre-trained on code, as demonstrated by tools such as GitHub Copilot (Microsoft, 2024), Amazon’s CodeWhisperer (Amazon, 2023), and ChatGPT (OpenAI, 2023), are revolutionizing how developers approach programming by automatically generating code for given contexts (e.g., naturallanguage descriptions or surrounding incomplete code). While existing LLM-based code generation tools excel in code generation within a small-scale and isolated context (i.e., a single file), integrating
*Also with National Engineering Research Center for Big Data Technology and System, Services Computing Technology and System Lab, Cluster and Grid Computing Lab, School of Computer Science and Technology, Huazhong University of Science and Technology, Wuhan, 430074, China.
†Yao Wan is the corresponding author.
LLM-based code generation into real-world software projects remains challenging (Li et al., 2024).
Practical code generation for software repositories is often associated with broader project-level contexts declared in other repository files due to the demands of modularity, structure, and comprehension in software management (Kemerer, 1995). Solely providing the task requirements and surrounding incomplete code can lead LLMs to overlook the complex hierarchies of APIs, classes, data structures, or type constraints specific to a software project’s repository, risking omitting essential logic during code generation (Gifany et al., 2013).
As a motivating example, consider the case shown in Figure 1. In this case, we use the OpenAI GPT-3.5-Turbo API to generate an imageenhancing function by first reducing the notice and then adjusting brightness and contrast. Given the prompt in Figure 1a, the LLM will produce a code snippet of calling a reduce_noise method as depicted in Figure 1b. Although the generated code follows a standard workflow specified in the task
2336
prompt, it leads to a compilation error in our application context because reduce_noise is not implemented. This is unsurprising as the prompt does not provide enough project-level context for the LLM, and can be fixed by providing the projectspecific function denoising.denoise as context, as illustrated in Figure 1c. While a carefully engineered prompt may resolve this issue, it is not always possible for the user to generate such prompts, and the project context may be too large to fit into the prompt. As we will empirically show later in (Section 2.2), such errors are commonly found when applying LLMs to repository-level code generation (Li et al., 2024).
This paper investigates a way to effectively integrate LLM-based code generation with existing code implementations within a software project. Our solution is to leverage project-level contextual information, such as project-specific implementation of classes, methods, and data structures, to reduce compilation errors and improve code quality. Directly incorporating the entire project code into a language model is infeasible due to model input sequence length limitations. Instead, we use compiler-based analysis to post-process the modelgenerated code by first detecting discrepancies between the generated code and the project’s context. We then utilize information extracted from the project code base to rectify the mismatches in using modules, APIs, and classes. Our approach combines well-established compiler techniques with emerging generative methods, allowing software developers to leverage the power of LLMs without being overwhelmed and discouraged by the frequent compilation and semantic errors in the model-generated code.
We present COCOGEN, a method to allow an LLM to leverage the code repository of a software project to enhance the quality of the generated code. For a given LLM-generated code sample, COCOGEN first compiles it and identifies context-related errors. It then retrieves the related context from the code repository to fix the errors. This iterative generation and verification process proceeds repetitively until no error is identified in the generated solution. We demonstrate that COCOGEN enhances the accuracy of generation (as indicated by pass rates) by bridging the gap between the repository context and the intended solution.
We evaluate COCOGEN by applying it to the CoderEval benchmarking dataset (Yu et al., 2024),
which consists of code generation tasks utilizing project-specific context. We test COCOGEN on two popular code generation models: the GPT-3.5Turbo (OpenAI, 2023) and Code Llama (Rozière et al., 2023). Experimental results demonstrate that COCOGEN significantly improves the repositorylevel code generation performance of different dependency levels, outperforming the baseline by over 80% relative pass rates in generating functions dependent on project-specific contexts. Moreover, our iterative method consistently enhances the performance of vanilla retrieval-augmented generation. We also provide a comprehensive analysis of the effectiveness and limitations of COCOGEN, offering insights for future research.
This paper makes the following contributions:
• An empirical study to analyze the error distribution in self-contained and repository-level code generation, highlighting the significance of precise and grounded program context in generating code at the project level (Section 2.2);
• A new iterative generation-verification-retrieval method that leverages the program compiler to eliminate context-related errors in repositorylevel code generation (Section 3);
• Extensive experiments and analysis based on two LLMs, i.e., GPT-3.5-Turbo and Code Llama (13B), showing the effectiveness of the proposed COCOGEN method (Section 5).
The source code and dataset used in this paper are available at: https://github.com/ CGCL-codes/naturalcc/tree/main/ examples/cocogen.

heading: 2 Background

heading: 2.1 LLM-based Code Generation
text: Our work targets the code generation task, which produces source code from a natural-language description complemented by programming context (e.g., project-specific APIs, and data structures). We denote this input as x. Given x, it is first converted to a sequence of tokens x = [x1, . . . , x|x|], and a generative Language Model (LM) pLM(x) predicts new tokens sequentially. At each step t, the LM calculates the probability distribution of the next token as pLM(xt|x1:t−1). The probability of generating a program y with token sequence y = [x|x|+1, . . . , x|x|+|y|] is computed as a prod-
uct of next-token distributions given left context:
p(y|x) = |x|+|y|∏
t=|x|+1 pLM(xt|x1:t) (1)
For few-shot learning with large LMs, the generation is also often conditioned on a fixed set of m exemplars, {⟨xi, yi⟩}i≤m. Thus, the LLM-based code generation can be formulated as:
pLM(y|x) = p(y|x, {⟨xi, yi⟩}i≤m) (2)
Practically, the probability of the next token xt depends on a fixed number of preceding tokens xmax(1,t−w) : xt−1, defined by the model’s context window length w, without encompassing the entire software project’s code base.

heading: 2.2 Error Analysis in Code Generation
text: The performance of simple function-level code generation has significantly improved, as demonstrated by an increase in the pass rate from 31.6% with CodeT5 (Wang et al., 2021) to 94.7% with the state-of-the-art Code Llama (Rozière et al., 2023) on the widely-used HumanEval benchmark (Chen et al., 2021). However, a recent study (Yu et al., 2024) shows that existing LLMs for code generation struggle to generate code snippets that are dependent on the project contexts, such as private APIs, classes, data structures, or type constraints. To this end, various benchmarks, including ClassEval (Du et al., 2024), CoderEval (Yu et al., 2024), and CrossCodeEval (Ding et al., 2023), have been devised to assess the performance of LLMs in generating context-dependent code within the project.
To better illustrate our motivation, we perform an empirical analysis of when the LLMs fail to generate complex code that is dependent on projectlevel context, on the CoderEval dataset (Yu et al., 2024). This dataset comprises 85 function-level tasks and 145 project-level tasks. Specifically, we select the GPT-3.5-Turbo (OpenAI, 2023) as a target LLM for function-level code generation. Based on it, we select a state-of-the-art method called RepoCoder (Zhang et al., 2023a). RepoCoder retrieves project-level context as an augmentation
Error Type API OBJECT UNDEF OTHER FUNC
14%
2% 7%
5%
72%
17%
1%
47%
1%
34%
Function−level Project−level
Figure 2: Distribution of error types in the generated solutions on CoderEval dataset
and incorporates the five code fragments with the highest similarity scores, as determined by dense passage retrieval (Karpukhin et al., 2020), into the prompt for improved code generation.
As the code is generated, we compile it and collect any errors reported by the compiler or encountered during testing. We have generated 10 candidate solutions for a task, comprising 850 solutions for function-level tasks and 1450 for project-level tasks. We report the Pass@10 rate as 53.57% for function-level tasks and 39.73% for project-level tasks. The code that does not pass the test has been taken into error analysis. Each generated code snippet contains precisely one type of error. If multiple errors are reported by the compiler, the most common error type is selected for analysis. The error distribution reveals that four specific types of errors constitute the majority of all errors encountered. We categorize these errors into: 1) UNDEF, involving Use of Undefined Symbol, 2) API, involving Incorrect Use of APIs, 3) OBJECT, involving Improper Use of an Object, 4) FUNC, involving Runtime or Functional Errors, and 5) OTHER, involving Other Syntax and Semantic Errors. Table 1 presents several errors encountered in compiling and testing. One example is the “UNDEF” error, where a variable AsyncBolt5x0 is referenced but does not exist in the specified module.
Figure 2 illustrates the distribution of error types, under the scenario of function-level code generation and project-level code generation. From this figure, we can observe that the majority of errors are runtime or functional errors, accounting for 72% and 34% for function-level and projectlevel code generation, respectively. Furthermore, the UNDEF errors and the API errors account for substantially high portions of 21% and 64% for function-level and project-level code generation, respectively. COCOGEN addresses both types of errors associated with project context by sup-
plying the relevant project context. Experiments demonstrate that COCOGEN not only fixes these two types of error but also mitigates other compilation errors by providing context feedback on error messages to the code LM, leading directly to an improvement in prediction accuracy.

heading: 3 Methodology

heading: 3.1 Overview
text: Figure 3 depicts the workflow of COCOGEN, consisting of two crucial components: 1) a method for extracting project-level code context through both syntactic and semantic approaches, and 2) a component responsible for iterative generation and evaluation of solutions. This process refines the generated solutions incrementally, ensuring they evolve towards an error-free state that seamlessly aligns with the codebase of the software project.

heading: 3.2 Project-Level Code Context Extraction
text: Supposing that the code generation tools are activated at a specific juncture. In light of the naturallanguage requirement and the code produced by LLMs after an initial iteration, our objective is to extract the semantic context of the generated code from the project’s code base.
Unlike plain texts, source code has syntactic structures that enable precise identification of elements in a project. Thus, in practice, we employ syntax-directed program analysis (Aho et al., 2006) at various points throughout the offline stage to extract the code context at the project level. We initially employ a parser to transform each source
code file within the project into an Abstract Syntax Tree (AST), extracting tree nodes that correspond to classes, functions, or variables. Subsequently, if a node of these types is found to be a child of another node (e.g., the function get_handler and the class AsyncBolt in Figure 3a’s AST) , an edge is created from the parent node to the child node, establishing a hierarchical relationship.
Take the function f1 from Figure 3a as an example. From this figure, we can see that both its semantics (e.g., its docstring), and its syntactic relation between the parent class AsyncBolt and file async/bolt.py are captured. This allows COCOGEN to find the function from the project syntactically using the function’s qualified name AsyncBolt.get_handler(), or semantically according to its docstring “Return Bolt protocol handlers”.

heading: 3.3 Retrieval-Augmented Code Generation
text: We leverage project-level code context in the retrieval-augmented generation paradigm (Zhang et al., 2023a; Ding et al., 2024; Karpukhin et al., 2020), which has been widely adopted to integrate factual knowledge into LLMs and address hallucination issues. In practice, we commence by extracting project-level context from the database through the construction of a Structured Query Language (SQL) query. Following this, we enhance the acquired context by retrieving similar code snippets based on the dense passage retrieval techniques (Karpukhin et al., 2020).
Structural Search. Based on the compiler feedback, we aim to retrieve the relevant project-level context from all extracted ones. We implement this by transforming the textual compiler feedback into an SQL query using the ChatGPT. The prompt used is presented in Figure 4a. Several examples of paired compiler feedback and SQL queries are provided as demonstrations for in-context learning*.
For instance, consider the compiler feedback: No name ’SyncBolt3’ found in module ’async._bolt3’, the resulting SQL query generated by ChatGPT is as follows:
FROM Module m, Class c WHERE m.contains(c) and m.getName() = ‘async._bolt3’ SELECT m, c
Using this SQL query, the code snippets that involve the implementations of AsyncBolt3 will be returned from our constructed database. The detailed process of constructing and querying such SQL database is refer to Appendix G.
Semantic Search. In addition to returning the project-level context retrieved by the SQL query, we also enhance the acquired context by retrieving similar code snippets using dense passage re-
*Only one demonstration example is illustrated in Figure 3(b), the full list of examples can be found in Appendix G.2.
trieval (Karpukhin et al., 2020). In the initial search round, no compilation error is reported, and COCOGEN utilizes the task description string for retrieval. In subsequent searches, COCOGEN employs the error report and the corresponding error line.
Given a natural-language query q, COCOGEN first converts it into an embedding vector by utilizing an encoder network, as follows:
hq = ENCODER(q) (3)
COCOGEN utilizes a pre-trained Transformer network (Vaswani et al., 2017) as the encoder. After generating the query vector, COCOGEN calculates the cosine similarity between the query and embedding vectors of each context entry hc. This similarity measure is defined as:
sim(hq,hc) = h⊺qhc
||hq|| · ||hc|| (4)
and the top-n entries exhibiting the highest similarity to the query are retrieved as results.

heading: 3.4 Refinement with Compiler Feedback
text: Figure 3(b) showcases the iterative refinement pipeline. Given the task requirement and partial function f1, a semantic retrieval is activated to identify similar functions. Specifically, the function f2, which provides equivalent functionality in synchronous scenario, is identified. Utilizing both the retrieved context and the prompt illustrated in Figure 4(b), the language model generates a solution.
However, the generated output mistakenly invokes SyncBolt3 due to its intended use in asynchronous scenarios, not aligning with the synchronous scenario in f2 The compiler’s feedback highlights this error. With this feedback, COCOGEN conducts the structural and semantic search, leading to the discovery of the correct function f3 for asynchronous scenarios. Incorporating the error details and context into the next iteration ensures accurate function invocation. This process goes iterative until no error is reported by the compiler, resulting in an error-free solution that aligns with the project’s environment.
It is noteworthy that COCOGEN does not take into account FUNC errors, which arise during execution despite successful compilation. COCOGEN focuses on addressing compilation errors, which constitute 66% of the total errors in the context of project-level code generation, as shown in Figure 6.

heading: 4 Experimental Setup

heading: 4.1 Models and Datasets
text: To validate the effectiveness of COCOGEN, we select GPT-3.5-Turbo and Code-Llama 13B base† language models for investigation. The technical details of models and their invocation for inference are presented in Appendix A.
We conduct experiments using the Python split of the CoderEval benchmark (Yu et al., 2024), referred to as CoderEval-Python. It is a benchmark designed to evaluate models within realistic software development scenarios. Without loss of generalizability, we concentrate on the Python programming language within this dataset. This benchmark categorizes 230 test samples into six levels of context dependency: 1) self-contained: builtin types/functions, no imports required; 2) slibrunnable: standard libraries/modules, no installation needed; 3) plib-runnable: publicly-available libraries on PyPI/Maven; 4) class-runnable: code outside the function but within a class; 5) filerunnable: code outside the class but within the file; and 6) project-runnable: code in other source files. We concentrate on the last three dependency types, where the solutions are dependent on projectspecific contexts. There are 55, 68, and 23 tasks associated with each dependency level, respectively.
We also evaluate COCOGEN on two functionlevel code generation benchmarks, namely HumanEval (Chen et al., 2021) and MBPP (Austin et al., 2021), as well as a project-level code completion benchmark named CrossCodeEval (Ding et al., 2023), to further validate its generalizability
†https://huggingface.co/codellama/ CodeLlama-13b-hf
across other coding tasks. the statistics and experimental results of these benchmarks can be found in Appendix B and Appendix F.

heading: 4.2 Baseline Methods
text: COCOGEN can function seamlessly and be integrated into existing LLMs, requiring only blackbox access to these models. In this paper, we select two state-of-the-art LLMs for code generation, namely GPT-3.5-Turbo (OpenAI, 2023) and Code Llama (13B) (Rozière et al., 2023), as our base models. To validate the effectiveness of COCOGEN, we compare it with the following baselines: ▷ Direct Generation (Yu et al., 2024). This line of method denotes directly inputting the task requirements into LLM for code generation, without providing additional context. ▷ ReACC (Lu et al., 2022). We employ the retrieval-augmented generation technique introduced in this baseline for code generation tasks. More precisely, we retrieve project contexts aligned with the task instructions semantically through embedding similarity, and leverage them to augment the prompts of LLMs for better code generation. ▷ RepoCoder (Zhang et al., 2023a). Similar to our work, the referenced baseline also proposes the iterative refinement of generated code. Specifically, it involves retrieving similar code snippets derived from the previously generated ones, and employing them to augment the prompts of LLMs. One distinguishing feature is that this baseline does not leverage compiler feedback.

heading: 4.3 Evaluation Metrics
text: We employ the Pass@k metric (Chen et al., 2021; Yu et al., 2024) for code generation tasks, and em-
ploy the code exact match (C-EM), code edit similarity (C-ES), identifier exact match (I-EM), and identifier F1 score (I-F1) to evaluate both the code match rate and the identifier match rate for code completion tasks, follows Ding et al. (2023). We present the details of the Pass@k metric for code generation. For metrics used in the code completion task, readers are referred to Appendix C.
Pass@k. Following previous studies (Chen et al., 2021; Yu et al., 2024), we evaluate the functional correctness of the generated code by executing test cases. We employ the Pass@k metric, where k denotes the number of programs generated for each task. A task is solved if at least one solution passes all unit tests, and we report the overall proportion of solved tasks. To reduce sampling variance, we generate n ≥ k solutions (for this study, n = 20 and k = 1, 5, 10) for each task, count the number of correct solutions c ≤ n that pass the unit tests, and calculate the unbiased estimator:
Pass@k = E [ 1− ( n−c k ) ( n k ) ]
(5)

heading: 5 Results and Analysis

heading: 5.1 Overall Performance of COCOGEN
text: Table 2 shows the overall performance of COCOGEN, assessed against various baselines, on the CoderEval (Yu et al., 2024) and the CrossCodeEval (Ding et al., 2023) dataset, respectively. This table shows that the COCOGEN can significantly outperform other baselines on the project-level code generation task. This trend persists, with a few exceptions noted specifically in terms of Pass@1. We attribute such exceptions to variations in the generated solutions. Selecting a significantly larger n (e.g., 1000), as discussed in (Li et al., 2022), stabilizes the result and eliminates these expectations. Moreover, it becomes evident that models incorporating contextual information, such as ReACC,
RepoCoder, and COCOGEN, exhibit a noteworthy performance superiority over the no context (i.e., direct) model, thereby affirming the practical value of the context in project-level code generation.

heading: 5.2 Effectiveness of the Iterative Refinement
text: Here, we investigate the effectiveness of iterative refinement in code generation with compiler feedback. We conduct an ablation analysis on both RepoCoder and COCOGEN, via removing or retaining the iterative refinement process. Figure 5 shows the performance of RepoCoder and COCOGEN, with respect to varying iterations, on different data splits. This figure clearly illustrates that as the number of iterations increases, the performance of COCOGEN also exhibits a corresponding improvement. The improvement substantiates the efficacy of our suggested iterative refinement process, demonstrating its ability to enhance the generated code through multiple iterations progressively.

heading: 5.3 Ablation on Components
text: To explore the impact of each component within COCOGEN on overall performance, we remove various components from COCOGEN, including the compiler, SQL retriever, and semantic retriever. Notably, the configuration that excludes the compiler and SQL, relying solely on semantic retrieval, is known as RepoCoder (Zhang et al., 2023a).
Table 3 demonstrates the pass rates of COCOGEN across CoderEval-Python. The data indicates that incorporating compiler feedback does improve accuracy, although not markedly substantial. The reason is that highlighting compilation errors alone does not provide related context for resolving them, which is important in project-level coding problems. Also, compared to RepoCoder—which relies solely on semantic retrieval, integrating compiler feedback with project-specific context results in a stable performance improvement. Results on each data split can be found in Appendix F.2.
Further analytical experiments, including the
evaluation of benefits from solely compiler feedback, the performance of COCOGEN on projectlevel code completion and function-level code generation, and the efficiency of static analysis and SQL queries, can be found in Appendix F.

heading: 5.4 Error Analysis and Case Study
text: We also perform an error analysis of the generated code in iterative generation. We follow the categorization of errors defined in Section 2.2, examples of each error type can be found in Table 1. Figure 6 shows the distribution of errors resolved iteratively by our COCOGEN and two baselines. From this figure, we can see that the errors of various types can be effectively resolved after a single iteration of refinement. For instance, UNDEF errors are notably reduced from 5,133 to 1,042 after one iteration. Additionally, it is observed that the RepoCoder baseline, which operates without compiler feedback, manages to rectify API and syntax errors, corroborating the findings in Zhang et al. (2023a). Nonetheless, RepoCoder proves ineffective against UNDEF and OBJECT errors, likely due to the model’s lack of awareness regarding these errors in the absence of compiler feedback.
To thoroughly evaluate COCOGEN’s effectiveness, we focus on scenarios where compilation is successful, but execution fails. In the case illustrated in Figure 7, COCOGEN incorrectly excludes the microseconds field and erroneously adds a month field. This error stems from ambiguously
stated task requirements and the model’s lack of familiarity with the Structure class’s format, despite its definition being available, resulting in misinterpretation of the intended functionality. Further examples are detailed in Appendix E. The observations inspire us to integrate a comprehensive reference comprising documentation, web search results, and code execution log to provide explicit guidelines for code generation in our future work.

heading: 6 Related Work
text: LLM-based Code Generation. Automated code generation has a history spanning several decades, with initial endeavors utilizing rule-based systems (Woods, 1973) and structured prediction (Zelle and Mooney, 1996; Zettlemoyer and Collins, 2005). In recent years, the development of LLMs has led to the emergence of many prominent models in coding tasks. These include openaccess models such as DeepSeek Coder (Bi et al., 2024), Code Llama (Rozière et al., 2023), and StarCoder (Li et al., 2023), alongside commercial offerings like GPT-3.5 (OpenAI, 2023) and GitHub Copilot (Microsoft, 2024). These models and tools have demonstrated significant promise in enhancing code generation capabilities.
Project-level Code Generation. Generating accurate code within a project poses challenges due to the modular design of software engineering, which results in cross-file dependency patterns (Parnas, 1972). Early works augmented N-gram, RNN, and LSTM models with an additional cache model to track project-level changes (Tu et al., 2014; Hellendoorn and Devanbu, 2017). Pashakhanloo et al. (2023) transformed projects into a relational database and proposes a graph walking method to traverse this database. Zan et al. (2022) first used
private API documentation to improve code generation, and Zan et al. (2023) investigated how various components of API documentation influence prediction accuracy. Shrivastava et al. (2023) proposed a code completion framework using a classifier to filter useful repository-level prompt proposals. Lu et al. (2022); Zhang et al. (2023a) proposed to use single or multiple levels of retrieval-augmented generation mechanisms for code generation. Liao et al. (2023) proposed A3-CodGen, which utilizes local, global, and third-party-library information for better context retrieval. (Yu et al., 2024; Liu et al., 2023; Zhou et al., 2022; Ding et al., 2023) proposed benchmarks and datasets for repositorylevel code generation tasks.
Post-processing of LLMs for Code Generation. To identify correct code generated by LLM, researchers employ post-processing techniques to further rank and filter the generated code. Inala et al. (2022) trained a fault-aware neural ranker that ranks multiple code samples based on compilation feedback. AlphaCode (Li et al., 2022) and Shi et al. (2022) employed filtering methods based on the execution feedback. Zhang et al. (2023c) reranked LLM outputs based on back translation. SelfEdit (Zhang et al., 2023b) employed a generateand-edit approach that utilizes execution results to improve the competitive programming task code quality. Chen et al. (2023) proposed a rubber duck debugging mechanism without human feedback, which identifies mistakes in generated code by investigating the execution results, and explaining the generated code in natural language.

heading: 7 Conclusion
text: In this paper, we show how to use project-specific context information, indexed structural and semantic, to fix compilation errors generated by compilers and improve the quality of code generated by LLMs. Our experimental results show the increased prevalence of errors related to project contexts in project-level code generation compared to function-level code generation. The presented COCOGEN can effectively fix the compilation errors by retrieving related context from the project, thus significantly improving the native LLM baselines on over 80% relative pass rates in generating functions dependent on project-specific contexts.

heading: Acknowledgements
text: This work is supported by the Major Program (JD) of Hubei Province (Grant No. 2023BAA024), and the National Natural Science Foundation of China under grant No. 62102157. This work is also partially supported by Huawei. Fangxin Lu is a visiting student at Huazhong University of Science and Technology, and she is from South-Central Minzu University for Nationalities.
Limitations
In this paper, we utilize compilation information as a means to validate programs. However, it is important to note that even programs that compile successfully can experience execution failures. Furthermore, the successful compilation of programs does not guarantee their safety for execution. As a result, while our findings indicate improvements in quality metrics through the correction of code to properly leverage context, it is imperative to undertake additional verification methods such as testing and manual review to ascertain the functional correctness of the generated code. The challenge of ensuring functional correctness of code encompasses various aspects, including compliance with task requirements, adherence to pre/post-conditions and security requirements, and preserving robustness in generating code. With many questions unanswered, we hope our study can promote a broader view of utilizing computational linguistic technologies in the realm of automated software engineering.
Ethics Statements
We meticulously ensure that all code and models integrated into our research adhere to openaccess policies as outlined by the Creative Commons license. The methodology ensures full compliance with copyright and intellectual property laws, thereby eliminating any potential for infringement or unauthorized use of protected materials. By exclusively utilizing resources that are freely available and legally distributable, we maintain the highest standards of ethical conduct in research. This approach fosters an environment of transparency and respect for the intellectual property rights of others. Our commitment to these principles ensures that our work advances the frontiers of knowledge in a manner that is both legally sound and ethically responsible.

heading: A More Details on Investigated Large Language Models
text: In this paper, we have selected GPT-3.5-Turbo and Code-Llama 13B for investigation.
GPT-3.5-Turbo (OpenAI, 2023). It is a largescale decoder-only model based on the Transformer architecture (Vaswani et al., 2017). It is pre-trained on a diverse array of data, encompassing both natural language and code, and can learn a specific task given an instruction and several demonstration examples. We perform the inference by invocating the model through an online API.
Code Llama (Rozière et al., 2023). It is a family of LLMs for code based on Llama 2 (Touvron et al., 2023) and further trained on 1TB code and natural language tokens. It can be used for code completion and generation following natural language instructions. For Code-Llama 13B, we utilize its base variant, specifically designed for general code generation and understanding.‡
‡The model can be accessed via https://huggingface.co/codellama/ CodeLlama-13b-hf.
In the inference stage, we set the decoding temperature to 0.7, and adopt the top-k sampling strategy. We implement the retrieval modules based on the text-adamodel introduced by OpenAI (OpenAI, 2022), which is effective in both natural language search and code search. The embedding dimension is 1,536 for the text-ada model. We retrieve at most 5 entries for each query. All the experiments in this paper are conducted on a Linux server with 128GB memory, with four 32GB Tesla V100 GPUs.

heading: B More Details on the Investigated Dataset
text: In this paper, we also use three recently proposed benchmarks evaluating the project-level code completion and function-level code generation ability. We provide the details of each benchmark.
CrossCodeEval (Ding et al., 2023). It is a project-level code completion benchmark that necessitates cross-file contextual understanding to complete the code accurately. CrossCodeEval is built on a diverse set of real-world, open-sourced, permissively-licensed repositories, including 471 repositories, 2,665 test cases across 1,368 files for Python language (Ding et al., 2023).
HumanEval (Chen et al., 2021). It is a functionlevel code generation benchmark that consists of 164 programming problems with corresponding Python solutions. These problems cover a range of difficulty levels and programming concepts. HumanEval is widely used in evaluating LMs for code generation tasks due to its focus on real-world coding scenarios and its comprehensive test cases.
MBPP (Austin et al., 2021). It is a functionlevel code generation benchmark consisting of 974 crowd-sourced Python programming problems. These problems are designed to be solvable by entry-level programmers and cover programming fundamentals and standard library functionalities. Each problem includes a task description, a code solution, and several automated test cases.

heading: C More Details on the Evaluation Metrics
text: In this work, we use several evaluation metrics for method generation and line completion tasks. This includes:
Exact Match. This metric assesses whether each character of the model’s predicted code exactly
matches each character of the correct answer. All characters are the same yields EM = 1, while any discrepancy results in EM = 0. This is a strict all-or-nothing metric, where a single character difference results in a score of 0. The average value of EM on all test cases is reported.
Edit Similarity (Levenshtein, 1965). This metric quantifies how dissimilar two strings are to one another. It is measured by counting the minimum number of operations required to transform one string into the other:
lev(a, b) =    |a| if |b| = 0, |b| if |a| = 0, lev(tail(a), tail(b)) if head(a) = head(b), 1 + min    lev(tail(a), b) lev(a, tail(b))
lev(tail(a), tail(b))
otherwise
(6)
F1 Score (Schütze et al., 2008). This metric evaluates the balance between precision and recall for the retrieved identifiers. The F1 score is based on the number of identifiers that both the prediction and the ground truth share. Precision is defined as the ratio of the shared identifiers to the total number of identifiers in the prediction, while recall is the ratio of the shared identifiers to the total number of identifiers in the ground truth. The F1 score is then calculated as the harmonic mean of the recall rate and the precision rate:
F1 = 2
recall−1 + precision−1 (7)

heading: D Errors Reported by the Compiler
text: We utilize pylint for error checking; it is a static code analyzer designed to inspect code within a project without executing it. The analyzer accepts the entire file containing the solution along with associated source files as input, and then it extracts errors that pertain directly to the lines in the generated solution from the entirety of identified errors. pylint generates a range of diagnostic messages, encompassing errors, warnings, recommendations for adhering to language conventions, and suggestions for refactoring to adhere to best coding practices. However, our focus is solely on the errors.
Each error is identified by an error code and described with an error message. The error code is a numerical identifier that reflects the error’s nature, while the error message provides a concise description of the error. pylint recognizes 133
distinct error types, and only 12 of these are reported frequently on the CoderEval-Python dataset. We have categorized these errors into four groups based on their characteristics, detailed in Table 1. The FUNC error category is also included in the error distribution analysis but is excluded from the compiler feedback pipeline. This exclusion is because such errors can only be identified with execution, a process that extends beyond the scope of static analysis and is unsuitable for the real-time generation and refinement pipeline. The specific types of each error and corresponding error codes are documented in Table 4.

heading: E More Details About the Error Cases
text: We delve into additional cases where COCOGEN struggles to generate viable solutions, elucidating the underlying causes to foster further research. We identify two primary factors leading to execution errors despite successful compilation. The first factor is the occurrence of degenerate solutions, such as overly simplistic or redundant code. This phe-
nomenon is discovered and explored in (Zhang et al., 2023c). Figure 8 exemplifies this issue, showcasing a solution that omits necessary validity checks and error handling, neither of which are specified in the task requirements, nor produced by COCOGEN’s outputs. Our analysis reveals that degenerate solutions frequently pass compilation, rendering compiler-based verification ineffective.
Another prevalent mistake involves misinterpreting task requirements, resulting in solutions that lack logical coherence. Figure 9 depicts an instance where the assignment is to leverage the pre-existing Point class; instead, the LM disregards this specification and redundantly recreates the class. This underscores the challenge LLMs face in accurately comprehending prompts and generating appropriate solutions.

heading: F More Analytic Experiments
text: F.1 Usefulness of Compiler Feedback
Here, we examine the usefulness of compiler feedback by integrating it into three baseline models: Vanilla, ReACC, and RepoCoder, each considered separately. We conduct experiments using the GPT3.5-Turbo and subsequently reported the average score across the entire CoderEval-Python dataset consisting of six levels of context dependencies, as shown in Table 5. The table provides clear evidence that incorporating compiler feedback yields a notable enhancement in model performance for code generation. Specifically, a comparison between RepoCoder with and without compiler feedback reveals a substantial increase in Pass@1 from 36.82 to 38.00.
F.2 More Results on the Ablation Study
To further examine the performance of each component of COCOGEN across different dependency levels of test cases, we evaluate COCOGEN at various dependency levels from the CoderEval-Python test suite. The results are presented in Table 6, Table 7, and Table 8. The results show that utilizing compiler feedback and structural queries consistently demonstrates performance improvements across most scenarios. Furthermore, it achieves better performance when stronger cross-file dependencies are present (i.e., at the file-level and project-level tasks), indicating that COCOGEN can accurately capture the context across the project.
F.3 Varying the Number of Iterations
To investigate whether iteration could continuously improve performance, we increase the number of iterations to ten. As shown in Table 9, with more iterations, the performance of COCOGEN reaches a plateau. This indicates that there are still errors that cannot got repaired by COCOGEN, leaving for further research.
F.4 COCOGEN on Project-level Code Completion
We also evaluate COCOGEN on project-level code completion tasks. Although COCOGEN is originally designed for code generation tasks, the compiler feedback may also benefit this code completion task. The results are presented in Table 10. From the table, we can see that the compiler feedback does improve performance, although not as significantly as it does in the code generation task.
F.5 COCOGEN on Function-level Code Generation
To assess the efficacy of COCOGEN in functionlevel code generation, we evaluate it on two commonly used datasets: HumanEval (Chen et al., 2021) and MBPP (Austin et al., 2021). We retain the compiler feedback model in COCOGEN, and conduct tests in a 0-shot scenario, generating 10 code samples for each of the 163 test cases for HumanEval and 974 cases for MBPP, totaling 11,370 samples. The generation and refinement loop iterates 3 times, as shown in Table 11 and Table 12.
The data presented in the table illustrates that compilation feedback improves the accuracy of the generated code, although its efficacy is somewhat limited. We manually review 7 compilation errors not resolved in HumanEval after the first iteration, and find that 6 out of 7 instances are caused by the LLM mistakenly generating code segments enclosed in Markdown code block markers (‘‘‘), resulting in compilation failures. Additionally, another instance of compilation error is identified in the following generated solution:
def is_simple_power(x, n): return x > 0 and (x == 1 or (n != 1 and x == n**int(round(math.log(x, n))))
This code snippet contains five left parentheses and four right parentheses in the return statement, causing the syntax error. Interestingly, despite the compiler has indicated this syntax error, it is not rectified over three iterations.
In MBPP, we manually observe the 7 errors not
fixed after the last iteration, and identify that two test tasks are responsible for all 7 errors: one named "sum" contributed to six errors, and another called "month_season" resulted in one error. We detail these error cases as follows:
def month_season(month, day): seasons = "spring": [(3, 20), (6, 20)], "summer": [(6, 21), (9, 22)], "autumn": [(9, 23), (12, 20)], "winter": [(12, 21), (3, 19)] for season, (start, end) in seasons.items(): if (month == start[0] and day >= start[1]) or (month == end[0] and day <= end[1]): return season return "Invalid input"
The error arises because ’start’ and ’end’ are not included in seasons.items, which appears strange. We speculate that this issue stems from the presence of functions with the same name within the training set.
def sum(a, b): common_divisors = [i for i in range(1, min(a, b) + 1) if a % i == 0 and b % i == 0] return sum(common_divisors) if common_divisors else 0
The root cause of this error lies in the model’s confusion when a user-defined function name coincides with that of a system library function sum, representing a category of potential issues.
F.6 Efficiency of Static Analysis
Utilizing static analysis tools to inspect code typically results in increased latency. To investigate whether the latency impacts the usability of COCOGEN, we measure its latency at each test case. We have observed that in experiments, the speed of static analysis and structural query is relatively fast. It is because the semantic checker (i.e., pylint), only analyzes the current file and its dependent files. The average latency is recorded at 1.27 seconds, with a minimum of 0.359 seconds and a maximum of 6.984 seconds. We deem this latency to be acceptable.

heading: G Technical Details of Structural Query System
text: G.1 The Design Principle
In COCOGEN, we employ the Structural Query Language (SQL) to perform structured queries on code repositories. The SQL syntax used here is CodeQL, a specialized version designed for software repository mining, capable of conducting complex control-flow and data-flow analyses on a set of code files.
We extracted 11 error codes from the compiler (shown in Table 4), each corresponding to one of four error categories as in Section 2.2. For efficiency, two authors manually inspected the most frequently occurring error codes, precomputed six data tables from the project graph, and hard-coded the structural context retrieval procedure for the following error codes: E0001 (syntax error), E0602 (undefined-variable), E1101 (nomember), E0213 (no-self-argument), and E0102
(function redefined). The tables to retrieve structural project context for these frequently occurred errors are presented in Table 13.
Whenever the compiler reports an error, if it is a frequently occurring error, its retrieval is done from these precomputed data tables. Otherwise, the COCOGEN invokes the LLM to generate an SQL query statement, which is then executed on the project graph.
In the CoderEval-Python benchmark, with iteration rounds set to 3, related project contexts of 93.2% of errors are successfully retrieved. For the remaining 6.8% of cases, there are two reasons for failure: first, the LLM does not follow the demonstrated CodeQL examples and generates queries that are syntactically or semantically incorrect, thus rejected by the CodeQL query system; second, the query does not return anything because the specified conditions in the query are not satisfied in project context.
G.2 Demonstration Examples of Structural Queries
We present a demonstration example in Section 3.3 to compose the structural query. The example focuses on identifying and addressing instances of missing or incorrectly utilized context entries. We detail several example error messages along with their corresponding structural queries. A comprehensive list of these examples can be found in Table 14. The four queries shown in the table are written manually by one of the authors, and designed to handle four representative types of compilation errors. These sampled error messages are reported by compilers when generating solutions without project context. The specific four messages are chosen randomly from the compilation log.

heading: H Algorithm for Project Database Construction
text: We present the comprehensive algorithm for constructing the project database and generating code with COCOGEN. The algorithm for building the project database is detailed in Algorithm 1. It involves identifying source files in the project by extracting all files that end with a .py extension. To parse Python source files, we employ the tree-sitter-python parser for generating abstract syntax trees and codeql-python to extract the property of a context entry node. To encode passages to vectors, we utilize text-embedding-ada-002, a text embedding model provided by OpenAI and accessible via online APIs (OpenAI, 2022).
Algorithm 1 Project Database Construction Require: SOURCEFILESET: A set of project source files Require: PARSER: A parser for source files Require: ENCODER: A passage encoder transforms text to numerical vector Ensure: databaseEntries: Entries in the project database
1: databaseEntries← ∅ 2: for each sourceF ile in SOURCEFILESET do 3: nodesForV isit← ⟨⟩ 4: propertyPrefixSeq ← ⟨⟩ 5: astF ile← PARSER(sourceF ile) 6: nodesForV isit.ADD(astF ile.rootNode) 7: while nodesForV isit is not empty do 8: currentNode← nodesForV isit.POP() 9: if currentNode is PREFIXMARK then
10: propertyPrefixSeq.POP() 11: end if 12: if currentNode.type is in [VARIABLETYPE, FUNCTIONTYPE, CLASSTYPE] then 13: nodeProperty ← GETPROPERTIES(currentNode) 14: nodeSchema← ⟨propertyPrefixSeq, currentNode, nodeProperty⟩ 15: nodeEmbedding ← ENCODER(nodeSchema) 16: databaseEntries.ADD([nodeSchema, nodeEmbedding]) 17: propertyPrefixSeq.PUSH(currentNode) 18: nodesForV isit.PUSH(PREFIXMARK) 19: end if 20: for childNode in currentNode.CHILDS() do 21: nodesForV isit.PUSH(childNode) 22: end for 23: end while 24: end for

heading: Future Work
text: The observations inspire us to integrate a comprehensive reference comprising documentation, web search results, and code execution log to provide explicit guidelines for code generation in our future work