RepoCoder: Repository-Level Code Completion Through Iterative Retrieval and Generation
Authors: Fengji Zhang
Bei Chen
Yue Zhang
Jacky Keung
Jin Liu
Daoguang Zan
Yi Mao
Jian-Guang Lou
Weizhu Chen
Venue: EMNLP 2023 Main (2023)
Keywords: code completion
large pre-trained language model
code repository
retrieval augmented generation

Abstract: The task of repository-level code completion is to continue writing the unfinished code based on a broader context of the repository. While for automated code completion tools, it is difficult to utilize the useful information scattered in different files. We propose RepoCoder, a simple, generic, and effective framework to address the challenge. It streamlines the repository-level code completion process by incorporating a similarity-based retriever and a pre-trained code language model in an iterative retrieval-generation pipeline. RepoCoder makes effective utilization of repository-level information for code completion and has the ability to generate code at various levels of granularity. Moreover, we propose a new benchmark RepoBench, which consists of the latest and high-quality real-world repositories covering line, API invocation, and function body completion scenarios. Experimental results indicate that RepoCoder significantly improves the In-File completion baseline by over 10% in all settings and consistently outperforms the vanilla retrieval-augmented code completion approach. Furthermore, we validate the effectiveness of RepoCoder through comprehensive analysis, providing valuable insights for future research. Our source code and benchmark will be publicly available after the paper review.

RepoCoder: Repository-Level Code Completion Through Iterative Retrieval and Generation
Fengji Zhang', Bei Chen', Yue Zhang", Jacky Keung', Jin Liu", Daoguang Zan", Yi Jian-Gunng Lou', Weizhu Chen' City University of Hong Kong, "Microsoft Corporation, "Wuhan University cocky.keungcci tyu. edu. eku hinliu@whu.een cn
fengji. Shang@myy ci tyu. edu. (beichen, zhayue, v-dazan, maoyi, jlou, wzchenJemin crosoft. con
Abstract
In-File Completion
Repo-level RAG
RepoCoder
The task of repository- -evell code completion is continue writing unfinished code based broader context of the repository. While for automated code completion tools, is diffi- cult utilize the useful information scattered in different files. We propose RepoCoder, sim- ple, generic, and effective framework to address the challenge. It streamlines the repository- level code completion process by incorporating a similarity- based retriever and a pre-trained code language model in iterative retrieval- generation pipeline. RepoCoder makes effec- tive utilization of repository-level information for code completion and has the ability to generate code at various levels of granularity. Moreover, we propose a new benchmark Re- pofval, which consists of the latest and high- quality real-world repositories covering line, API invocation, and function body completion scenarios. Experimental results indicate that RepoCoder significantly improves the In-File completion baseline by over 10% in all set- tings and consistently outperforms the vanilla retreeeal-aggmeeeed code completion approach. Furthermore, validate the effectiveness of RepoCoder through comprehensive analysis, providing valuable insights for future research. Our source code and benchmark are publicly available: https: https: com/m cwm/micmc/ceeeeeccccoeccceec.ece crcooooft/ TodeT//treee/ain/Reeooon
Unfinished
Repo Files
Unfiniehed
Rapo
Unfinished Code
Code
Retrieval
Retrieval
Generation
Generation
Generation
Predicted Completion
Predicted
Predicted Completion
Completion
Illustration of the In-File code comple- tion method, repository-level Retrieval-Augmented Generation (RAG) method, and the iterative retrieval- generation RepoCoder method.
Figure 1:
2014). Additionally, each repository typically fol- lows customized naming conventions and coding styles (Zou et al., 2019), which contribute to en- hanced readability and maintainability. However. developing effective repository-level code comple- tion tools remains an open problem. Although approaches relying on static code analysis and heuristic rules (Raychev et al., 2014; Svyatkovskiy et al., 2019, 2021) can reliably parse specific repos- itory context, they have limitations in the comple- tion scenario, limiting capability for varying-length completions anywhere in a filc. Mcanwhile, stud- ies (Hellendoorn and Devanbu, 2017; Svyatkovskiy et al., 2020: Ding et al., 2022) tuning language mod- els on labeled data excel in their respective evalua- tion scenarios but face challenges generalizing to unseen repositories without retraining.
1 Introduction
real-world software production, it is crucial for developers to be aware of other files within the repository during programming. This challenge gives rise to the task of repository-level code com- pletion, where automated tools are expected to uti- lize the broader context of repository rather than relying solely in-file information complete unfinished code. Code files within a repository often exhibit interrelated dependencies, including shared utilities, configurations, and cross-API in- vocations resulting from modularization (Tu et al.,
In this paper, we propose an approach to lever- age off-the-shelf retrievers in order to locate valu- able information within a repository and enhance the context for language models. We introducc novel framework called RepoCoder that aims to improve code retrieval and completion perfor- mance. As depicted in Figure/ 1, we enhance the conventional In-File code completion method by incorporating the Retrieval- Aarrival-AAumentte Generation

---

(RAG) technique, which allows us to search for rel- evant code snippets from the repository to assist in generating the code completion. Additionally, we introduce RepoCoder, which employs an iterative pipeline that utilizes the generated code completion to enhance the retrieval process, thus bridging the gap between the retrieval context and the intended completion target. Figure 2 provides example that illustrates the rationale behind our design. We demonstrate that relying solely on the unfinished code insufficient to retrieve useful information from the repository. In the example, the model improvises a statement calling the COLMAP API in the first iteration. The predicted parameters are reasonable yet incorrect. This is because the in- complete code preceding the code completion does not serve adequate retrieval query for the in- tended completion target. However, by performing a subsequent retrieval from the repository using the model's generated completion, we can successfully retrieve the target APT signature and complete the code effectively.
Unfinished Code
def maiinsseff, path)
First Iteration Retrieved Code
imrge
uti lttlllcoy..nnneerrr
ingdeee, img_dtn) sownscale
wtwsssdossscaet.......ooos contig)
def process, Ings(...
call CouPse executable
cozy inages to din utils. list( downscale lmages utils. dranscale save 1son
First Iteration Incorrect Completion
utils.si up (son(
caleeg_peth patn (eolinee""""
sodel_path path("eooe]")
trein_path path( tast_pate path "tast"]
Second Iteration Retrieved Code
Second Iteration Completion
Pathy
model Model, True
Pun_colw:pp
teage_sat" path:"leeae""
colnap_pati pathi"coalaa"") cavera_racel sel", modde,, apu
Figure 2: motivating example showcasing the utiliza- tion of model predictions to enhance the performance of code retrieval.
We propose RepoCoder, a novel itera- tive retrieval-generation framework for the repository-level code completion task.
Furthermore, we introduce the RepoEval bench- mark designed for evaluating the repository-level code completion task, which is constructed using the latest high-quality repositories sourced from GitHub'. By introducing RepoEval, we address the lack of established benchmarks in the repository- level scenario. Notably, RepoEval is the first bench- mark that encompasses three levels of code com- pletion granularity: line, API invocation, and func- tion body. We also leverage unit tests present in the repository to enhance the accuracy of evalua- tion, which overcomes the limitations of similarity- based metrics. To rigorously validate the effective- ness of RepoCoder, we conduct extensive exper- iments using different language models of vary- ing sizes, including GPT-3.5-Turbo' and CODE- GEN (Nijkamp et al., 2022). Experimental re- sults demonstrate that RepoCoder achieves signifi- cant improvements over In-File completion perfor- mance, surpassing the baseline by over 10% across different experimental settings. Moreover, our iter- ative framework consistently enhances the perfor- mance of vanilla retrieval- auggented generation. We also provide a comprehensive analysis of the effectiveness and limitations of RepoCoder, offer- ing insights for future research. Our contributions can be summarized as follows:
We introduce the RepoEval benchmark, en- abling the evaluation of repository-leww code completion with varying levels of granularity and improved evaluation accuracy through the utilization of unit tests.
Through rigorous experimentation, we demon- strate that RepoCoder significantly outper- forms the In-File code completion paradigm and enhances the performance of vanilla retrieval- augmented generation.
2
Methodology
2.1 Overall Framework
The task of code completion using language model M can be generally described as Y M(),, where Y represents the predicted tokens and X corresponds to the in-file unfinished code. By introducing an additional code retrieval model R, we can transform the code completion pipeline into Retrieval-Augmenter Generation (RAG) ap- proach. Initially, we establish retrieval databasc by partitioning the code files from the reposi- tory into collection of code snippets Crepo {c, (2, ....... Subsequently, we utilize the retrieval model R to extract the most relevant code snippets from Crepo by employing the unfinished code
'https://github. 'nttps://plaffot openai. com/docs//mdells. gpt-3-5

---

the retrieval query. This process yields set of retrieved code snippets Cret = R(Crepo> X). Fol- lowing this, we leverage the language model M to perform code completion, resulting in the predic- tion y = (Cree,, X). Consequently, we are able to incorporate the contextual information from the repository level during the code completion task. However, using the unfinished code X as the sole retrieval query introduces a gap between the retrieval context and the intended compiction tar- get, as exemplified in Figure 2. To address this limitation, we propose RepoCoder, retrieval-generation pipeline designed to further en- hance the performance of the vanilla RAG method. Specifically, for the i-th retrieval- -geerratinnnnnnionnn iteration, RepoCoder utilizes the previous model prediction ý to construct new query for the retrieval process. This leads to the generation of another set of relevant code snippets Oiet (Crrpoo X, Y 1). Subsequently, new prompt is constructed using Ciee, resulting in the genera- tion of new prediction y M( X). The newly generated code completion can serve as ei- ther the output of RepoCoder or be utilized for the subsequent eetrieval-generation iteration.
# Below are sone referential code fragments
Retrieved
from other files:
Code
#
code fragient found
# http://sst__ppeiie__comm
# wtt.teesssspppfooorccctct
1- "cuda")
def test to device(self):
components caappeiine_csss(*ccceeeeee class((*corponents)
pipe self-pipeline
wttp:prres__srddissoeeee pipe.to("cpu"))
"a"Ba on above, complete the following code:""e
Sunittest.skipIf(tten device != "cuda")
Unfinished Code
def test float16_ (eriiiiinnneenneeee))
components self.get_dumy, get__dumy_comppnenss.
iterative
pipe pipf.pipeliee carliiieeeesssseeeee class(("componentt)) pipe.to wowttooooo_ceeiccee
Model
Prediction
Figure 3: A visual example demonstrating the format of the RepoCoder prompt, which combines the retrieved code snippets from the repository with the unfinished code present in the target file.
1)
between the retrieval context, based on X, and the intended completion target, which is to continue writing X. A possible solution is adjust Ctaa by shifting each code snippet down by a few lines include the subsequent code. Although this shift- ing approach has shown effectiveness in previous work (Lu et al., 2022), indiscriminately shifting all retrieved code snippets without considering their content may not always be appropriate.
Importantly, it is worth noting that the param- eters of M and R remain unchanged throughout the entire process. Moreover, there is no require- ment for static code analysis tools or heuristic rules to construct the retrieval database. the follow- ing subsections, we provide a detailed explanation of the code retrieval process (Section 2.2) and the code generation process (Section 2.3).
To address this issue, RepoCoder augments the retrieval query during the i-th iteration (i > 1) with the previously generated code Y Despite the lack of customized information for new repos- itories, pre-trained code language models have demonstrated impressive general-domain under- standing and generation capabilities. The generated code y can provide valuable supplementary in- formation for the retrieval process, even though its correctness may not be guaranteed. There- fore, for the i-th iteration of retrieval (i > 1), the query is constructed by concatenating the last (S - -S,) lines of X with the first S, lines of pi-1, This approach yields the grounded retrieval results Ct R(Crepo X, fi-1).
2.2 Code Retrieval
The retriever utilized within the RepoCoder frame- work can be any model capable of searching for relevant documents given a specific query. To con- struct the retrieval database, a sliding window ap- proach is employed. The sliding window traverses the files in the repository and extracts contiguous lines of code that fit within the window size, de- noted as S.. The sliding window moves a fixed number of lines at each iteration, which is referred to as the sliding size, denoted as S.,
2.3
Code Generation
The generator employed within the RepoCoder framework can be any pre-trained language model capable of predicting subsequent tokens given a specific prompt. As mentioned carlier, it is crucial to incorporate both the context from the repository Crepo and the context within the target file for ef- fective code completion. This enables the model to
During the initial retrieval process, when model prediction is available, the query formu- lated using the last Sww lines of the unfinished code X. Consequently, the most similar code snippets are retrieved using the retrieval model, resulting in Cj = RCCrepo: X). However, a gap exists

---

ID
Name
License
Created
N.
sponding source repository, file path, line numbers, and ground truth completion. For analysis and unit test execution, complete copies of the repositories are archived as of January 2023.
Function Body Completion Dataset
1.
imagen
MIT License
2022-05 23
14
67
2.
tracr
Apache V2.0
2022-12-01
56
146
lightmmm
Apache V2.0
2022-02-10
64
inspection
Apache V2.0
2022-05-05
32
To construct RepoEval, we first meticulously curate a collection of Python repositories from GitHub that satisfy the following criteria: open- source license, created after January 1, 20223, non- fork original repositories, over 100 stars, over 80%
5.
omnivore
CC BY-NC 4.0
2022-01-20
66
22
6.
redframes
BSD-2-Clause
2022-08-21
49
42
Line and API Invocation Completion Datasets
7.
rl
MIT License
2022-02-01
163
400
8.
ACE
Apache V2.0
2022-11-23
425
400
9.
vizier
Apache V2.0
2022-02-16
188
400
of files written in Python, and explicit unit tests. Furthermore, mitigate potential biases, we em- ploy a random selection process for the repositories and create three distinct datasets for line comple- tion, API invocation completion, and function body completion. Additional details regarding the se- lected repositories can be found in Table 1.
10.
fortuna
Apache V2.0
2022-11-17
168
400
11.
evaluate
Apache V2.0
2022-03-30
180
400
12.
diffusers
Apache V2.0
2022-05-30
305
400
13.
nerfstudio
Apache V2.0
2022-05-31
157
400
14.
FedScope
Apache V2.0
2022-03-24
443
400
Table I: The repositories utilized in our RepoEval benchmark, presenting statisties obtained from the Github API as of January 2023. ID denotes the reposi- tory IDs. F. indicates the total number of Python source files. N. represents the number of extracted test samples from each repository. For brevity, additional repository information can found Appendix A.
Line completion: In adherence to the conven- tions of code completion benchmarks (Lu et al., 2021, 2022), we implement the line completion scenario. First, according to the above-mentioned criteria, we select 8 repositories that vary in size and cover different domains. Then we randomly select 200 lines to complete from each repository, ensuring the lines are non-repetitive, not code com- ments, and each line contains at least 5 tokens. Eventually, a total of 1600 test samples are gener- ated for the line completion dataset.
leverage grounding information and enhances its generalization ability to unseen repositories. In the RepoCoder framework, we retrieve the relevant code examples, denoted as Cret, from the repository and concatenate them with the un- finished code X. To ensure readability and com- prehension, we create a prompt template that seam- lessly integrates X and Crez, as illustrated in Figure 3. The retrieved code snippets are arranged in as- cending order based their similarity scores to the query. Each code snippet is accompanied by its original file path, and the maximum number of code snippets included in the prompt, denoted as K; depends on the available prompt length. Ultimately, the prompt contains as much relevant information as possible to facilitate code completion.
API Invocation Completion: We also choose to test the API completion scenario, especially in- repository defined APIs. is a harder problem than the completion of built-in or third-party APIs due to the lack of customized training data (Hellen- doorn et al., 2019). We utilize the same group of repositories in the line dataset and parse the target repositories to locate invocations of in-repository APIs. From these candidates, we then randomly lect 200 non-repetitive API invocations from each repository, resulting in total of 1600 test samples for the API invocation completion dataset.
3 Benchmark Construction
To facilitate the evaluation of code completion tools in the repository-level scenario, we propose a novel RepoEval benchmark. This benchmark is carefully constructed using the latest high- qushh-qulttyy reposito- ries sourced from GitHub and encompasses three levels of code completion granularity: line, API in- vocation, and function body. To assess the correct- ness of completed functions, we utilize unit tests present in the repository instead of relying solely on similarity- based metrics. Each sample in the RepoEval benchmark is annotated with the corre-
Function Body Completion: Alongside the line and API completion evaluations, we also assess the ability to complete function bodies, which quires executing unit tests present in the repository. However, running tests can be time-consuming and computationally expensive. To address this, we ran- donly select a separate set of smaller-scale reposi- tories that are easy to deploy. Within these repos- itories, we locate functions covered by unit tests
"The training data of GPT-3.5-Turhe and CODEGEN is to 2021. We use data from 2022 to provent data leakage.

---

and select function bodies containing 3 to 30 lines of code to complete. This yields a total of 373 test samples for the function body completion dataset.
ing code generation capabilities. The first model, GPT-3.5-Turbo, state-of-the-art commercial code generation model with billions of trainable pa- rameters and has been pre-trained on an extensive code corpus. Access to GPT-3.5- Turbo is obtained through the API provided by OpenAI. The second model, CODEGEN, is an open-source code gener- ation model that has multiple published versions with varying model sizes and training data. In our experiments, we utilize three versions of CODE- GEN model with 6B, 2B, and 350M parameters.
4
Experimental Setup
4.1 Methods for Comparison
In-File Completion: Previous studies (Chen et al.. 2021; Nijkamp et al., 2022: Chen et al., 2022) have demonstrated the effectiveness of uti- lizing large pre-trained language models for code generation in zero-shot completion manner, con- ditioned on the provided context. Furthermore, it has been established that incorporating in-file context is beneficial for code completion scenar- ios (Clement et al., 2021). Hence, as a baseline, we implement an In-File completion method by pop- ulating the prompt with the unfinished code and directly utilizing the pre-trained code generation model to predict the code completion.
Mlyper-parameters We found that RepoCoder's performance was not highly sensitive to changes in apper-parameter.. Therefore, for our experiments on RepoEval, we assign hyper-parameeerss based on our experience. Specifically, the maximum number of tokens for the combined input prompt and out- put prediction is set to 4, 096 for GPT-3.5-Turbo and 2, 048 for CODEGEN. The length of retrieved code snippets is set half the prompt length. For line and API completion, the maximum number of tokens in the generated completion (Y), the line length of the sliding window (Sw), and the sliding size (Sw) are set to 100, 20, and 10 respectively. For function body completion, these values are ad- justed to 500, 50, and 10. The maximum number of retrieved snippets (K) is set to 10. The same hyper- parameters were used for the single-iteration RAG, iterative RepoCoder, and Oracle baselines, ensur- ing fair comparison between methods. Notably, given that these parameters are intricately linked to the programming language and contextual sce- narios, practitioners should make adjustments to ensure optimal real-world performance.
A key contribution of Re- poCode is the integration of model predictions for retrieval, bridging the gap between retrieval and the intended completion target. To showcase the effectiveness of this approach, we devise an oracle generation method for com- parison purposes. This method performs single retrieval process obtain relevant code snippets. denoted as C%e,. by utilizing the last S.. S, lines of X and the first S₂ lines of the ground truth code, Y. Subsequently, the completion code, denoted as Y, is generated through M (CCr X). This allows us to achieve the upper bound of performance for RepoCoder, conditioned on the retrieval model R and the generation model M.
Oracle Method:
red:
4.3 Evaluation Metrics
4.2 Implementation Details
Similarity-based Evaluation: Following estab- lished practices in code completion research (Lu et al., 2021, 2022), we evaluate our line and API completion datasets using two metries: Exact Match (EM) and Edit Similarity (ES). The EM score a binary metric that takes the value of if the predicted code exactly matches the ground truth code, and otherwise. The ES metric provides a more fine-grained evaluation and is calculated as Lev(P.Y) where Lev represents the
Retrieval Model: For our main experiments, we employ a sparse bag-of-words model as the re- trieval model, which has demonstrated effective- ness in retrieving similar code snippets (Lu et al., 2022). This model transforms the query and candi- date code snippets into sets of tokens and calculates their similarity using the Jaccard index (Jaccard, 1912), computed as Jaccord(S,, S.) S,nS. SS,S.. where Sy and Se represent the tokens of the query and candidate code snippets, respectively. We also experiment with dense retriever based on UniX- coder (Guo et al., 2022), detailed in Appendix B.
ES = 1 max([V].|IT)) Levenshtein distance (Levenshtein et al., 1966).
Execution-based Evaluation: For the function body completion dataset, we utilize unit tests present in the repository to evaluate functional
Generation Model: We evaluate RepoCoder us- ing four pre-trained language models with vary-

---

RepoCoder Iterations
RepoCoder Iterations
Metric
Oracle
In-File
Metric
Oracle
In-File
2
4
2
3
GPT-3.5 Turbo
GPT-3,5 Turbbo
FM
37.75
40.56
55.31
56.81
57.00
56.63
FM ES
50.13
34.06
47.69
49.19
49.44
49.56
ES
75.43
65.06
74.38
75.11
75.30
75.10
74.50
63.22
73.63
74.43
74.59
74.48
CODRGEN
MONG-6B
CODEGEN
EM ES
48.81
34.56
45.81
47.06
47.75
47.44
EM
40.25
26.19
36.69
38.88
39. 19
39.31
71.02
60.67
69.21
70.10
70.73
70.19
ES
67.94
56.45
64.20
05.52
65.53
65.90
CODEGEN- CODEGEN-MONO-28
MONG-2B
CODEGEN-MONE
EM
47.31
33.63
44.56
46.94
46.69
47.13
FM
39.44
25.44
35.44
37.56
38.44
38.25
ES
69.80
58.99
67.68
68.82
68.62
68.92
ES
66.78
56.88
63.47
64.15
61.53
64.60
CODEGEN
MON0-350M
EM
15.19
29.56
11.88
43.06
43.94
43.06
EM
34.88
22.19
31.75
33.88
33.75
33.81
ES
67.20
55.39
65.05
65.66
65.97
65.62
ES
63.06
52.24
59.82
61.03
60.96
61.06
(a) Completion.
(b) API Invocation Completion.
Table 2: Performance comparison on the line and APT invocation completion datasets. Results present the average performance of each method evaluated using Exact Match (EM) and Edit Similarity (ES) scores, Numbers are shown in percentage (%), with the best performance highlighted in bold.
correctness. This approach is more reliable than similarity-basee metrics assessing the behavior the completed functions. While collecting unit tests can be time-consuming we focus on a real- istic scenario and utilize the unit tests available in GitHub repositories to validate the generated code. We execute the completed code and report the Pass Rate (PR), where PR is if the code passes all the corresponding test cases, and otherwise.
RepoCoder Iterations
Oracle
In-File
2
1.
67
56.72
29.85
53.73
55.22
$5.22
55.22
2.
146
43.84
27.40
41.78
43.84
44.52
44.52
3.
32.81 81
10.94
25.00
34.38
31.25
32.81
4.
34.38
28.13
34.38
37.50
34.38
31.38
5.
31.82
36.36
31.82
6.
38.10
9.52
28.57
38.10
38.10
38.10
All
373
42.66
23.32
38.34
42.63
41.82
42.36
Table 3: Performance comparison on the function body completion dataset using GPT-3.5-Turbo. Results dis- play the Pass Rate (PR) of each method as evaluated using test cases. Numbers are presented in percentage (%), with the best performance highlighted bold. represents the repository IDs, and N. indicates the num- ber test samples in each repository.
5 Experimental Results
5.1 Line and API Completion Datasets
We compare the performance of RepoCoder with the In-File completion method and the Oracle method on the line and API invocation completion datasets using four pre-trained language models and different retrieval-generation iterations. From the results listed in Table 2a and 2b, we find that RepoCoder consistently improves the In-File completion performance on both datasets across all model sizes. The absolute improvements in the Exact Match (EM) and Edit Similarity (ES) scores exceed 10% and 8%, respectively. Re- poCoder also shows competitive results compared to the Oracle method. With two or more iterations, RepoCoder consistently outperforms the vanilla Retrieval-Augmentte Generation (RAG) approach for all language models. Additionally, the CODE- GEN model with 350M parameters is comparable the GPT-3.5-Turbo model with In-File comple- tion when integrated with RepoCoder. We also test RepoCoder using a dense retriever powered
by UniXcoder (Guo et al., 2022) (detailed in Ap- pendix B) and find that the simple sparse retriever achieves equivalent performance, highlighting the robustness of RepoCoder across different code re- trieval and generation models.
5.2 Function Completion Dataset
We proceed to assess the performance of Re- poCoder on the function body completion dataset. To tackle the greater difficulty of function body completion,
employ the GPT-3.5-Turbe model due its superior code understanding and genera- tion capabilities, as well as its larger prompt length suitable for longer function code snippets. The eval- uation results, presented in Table 3, showcase simi- lar trends to our findings on the line and API invo-

---

RepoCoder Iter-
Method
Oracle
RepoCoder Iter-2
Metric
GT-Code
In-File
1
2
Dataset
Line
API
Line
API
GPT-3.5-
Location
Statistics
EM
55.54
34.42
53.63
35.07
Imported
4.22%
8.16%
3.22%
9.06%
77.67
62.75
77.68
78.40
Current File
3.86%
4.05%
3.32%
4.10%
Recall
100.0
86.04
90.34
Current Directory
46.41%
58. 18%4
45.68%
59.58%
Similar Import
82.15%
86.71%
82.80%
87.70%
GODEGEN-MONO-6N
Similar Name
52.23%
65.10%
53.40%
64.20%
44.78
26.87
41.09
44.04
Others
7.27%
4.43%
7.77%
3.35%
ES
71.47
56.42
67.
68.55
Recall
100.0
76.27
82.92
Eligible Samples
Samples 333
294
312
276
CODEGEN- OODEGEN-MONO-330 35500-500
Code Snippers
2202
1851
2047
1732
FM
37.86
22.25
35.64
38. 13
66.20
52.40
62.82
64.26
Table 5: Locations of retrieved code snippets when the Oracle/Repooodes method outperforms the In-File completion method using UPT-3.5-Turbbe on the line and API completion datasets.
Recall
100.0
76.27
80.89
Table 4: Performance comparison on the test samples extracted from the API completion dataset using GT- Code, In-File, and RepoCoder methods. Results present the averaged performance of each method as evaluated by Exact Match (EM), Edit Similarity (ES), and Recall, Numbers are shown in percentage (%).
ones to include in the completion prompt. We refer to this method as GT-Code and compare its perfor- mance against the In-File and RepoCoder methods. Additionally, we show the recall performance of RepoCoder by counting the number of retrieved code snippets containing invocation examples of the ground truth API.
cation completion datasets. Across most reposito- ries, RepoCoder exhibits significant improvement over the In-File completion method and competi- tive performance compared to the Oracle method.
Since not every API the test dataset has invoca- tion examples in other files, and we also exclude the invocation examples existing in the input prompt for the model, we finally extract from the API in- vocation dataset 1046 and 1083 eligible test sam- ples respectively for the GPT-3.5-Turbu and CODE- GEN models conduct the experiment. From the obtained results in Table 4, we observe that the GT-Code method, which utilizes ground truth API invocation examples, generally achieves the best performance among all methods. Furthermore, Re- poCoder with two iterations exhibits higher recall for ground truth API invocations compared to a single-iteration, which likely contributes to its su- perior code completion performance. Notably, the language model grows more powerful, the re- call value using RepoCoder Iter-2 also increases, indicating the model predictions indeed assist the retrieval process and emphasizing the effectiveness of RepoCoder.
Moreover, with additional vetrieval-generation it- erations, RepoCoder consistently outperforms the vanilla Retrieval-Augmented Generation (RAG) ap- proach. These results reaffirm the effectiveness of our approach.
6 Analysis
In this section, we conduct further analyses on the retrieved code snippets to gain a deeper understand- ing RepoCoder and provide valuable insights for future research.
6.1 Quality of Retrieved Code
We observe significant impact of the retrieved code's quality on code completion performance. And the most helpful code snippets typically con- tain code statements similar to the target comple- tion or demonstrate example usages of the target API invocation. Then, to validate the correlation between retrieval quality and completion perfor- mance, we design an analysis experiment using the API invocation completion dataset. In this exper- iment, we leverage a static code analysis tool to locate code snippets in other files that include invo- cations of the ground truth API. Subsequently, we rank these code snippets based on their similarity to the unfinished code and select the most similar
6.2 Locations of Retrieved Code
The retrieval of code snippets provides valuable contextual information from other files to enhance the context for language models. We conduct a separate experiment to study the various locations from which effective retrieval occurs. Specifically, we select test samples that are successfully pre-

---

dicted by the Oracle/RepoCoder method but not by In-File completion using GPT-3.5-Turbo. This yields a number of eligible test samples and re- trieved code snippets for the line and APT invoca- tion completion datasets. To determine the original source of these code snippets, we adopt a classifi- cation scheme inspired by Shrivastava et al. (2022), consisting of five distinct file locations: 1. Im- ported: code from file imported by the target file. 2. Current File: code from the excluded content of the target file. 3. Current Directory: code from a file in the same directory as the target file. 4. Sim- ilar Import: code from a file sharing at least one same API import with the target file. 5. Similar Name: code from a file with a file name sharing at least one token with the target file (assuming snake-case style file names).
labeled data and fine-tuning models for different applications remains resource-imlensive. recent years, there has been significant attention on Large Language Models (LLMs) for code completion. A study by Shrivastava et al. (2022) also explores the scenario of repository-level code completion. Despite its innovative approach, the study relies on inflexible heuristics and classifier training for prompt construction. This highlights the ongoing challenges in effectively leveraging LLMs for code completion and the need for further research.
Joint Modeling Retrieval and Generation: De- spite the impressive capabilities of LLMs (Brown et al., 2020; Thoppilan et al., 2022; Chowdhery et al., 2022), their offline training paradigm often limits access to customized and up-to-date informa- tion. Recent studies have started exploring the joint modeling of retrieval and generation in knowledge- intensive tasks, such as question answering (Guu et al., 2020; Lewis et al., 2020; Izacard et al., 2022) and dialogue generation (Zhang et al., 2022). This approach has also been extended to code generation by incorporating retrieved documents code ex- amples into the generation process (Rizwan Parvez et al., 2021; Zhou et al., 2022; Lu al., 2022; Zan et al., 2022). As language models have become increasingly sophisticated, there a growing trend towards in-context joint retrieval and generation, treating the LLM as fixed black box (Levine et al.. 2022; Ram et al., 2023; Shi et al., 2023). More- over, some studies have investigated utilizing the model's predictions supplementary context to inform the retrieval process (Mao et al., 2020; Li et al., 2022; Wang et al., 2023; Zemlyanskiy et al., 2022). this work, we demonstrate that adopting an iterative paradigm that combines code retrieval and generation can serve as an effective method for repository leveel code completion.
The results are as outlined in Table 5. Our findings indicate a similar distribution of retrieved code snippets between the Oracle method and Re- poCoder. The majority of code snippets fall within our defined categories, and significant portion of code snippets originates from files with "Similar Import", "Similar Name", or "Current Directory" locations, underscoring the importance of contex- tual information in code completion tasks. Further- more, we conduct an ablation study, wherein we restrict the retrieval process to only the aforemen- tioned file locations. The results reveal degrada- tion in performance, highlighting the efficacy and simplicity of RepoCoder in the retrieval process.
7 Related Work
Repository Context in Code Completion: In- corporating repository-level context into code com- pletion tools has been a long-standing challenge. Traditional code completion techniques typically involve analyzing code to identify potential sugges- tions, followed by re-ranking them (Raychev et al., 2014; Svyatkovskiy et al., 2019, 2021). While this approach offers efficient performance, it lacks the flexibility to generate code at arbitrary granularity. Another line of research treats code completion as a language modeling task, where the next tokens are generated based on the given context. Several methods have been proposed to incorporate reposi- tory context into the training of language models, including n-grams (Tu et al., 2014), LSTMs (Hel- lendoorn and Devanbu, 2017), and Transformers (Svyatkovskiy et al., 2020; Liu et al., 2022: Ding et al., 2022). However, the process of collecting
8 Conclusion and Future Work
In conclusion, we introduce RepoCoder, a straight- forward and effective framework for the repository- level code completion task. By leveraging retriever and a language model, RepoCoder effectively utilizes repository-level information. Through iterative process of retrieval and gen- eration, RepoCoder bridges the gap between re- trieval context and the target code, resulting in im- proved code completion performance. Our exten- sive experiments conducted on the RepoEval bench- mark demonstrate that RepoCoder consistently and

---

significantly enhances In-File completion perfor- mance, surpassing the vanilla Retrieval-Augmented Generation (RAG) approach. Furthermore, our analysis provides valuable insights into the ratio- nale and limitations of RepoCoder. With its sim- plicity, versatility, and effectivvness,, RepoCoder has the potential to become indispensable tool in real-world software development, and aim to further enhance its usability and robustness.
and contextual needs balance accuracy and effi ciency. Nevertheless, improving time efficiency is another important topic that is out of the scope of our current paper.
Limited Exploration of Different Experimental Settings: First, while we have validated the effec- tiveness of RepoCoder, we have not yet explored the potential improvements that can be achieved through different prompt templates. We believe that more careful prompt engineering could en- hance the performance of our approach even fur- ther. Second, our focus this study has primarily been on exploring similarity-
Limitations
Limited Effectiveness for Repositories with Low Code Duplication: Despite we have demon- strated the effectiveness of RepoCoder through ex- tensive experiments and analysis, RepoCoder may not bring significant performance improvements when a repository has few instances of code dupli- cation. In such scenarios, the code retrieval pro- cess struggles to find sufficient relevant information from the repository to facilitate code complction. This issue is further highlighted in the study pre- sented in Appendix C.
retrieval mod- cls. The reason for this limited scope is rooted the complexity of code retrieval, which involves numerous intricate details that are not directly rel- evant to the RepoCoder framework. Considering alternative retrieval models or expanding the ex- ploration to other code retrieval techniques could provide further insights and comparative evalua- tions. Third, we have observed significant advance- ments in code generation models, such as GPT- (OpenAI, 2023), StarCoder (Li et al., 2023), and WizardCoder (Luo et al., 2023). While our ex- periments demonstrate the efficacy of RepoCoder across different language models (GPT-3.5-Turi) and CODEGEN), it would be valuable to investigate how our approach performs with these advanced code generation models. Incorporating them into our experimental setup would provide a broader evaluation of RepoCoder across wider range of language models. Fourth, our experiments primar- ily use the In-File and Oracle methods as baselines. This decision stems from the fact that repository- level code completion using language models is a relatively new task, lacking well-established and reprodueible baselines. To provide further insights, we include comparisons other commercial code completion products. Nonetheless, to systematically benchmark against complex, con- fidential commercial products. We instead conduct study in Appendix E showcasing the repository- level completion ability of RepoCoder and another three major commercial products, where illustrate their qualitative differences. In summary, future work should aim to explore different prompt designs, consider alternative retrieval or generation models, and incorporate additional baselines.
Difficulty in Identifying the Optimal Number of Iterations: While RepoCoder with two itera- tions outperforms the RAG method, determining the optimal number of iterations remains chal- lenge. Subsequent iterations of RepoCoder may exhibit unstable performance compared previous iterations. Appendix D provides demonstration of this issue. To mitigate this, we have explored different approaches to automatically terminate the iteration process when necessary. However, finding an optimal stopping criterion without significantly impacting RepoCoder's performance remains ongoing challenge. Further research is required to develop techniques that can identify the iteration which RepoCoder achieves the best performance.
Time Efficiency for Real-Time Deployment: While RepoCoder demonstrates promising gains code completion accuracy through iterative re- trieval and generation, concerns may arise due to the latency of additional retrieval-generation steps. For real-time deployment scenarios with strict latency requirements, we can further improve RepoCoder through model optimizations such as quantization, distillation, and hardware accelera- tion expedite inference. Techniques like caching frequent code snippets and pre-processing reposi- tories can also boost speed. The model iterations can be dynamically adapted based on latency goals
impractical

---

References
Gautier Izacard, Patrick Lewis, Maria Lomeli, Lu- cas Hosseini, Fabio Petroni, Timo Schick. Jane Dwivedi- Yu, Armand Joulin, Sebastian Riedel, and Edouard Grave. 2022. Few-shot learning with re- trieval augmented language models. arXiv preprint arXiv:2208.0222 033999.
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. 2020. Language models are few-shot learners. Advances in neural information processing systems, 33:1877--101..
Paul Jaccard. 1912. The distribution of the flora in the alpine zone. 1. New phytologist, (1)3:377-50
Bei Chen. Fengji Zhang, Anh Nguyen. Daoguang Zan, Zeqi Lin, Jian-Guang Lou, and Weizhu Chen. 2022. Codet: Code generation with generated tests. arXiv preprint arXiv:2207.1033
Vladimir I Levenshtein et al. 1966. Binary codes capa- ble of correcting deletions, insertions, and reversals. In Soviet physics doklady, volume 10, pages 707-710. Soviet Union
Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde de Oliveira Pinto, Jared Ka- plan, Harri Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, al. 2021. Evaluating large language models trained on code. arXiv preprint arXiv:2107.03374.
Yoav Levine, Itay Dalmedigos, Ori Ram, Yoel Zeldes. Daniel Jannai, Dor Muhlgay, Yoni Osin, Opher Lieber, Barak Lenz, Shai Shalev-Shwartz, et al. 2022. Standing on the shoulders of giant frozen language models. arXiv preprini arXiv:2204. 100119
Aakanksha Chowdhery, Sharan Narang, Jacob Devlin, Maarten Bosma, Gaurav Mishra, Adam Roberts, Paul Barham, Hyung Won Chung, Charles Sutton, Sehastian Gehrmann, al. 2022. Palm: Scaling language modeling with pathways. arXiv preprint arXiv:2204.02311.
Patrick Lewis, Ethan Perez, Aleksandra Piktus, Fabio Petroni, Vladimir Karpukhin, Naman Goyal, Hein- rich Küttler, Mike Lewis. Wen-tau Yih, Tim Rock- täschel, et al. 2020. Retrieval-augmenter generation for knowledge-intensingg nlp tasks. Advances in Neu- ral Information Processing Systems,
Colin B Clement, Shuai Lu, Xiaoyu Liu, Michele Tu- fano, Dawn Drain, Nan Duan, Neel Sundaresan, and Alexey Svyatkovskiy. 2021. Long-range mod- cling of souree code files with ewash: Extended window access by syntax hierarchy, arXiv preprint
Dong Li, Yelong Shen, Ruoming Jin. Yi Mao, Kuan Wang, and Weizhu Chen. 2022. Generation- augmented query expansion for code retrieval. arXiv preprint arXiv2221..1062
Raymond Li, Loubna Ben Allal, Yangtian Zi, Niklas Muennighoff, Denis Kocetkov, Chenghao Mou, Marc Marone, Christopher Akiki, Jia Li, Jenny Chim, et al. 2023. Starcoder: may the source be with you! arXiv preprint 2rXiv:2305.06161.
Yangruibo Ding, Zijian Wang, Wasi Uddin Ahmad, Murali Krishna Ramanathan, Ramesh Nallapati, Parminder Bhatia, Dan Roth, and Bing Xiang. 2022. Cocomic: Code completion by jointly mod- eling in-file and cross-file context. arXiv preprint arXiv:2212. 10007.
Fang Liu, Ge Li, Zhiyi Fu, Shuai Lu, Yiyang Hao. and Zhi Jin. 2022. Learning to recommend method names with global context. In Proceedings of the 44th International Conference on Software Engineer- ing, pages 1294-1306.
Daya Guo, Shuai Lu, Nan Duan, Yanlin Wang, Ming Zhou, and Jian Yin. 2022. Unixcoder: Unified cross- modal pre-training for code representation. In Pro- ceedings of the 60th Annual Meeting of the Associa- tion for Computational Linguistics (Volume 1: Long Papers), pages 7212-7225
Shuai Lu, Nan Duan, Hojac Han, Daya Guo. Scung- won Hwang, and Alexey Svyatkovskiy, 2022. Reace: A eetrieval-augmentt code completion framework. arXiv preprint (xXXx:2203007222
Kelvin Guu, Kenton Lee, Zora Tung. Panupong Pasu- pat, and Mingwei Chang. 2020. Retrieval augmented language model pre-training. In International confer- ence on machine learning, pages 3929-3938. PMLR.
Shuai Lu, Daya Guo, Shuo Ren, Junjie Huang, Alexey Svyatkovskiy. Ambrosio Blanco, Colin Clement, Dawn Drain, Daxin Jiang. Duyu Tang, et al. 2021. Codexglue: A machine learning benchmark dataset for code understanding and generation. arXiv preprint arXiv:
Vincent J Hellendoorn and Premkumar Devanbu. 2017 Are deep neural networks the best choice for model- ing source code? In Proceedings of the 2017 11th Joint Meeting on Foundations of Software Engineer- ing, pages 763-773.
Ziyang Luo, Can Xu, Pu Zhao Qingfeng Sun, Xi- ubo Geng, Wenxiang Hu, Chongyang Tao, Jing Ma. Qingwei Lin, and Daxin Jiang. 2023. Wizardcoder: Empowering code large language models with evol- instruct. arXiv preprint arXiv: 2306v:2306.0833 08568.
Vincent Hellendoorn, Sebastian Proksch, Harald Gall, and Alberto Bacchelli, 2019. When code com- pletion fails: A case study real-world completions. 2019 JEEE/ACM 41st International Conference on Software Engineering (ICSE), pages 960-970. IEEE.
Yuning Mao, Pengcheng He, Xiaodong Liu, Ye- long Shen, Jianfeng Gao, Jiawei Han, and Weizhu Chen. 2020. Generation-augmennt retrieval for

---

open-domain question answering. arXiv preprint arXiv:2009.08553.
Zhaopeng Tu, Zhendong Su, and Premkumar Devanbu. 2014. On the localness of software. In Proceedings of the 22nd ACM SIGSOFT International Symposium on Foundations of Software Engineering, pages 269- 280.
Erik Nijkamp, Bo Pang, Hiroaki Hayashi, Lifu Tu, Huan Wang, Yingbo Zhou, Silvio Savarese, and Caiming Xiong. 2022. Codegen: An open large language model for code with multi-turn program synthesis. arXiv preprint arXiv:2203. 13474.
Liang Wang, Nan Yang, and Furu Wei. 2023. Query2doc: Query expansion with large language models. arXiv preprint 20X1v:2303.0667.
OpenAl. 2023. Gpt-4 technical report.
Daoguang Zan, Bei Chen, Zeqi Lin, Bei Guan, Yongii Wang, and Jian-Guan Lou. 2022. When lan- guage model meets private library. arXiv preprint wwxiv:2200.122..
Ori Ram, Yoav Levine, Itay Dalmedigos, Dor Muhlgay, Amnon Shashua, Kevin Leyton-Brown, and Yoav Shoham. 2023. In-context retrieval- -aaeeeeeddegeeeeed lan- guage models. arXiv preprint arXiv:2302.0000
Yury Zemlyanskiy, Michiel de Jong, Joshua Ainslie, Panupong Pasupat, Peter Shaw, Linlu Qiu, Sumit Sanghai, and Fei Sha. 2022. Generate-anc retrreee:: Use your predictions to improve retrieval for se- mantic parsing. In Proceedings of the 29th Inter- national Conference on Computational Linguistics. pages 4946-4951.
Veselin Raychev, Martin Vechev, and Eran Yahav. 2014, Code completion with statistical language models. In Proceedings of the 35th ACM SIGPLAN conference on programming language design and implementa- tion, pages 429-
Yizhe Zhang, Siqi Sun. Xiang Gao, Yuwei Fang, Chris Brockett, Michel Galley, Jianfeng Gao, and Bill Dolan. 2022, Retgen: joint framework for retrieval and grounded text generation modeling. In Proceed- ings of the AAAI Conference Artificial Intelligence, volume 36, pages 1179-1111 1747.
Md Rizwan Parvez, Wasi Uddin Ahmad, Saikat Chakraborty, Baishakhi Ray, and Kai-Wei Chang. 2021. Retrieval augmented code generation and sum- marization. arXiv e-prints, pages arXiv-2108.
Weijia Shi, Sewon Min, Michihiro Yasunaga, Min- joon Seo, Rich James, Mike Lewis. Luke Zettle- moyer, and Wen-tau Yih. 2023. Replug: Retrieval- augmented black-box language models. arXiv preprint 23X::2301111652.
Shuyan Zhou, Uri Alon, Frank F Xu, Zhengbao Jang. and Graham Neubig. 2022. Doccoder: Generating code by retrieving and reading does. arXiv preprint arXiv:2207.c .05977.
Disha Shrivastava, Hugo Larochelle, and Daniel Tar- low. 2022. Repository-level prompt generation for large language models of code. arXiv preprint arXiv:2206.12839
Weiqin Zou, Jifeng Xuan, Xiaoyuan Xie, Zhenyu Chen, and Baowen Xu, 2019. How does code style inconsis- tency affect pull request integration? an exploratory study on 117 github projects. Empirical Software Engineering, 24:3871-3903.
Alexey Svyatkovskiy, Shao Kun Deng, Shengyu Fu, and Neel Sundaresan. 2020. Intellicode compose: Code generation using transformer. In Proceedings of the 28th ACM Joint Meeting European Software Engineering Conference and Symposium on the Foun- dations of Software Engineering, pages 1433-1443.
A Repository Details
As mentioned in Section 3, we meticulously se- lected repositories for our RepoEval benchmark based on criteria such as source license, cre- ation date, code quantity, and quality. Detailed information about these repositories is provided in Table 7.
Alexey Svyatkovskiy, Sebastian Lee, Anna Hadjitofi, Maik Riechert. Juliana Vicente Franco, and Miltiadis Allamanis. 2021. Fast and memory-efficient neural code completion. In 202/ IEEE/ACM 18th Interna- tional Conference on Mining Software Repositories (MSR), pages 320-340. IEEE.
B Using the Dense Retriever
In our main experiments (as described in Sec- tion 4.2), we utilize a sparse retrieval model for RepoCoder due to its acceptable performance and computational efficiency. However, RepoCoder is versatile framework that can be applied with other code retrieval models as well. To further validate the effectiveness of RepoCoder, we conduct addi- tional experiments using a dense code retriever. Specifically, we employ UniXcoder (Guo et al., 2022), a state-of the-art code embedding model, to
Alexey Svyatkovskiy, Ying Zhao, Shengyu Fu, and Neel Sundaresan. 2019. Pythia: Ai-assisted code com- pletion system. In Proceedings of the 25th ACM SIGKDD international conference knowledge dis- covery data mining, pages 2727-2735.
Romal Thoppilan, Daniel De Freitas, Jamie Hall, Noam Shazeer, Apoory Kulshreshtha, Heng-Tze Cheng, Alicia Jin, Taylor Bos, Leslie Baker, Yu Du et al. 2022. Lamda: Language models for dialog applica- tions. arXiv preprint 2201 082399.

---

RepoCoder Iterations
Metric
Oracle
In-File
Ratio Duplicated Lines
0.5
Duplicate Ratio
2
4
API Dataset
0.8
EM Score Improvement
GPT-3.5- Turboo
0.4
Line Dataset
FM
$7.25
40.56
54.56
56.25
56.31
55.31
ES
75.80
65.06
73.96
74.70
74.31
74.01
0.3
0.4
CODEGEN MOORGEN--MNN-GR
MONO-GB
0.2
EM
47.25
34.56
15.56
46.75
16.56
46.63
ES
70.03
60.67
08.89
69.51
69.33
69.41
0.1
0.2
CODEGEN-MONEE MONO-2B
0.0
0.0
EM
45.75
33.63
44.94
45.13
45.81
45.50
vizier
nerfstudio
FedScope
evaluate
ACE
fortuna
diffusers
ES
68.35
58.99
67.43
68.01
68.35
68.20
MOORGEN--MNO-BR
EM
42.88
29.56
11.50
42.69
42.13
42.31
Figure 4: Correlation between the absolute performance improvements achieved by RepoCoder Iter-2 over the In-File method and the repository duplication ratios.
ES
65.97
55.39
64.71
65.64
65.13
65.53
(a) Line Completion.
RepoCoder Iterations
Metric
Oracle
In-File
1
of the repositories. Intuitively, since RepoCoder utilizes similarity- based retrieval find code ex- emplars, one might expect a positive correlation between its performance and the code duplication ratio. To assess this relationship. we calculate the code duplication ratio of the repositories by deter- mining the ratio of duplicated code lines to the total code lines.
GPT:3.5-Turhs
EM
50.06
34.06
47.56
49.13
49.19
49.63
ES
74.95
63.22
72.66
74.22
72.83
74.41
CODEGEN-MONOO
EM
39.56
26.19
37.00
38.14
39.13
38.44
ES
66.82
56.45
64.21
65.33
65.26
65.53
CODEGEN-MONOL
EM
37.06
25.44
35.00
37.19
36.13
36.63
ES
64.69
56.88
62.22
63.39
63.17
63.65
Figure presents the results, demonstrating the correlation between RepoCoder's performance, as measured by the Exact Match (EM) metric, and the code duplication ratio on the line and API com- pletion datasets using GPT-3.5-Turbo. Notably, the repository "diffusers" exhibits the highest du- plication ratio, which corresponds significant performance improvement for RepoCoder on both datasets. Conversely, "rl" and "vizier" have low duplication ratios, resulting in comparatively lower performance for RepoCoder. However, the corre- lation between RepoCoder's performance and the code duplication ratio is not absolute. For example, "FedScope" and "evaluate" have similar duplica- tion ratios but show different performance gains for RepoCoder.
CODEGEN-MONCN
M3N00350M
EM
33.13
22.19
31.50
32.69
32.75
31.91
ES
61.37
52.24
59.35
60.69
60.49
60.34
(b) API Invocation Completion.
Table 6: Performance comparison on the line and API invocation completion datasets using the dense retriever. Results display the average performance of each method evaluated using Exact Match (EM) and Edit Similarity (ES) scores. Numbers are presented in percentage (%), with the best performance highlighted in bold.
transform code snippets into hidden vectors. We then calculate the similarity between code snippets using cosine similarity. The experimental results on the line and API invocation completion datasets using the dense retriever are presented in Table 6a and Table 6b. Notably, the performance of Re- poCoder using the dense retriever comparable that using the sparse retriever. Furthermore, the findings remain consistent across both retrievers, highlighting the robustness and generalizability of RepoCoder.
D Failed Cases between Iterations
In Section 5, the evaluation results demonstrate that increasing the number of RepoCoder iterations does not necessarily guarantee performance im- provements. To further investigate this issuc, analyze the changes in the number of correct code completions achieved by different methods on the API invocation completion dataset. The predic- tion is considered correct when the EM score is 1. Table 8 presents the results, showing the counts
C Code Duplication in Repositories
We explore the relationship between the perfor- mance of RepoCoder and the code duplication ratio

---

Name
Github Link
License
Created
Stars
F.
L..
Function Body Completion Dataset
1.
imagen
luckdrainw/mmagen--yyoee
MIT License
2022-05-23
6. 160
14
7, 324
2.
tracr
dcepmind/traer
Apache V2.0
2022-12-01
284
56
9, 110
3.
lighunmm
google/lightweight mmm
Apache V2.0
2022-02-10
353
36
9. 676
inspection
amazon-sccicnee https//cccccccccccccceeeeeaaaaie
Apache V2.0
2022-05-05
304
16
2,532
5. omnivore
fucchonkreecarrch/mmiivoe
CC BY-NC 4.0
2022-01-20
459
66
11, 797
6. redframes
maxhumberrreframes
BSDD-2-Clause
2022-08-21
283
49
3, 881
Line and API Invocation Completion Datasets
7.
rl
pytorch/1l
MIT License
2022-02-01
873
165
59, 522
8.
ACE
opendilab/ACE
Apache V2.0
2022-11-23
290
425
66, 497
9.
vizier
google/vizier
Apache V2,0
2022-02-16
688
188
43, 13.393
10.
fortuna
awslabs/fortuma
Apache V2.0
2022-11-17
168
18, 738
11.
evaluate
huggingface/evallute
Apache V2.0
2022-03-30
1. 0,082
180
21, 21,418
12.
diffusers
huggingface/ddffusess
Apache V2.0
2022-05-30
9, 521
305
98, 181
13.
nerfstudio
nerfstudio-projeet project/nerfstuiio
Apache V2.0
2022-05-31
3, 151
157
27, 289
FedScope
Tubbab//FederatedScopp
Apache V2.0
2022-03-24
811
443
48, 545
Table 7: Detailed information of the Github repositories used for RepoFvvl.. ID represents the repository IDs. F.
denotes the total number Python source files, while L. indicates the total number of non-empty Python code lines. Statistics accurate of January 2023.
Coder Iterations
helpful code completion. Furthermore, our investi- gation reveals that many cases in the line and API datasets are actually correct despite being evalu- ated as incorrect by the EM score. This highlights the importance of considering the actual function- ality of the code, rather than solely relying exact matching, and suggests incorporating unit tests assess code correctness.
In-File
3
4
GPT-3.5- Turboo
+545
46
763
-16
787
-83
794
-13
790
+268
+70
+90
+10
CODEGEN Moxo-6B
+424
67
587
-36
622
-12
626
-7
629
+420
+70
+16
+10
CODEGEN- MoNO-2B
+412
567
-32
-12
615
612
+06
BODEGEN-MONO-3500
+352
508
542
540
541
Case Study of Commercial Products
Table 8: The changes in the number correct code completions achieved using different methods on the API invocation completion dataset.
We conduct study showcase the repository- level code completion ability of RepoCoder and another three major commercial code completion tools: Github Copilot 4, Tabnine 5, and Amazon CodeWhisperer' 6 The experiment was conducted using the Visual Studio Code IDE, with each prod- uct providing completions as a plugin. These prod- ucts are based on large language models pre-trained on code data and can perform line-level and block- level completion, similar to our study scenario. We selected simple API invocation example from the RepoEval dataset. The task was to complete the function body for initializing a StableDiffusionKD- iffuusonnPipeline, where the prefix in-file context provided little information. As shown in Figure 5, none of the commercial products generated the cor- rect completion. The implementation details of these commercial products confidential, it is
of correct code completions for each iteration of RepoCoder. We observe that each iteration of Re- poCoder both passes cases that the previous itera- tion failed and fails cases that the previous iteration has passed.
Upon manually examining the failed cases, we have the following observations: Firstly, a majority of failures are caused by misleading retrieved code, which leads to incorrect predictions. For instance, the same API may have different sets of parameters across different files, and the retrieved API usage example can be misleading in such cases. Secondly, the model's predictions are not always suitable for retrieval. This is because the query is constructed using a fixed length of the predicted code, which may include noisy code beyond the initial lines of
"https://github. comm/features/cppllott phttps://www/ tabnine. com "https://aww //aws, amazon, con/ codewhisperer

---

diffuserclutilssddieee torch and transforners and K diffusion objects.py
..utils import Dunrydjject, requires_ backends
class http://wwooo..iuuoooeeeeeeeeeeee
backends
("torch", "transforners", "k_diffusion")
[sel_ targs, *Iwargg)):
raise RuntineError("Mon Instantiate Sselaeafffssiimmfffffssmmmehhees
(a) Incorrect code completion of Githuh Copilot.
diffusersluiilssuumeees torch and transformens and diffusion objects.py fron ..utils import Dunnybjject, pouires_backenee
class fustinot http:///eouuooooooooooooooaaaaaa bbckendds ("torch", "transformers", [_.ffussion"]
def init isit_ targs, *kuargs): pass
grequires, sacpiree__acceedsseeee backends (backends) def fselard(self,, *args, *kuargs):
(b) Incorrect code completion Tabnine.
diffuserslutils dunny torch and transformens and diffusion objects.pp from ..utils Import DumyObject, requkless_aackedd
class http://uwweoootuuooooeeeeennnnnn bbackands
("tarch", "transformers", "k_ciffusion"]
def (self, *args, **kwargs): requires, requires_backends backends "k diffusiion")
Incorrect code completion of Amazon CodeWhisperee
diffusenstuti cidurny torch and transformers and difffutton objects.
the below code fregnent can found in:
difuusersluttlssduuee htwputtlsssuuutttorrr_too. torch_and___h_oeeeeeee baansforeers_nnd httprerrs__anoo_ffffftteeeeeee l_loiffusooooooeetttt
class AltDEffussonnnngiiiiieeennne batarigpppeeiinnaaaaeeeeeeaaaaae
backends ["torch", "transforners"]
def init (self, targs, **kaaggs):
Soquires_backends(eee ("torch", "transforners"))
fron ..utils import Dunnybjject, requires_ backends
class
www.:r/wwwnniiinennnccsssssaaaao backends ["torch", "transforners", "k "k_diffusion")
def init_ (self, targs, **kaargs):
requires_ bself, ("torch", "transformers" "k_diffusion")
(d) Correct code completion of RepoCoder.
Figure 5: Code completion examples of RepoCoder and three major commercial products.
difficult to perform systematic comparison. How- ever, in case, RepoCoder successfully predicted the correct completion by retrieving relevant code snippet from the repository context. This demon- strates the need for state-of-the-ar tools to effec- tively leverage repository-Ieww context.


Reviews:
[{"review_id": "xvQUDJixmC", "reviewer": "Reviewer_LvEZ", "summary": "", "questions": "I do not fully understand the criticism of existing approaches in the first introduction paragraph. Can you please elaborate on these points?\n1. Why does static analysis have limitations in terms of usability? Static analysis tools are quite reliable and efficient from my perspective.\n2. Is there any evidence or reference showing those fine-tuning approaches have generalization problems? If the data can be automatically labeled, e.g. with static analysis, we can potentially leverage the large collection of Github repos to overcome this problem.", "limitations": "", "rating": 3, "confidence": 4, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We value your feedback and appreciate the differing viewpoints on certain issues you highlighted. To address these concerns, we provide more detailed explanations for each weakness and question. We kindly ask you to reconsider your review in light of these responses.\n\n**W1: The in-file baseline is not strong enough. It would be better to include existing cross-file methods mentioned in the first introduction paragraph.**\n\n**A:** As we mentioned in the Limitations section, \"the task of repository-level code completion using language models is relatively new, lacking well-established and reproducible baselines\".\n\nWe did not include existing cross-file methods mentioned in the Introduction section [1-5], because they can not be easily applied to our scenario. Specially, [1-4] are primarily focused on completing token-level code portions at specific positions. These approaches fall outside the purview of our research's scope. As for the Cocomic method [5], the absence of publicly available data and source code from their work hampers our ability to replicate their methodology.\n\nGiven this shortage of suitable baselines, we meticulously designed our experiments to comprehensively evaluate the efficacy of RepoCoder. We introduced the Oracle method for comparative analysis, leveraging its capacity to utilize provided ground truth code completions and retrieve pertinent code from the repository. As a result, it serves as an upper bound performance benchmark for the Retrieval-Augmented Generation (RAG) approach. Our results, as showcased in Table 2 and Table 3, illustrate how RepoCoder's performance, especially with multiple iterations, closely approximates that of the Oracle method. This alignment unequivocally demonstrates RepoCoder's effectiveness.\n\nAdditionally, in Section 6.1, we delve deeper into bolstering RepoCoder's efficacy through the implementation of a comparison method called GT-Code. This approach utilizes static analysis tools to directly pinpoint code snippets involving referencing API invocations. The outcomes of this comparison method also demonstrates the good performance of RepoCoder in a more comprehensive context. Hopefully, these explanations will provide a clearer understanding of the sufficiency of our chosen baselines.\n\n**W2: The hyper-parameter tuning in section 4.1 seems performed on the test sets.**\n\n**A:** RepoCoder operates as a framework without any model training. All the hyper-parameters mentioned in section 4.1 are for the inference stage, such as the length of the code window and the sliding size. Given that these parameters are intricately linked to the programming language and contextual scenarios, their adjustment is vital to ensure optimal real-world performance. We will make it clearer in the revised version of our paper.\n\n**W3:  The proposed method does not fundamentally solve the repo-level code completion problem. As already mentioned in the paper, the method becomes ineffective when exemplars of API calls are missing or misleading, which is quite common in reality. A better approach is probably to resort to API definitions rather than exemplars.**\n\n**A:** We respectfully disagree with your assertion. Firstly, it's important to clarify that RepoCoder's code retriever can locate not only API call exemplars but also relevant API definitions from the repository. This capability is illustrated in Figure 2, where RepoCoder successfully identifies the API definition for \"run_colmap,\" enhancing its code completion abilities.\n\nSecondly, though our paper outlines failed cases of RepoCoder when API calls are misleading, these cases do not undermine RepoCoder's fundamental capability to address the repository-level code completion challenge. Instead, they are presented to emphasize the potential for the future improvement of code retrieval.\n\n**Q1:  Why does static analysis have limitations in terms of usability? Static analysis tools are quite reliable and efficient from my perspective.**\n\n**A:** Approaches relying on static analysis [1-4] have limitations in terms of usability, which primarily stem from their constrained code completion contexts. These studies primarily operate within token-level completion scenarios, triggered at specific positions, such as suggesting method names for Java objects. In these specific contexts, static analysis tools demonstrate their reliability and efficiency by parsing potential completion candidates and subsequently ranking. While these methods are not able to perform code completions that involve varying lengths at arbitrary positions.\n\n**Q2: Is there any evidence or reference showing those fine-tuning approaches have generalization problems? If the data can be automatically labeled, e.g. with static analysis, we can potentially leverage the large collection of Github repos to overcome this problem.**\n\n**A:** The concern of the potential generalization problems with fine-tuning approaches is indeed valid. We can elaborate this concern through the cited studies[2, 5, 6] in the Introduction section of our paper.\n\nTo begin with, Svyatkovskiy et al. [6] trained a large Transformer model on an extensive corpus of open-source code. However, the model cannot predict unseen variable names. Furthermore, Hellendoorn and Devanbu [2] proposed a dual-model methodology, involving a global model trained on public data and a local model fine-tuned to the specific codebase of the target repository. Despite this approach, the need to retrain the local model for every new repository introduction remains a notable drawback, inhibiting seamless generalization.\n\nThirdly, Ding et al. [5] presented an approach that imparts repository-level knowledge to the model and employs static analysis tools for data labeling. Nevertheless, their training strategy assumes the presence of project context during code completions, leading to a decline in performance when confronted with scenarios lacking cross-file context, as highlighted in their outlined limitations.\n\n**References:**\n\n[1] Veselin Raychev, Martin Vechev, and Eran Yahav. 2014. Code completion with statistical language models. In Proceedings of the 35th ACM SIGPLAN conference on programming language design and implementation, pages 419–428.\n\n[2] Vincent J Hellendoorn and Premkumar Devanbu. 2017. Are deep neural networks the best choice for modeling source code? In Proceedings of the 2017 11th Joint Meeting on Foundations of Software Engineering, pages 763–773.\n\n[3] Alexey Svyatkovskiy, Ying Zhao, Shengyu Fu, and Neel Sundaresan. 2019. Pythia: Ai-assisted code completion system. In Proceedings of the 25th ACM SIGKDD international conference on knowledge discovery \\& data mining, pages 2727–2735.\n\n[4] Alexey Svyatkovskiy, Sebastian Lee, Anna Hadjitofi, Maik Riechert, Juliana Vicente Franco, and Miltiadis Allamanis. 2021. Fast and memory-efficient neural code completion. In 2021 IEEE/ACM 18th International Conference on Mining Software Repositories (MSR), pages 329–340. IEEE.\n\n[5] Yangruibo Ding, Zijian Wang, Wasi Uddin Ahmad, Murali Krishna Ramanathan, Ramesh Nallapati, Parminder Bhatia, Dan Roth, and Bing Xiang. 2022. Cocomic: Code completion by jointly modeling in-file and cross-file context. arXiv preprint arXiv:2212.10007.\n\n[6] Alexey Svyatkovskiy, Shao Kun Deng, Shengyu Fu, and Neel Sundaresan. 2020. Intellicode compose: Code generation using transformer. In Proceedings of the 28th ACM Joint Meeting on European Software Engineering Conference and Symposium on the Foundations of Software Engineering, pages 1433–1443.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. The proposed RepoCoder outperforms the in-file baseline by a good margin.\n2. The evaluation dataset will be useful to the community if open-sourced. \n3. The paper is written clearly and easy to follow.", "reasons_to_reject": "1. The in-file baseline is not strong enough. It would be better to include existing cross-file methods mentioned in the first introduction paragraph. \n\n2. The hyper-parameter tuning in section 4.1 seems performed on the test sets.\n\n3. The proposed method does not fundamentally solve the repo-level code completion problem. As already mentioned in the paper, the method becomes ineffective when exemplars of API calls are missing or misleading, which is quite common in reality. A better approach is probably to resort to API definitions rather than exemplars.", "questions_to_authors": "I do not fully understand the criticism of existing approaches in the first introduction paragraph. Can you please elaborate on these points?\n1. Why does static analysis have limitations in terms of usability? Static analysis tools are quite reliable and efficient from my perspective.\n2. Is there any evidence or reference showing those fine-tuning approaches have generalization problems? If the data can be automatically labeled, e.g. with static analysis, we can potentially leverage the large collection of Github repos to overcome this problem.", "paper_topic_and_main_contributions": "This paper studies the task of repository-level code completion, where both in-file and cross-file context must be utilized. The authors proposed RepoCoder, which incorporates cross-file context through iterative retrieval and generation. Specifically, on top of the standard RAG framework, model-generated code completion from the previous iteration is used as part of the query to improve the subsequent retrieval. \n\nThe authors also proposed and constructed a new benchmark dataset for evaluating the task. The dataset consists of line completions, API completions, and function completions with executable unit tests. Results demonstrate more than 10% improvement of RepoCoder over the in-file baseline.", "excitement": 3, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "2ekSbxPmeJ", "reviewer": "Reviewer_ReXb", "summary": "", "questions": "1. Have you evaluated existing code completion commercial applications on your benchmark?\n\n2. Have you evaluated the time efficiency of your method on different settings?\n\n3. This is a concern about potential data leaks: it seems that the training data of GPT-3.5-Turbo is not only up to 2021 as the author stated in the paper, according to [1]. In my opinion, I think the data leak is hard to fix, but it is still essential to claim potential limitations in this paper.\n\n[1] https://openai.com/api-data-privacy (I just noticed that OpenAI has modified the content of this website page and deleted original claims of the training data.)", "limitations": "", "rating": 4, "confidence": 5, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We appreciate your positive feedback and acknowledgment of our efforts. We have taken steps to address the weaknesses and questions you highlighted as follows, aiming to alleviate any concerns you may have.\n\n**W1 & Q1: Missing some baselines: For code completion tasks, I suggest that the authors compare with existing code completion commercial applications, such as Copilot. It can be tested on a smaller subset of RepoEval, and it is essential to compare with these state-of-the-art code completion systems.**\n\n**A:** While we greatly appreciate the suggestion to incorporate comparisons with commercial code completion applications such as Copilot, we wish to underscore the inherent challenges in undertaking such comparisons. Notably, systems like Copilot are sophisticated software solutions that leverage proprietary training data, meticulously tuned lnguage models, and specialized mechanisms to achieve their impressive performance. The inclusion of systematic comparison against such commercial products might introduce biases and unfair comparisons due to the intricate and distinct nature of their underlying complexities.\n\nHowever, in response to your suggestion, we are open to including comparisons between RepoCoder and other commercial code completion products in the revised version of the paper. By doing so, we hope to extract valuable insights from these comparisons while acknowledging the nuanced context in which they are conducted. We firmly believe that RepoCoder presents a promising avenue for enhancing the capabilities of products like Copilot, offering potential benefits to the broader coding community.\n\n**W2 & Q2: Time efficiency: For code completion tasks, it is also important to focus on the time efficiency. I recommend the authors could add corresponding experiments to make it clear.**\n\n**A:** Undoubtedly, time efficiency is an important demand for real-world applications. While quantifying the time efficiency of RepoCoder presents a challenging task. As outlined in our response to Reviewer rrM2, the time efficiency of RepoCoder is influenced by many engineering strategies such as model quantization for expedited code generation and caching mechanisms for accelerated code retrieval. The implementation of these techniques extends beyond the current scope of our paper. However, we are committed to adding discussions on time efficiency to the Limitations section of our paper.\n\n**W3: Missing some related work: Recently there are some papers on similar topics, such as repo-level benchmark [1], API invocation [2]. The authors could consider discussing them in related work to better tease out research directions.**\n\n**A:** Thank you for bringing this to our attention. We sincerely appreciate your insight regarding the recent papers [1] and [2]. We will definitely incorporate these concurrent works into our related work section to provide a more comprehensive overview of the research landscape. \n\n**Q3: This is a concern about potential data leaks: it seems that the training data of GPT-3.5-Turbo is not only up to 2021 as the author stated in the paper, according to [3]. In my opinion, I think the data leak is hard to fix, but it is still essential to claim potential limitations in this paper.**\n\nOur reference to the training data of GPT-3.5-Turbo being up to 2021 is based on the information provided in the API documentation by OpenAI [4]. We admit that there remains a possibility of data leakage since  the training details of the model are unknown. And we are glad to claim the potential data leakage in the Limitations section.  Importantly, we want to emphasize that despite this concern, the conclusions drawn in our paper retain their validity, as our comparison is fair, and we also incorporates the open-sourced CodeGen model.\n\n**References:**\n\n[1] Liu, Tianyang et al. “RepoBench: Benchmarking Repository-Level Code Auto-Completion Systems.” (2023)\n\n[2] Zhang, Kechi et al. “ToolCoder: Teach Code Generation Models to use API search tools.”(2023)\n\n[3] https://openai.com/api-data-privacy\n\n[4] https://platform.openai.com/docs/models/gpt-3-5", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. Interesting viewpoint: This paper focuses on the repository-level code completition, which is a research question that fits the actual scene and is worth discussing. The proposed method RepoCoder uses the iterative retrieval-generation framework and achieves good performance.\n\n2. New dataset. I further support the eagerness of the authors to release their dataset to the community. And I believe the proposed RepoEval benchmark will provide a platform for further study.\n\n3. Experiments: The paper provides exhaustive experiments demonstrating the effectiveness of the method.\n\n4. Readability. Moreover, the paper is easy to follow, in my opinion, for a broad NLP community, and it provides visualization of data and the main algorithm.", "reasons_to_reject": "1. Missing some baselines: For code completion tasks, I suggest that the authors compare with existing code completion commercial applications, such as Copilot. It can be tested on a smaller subset of RepoEval, and it is essential to compare with these state-of-the-art code completion systems.\n\n2. Time efficiency: For code completion tasks, it is also important to focus on the time efficiency. I recommend the authors could add corresponding experiments to make it clear.\n\n3. Missing some related work: Recently there are some papers on similar topics, such as repo-level benchmark [1], API invocation [2]. The authors could consider discussing them in related work to better tease out research directions.\n\n[1] Liu, Tianyang et al. “RepoBench: Benchmarking Repository-Level Code Auto-Completion Systems.” (2023)\n\n[2] Zhang, Kechi et al. “ToolCoder: Teach Code Generation Models to use API search tools.”(2023)", "questions_to_authors": "1. Have you evaluated existing code completion commercial applications on your benchmark?\n\n2. Have you evaluated the time efficiency of your method on different settings?\n\n3. This is a concern about potential data leaks: it seems that the training data of GPT-3.5-Turbo is not only up to 2021 as the author stated in the paper, according to [1]. In my opinion, I think the data leak is hard to fix, but it is still essential to claim potential limitations in this paper.\n\n[1] https://openai.com/api-data-privacy (I just noticed that OpenAI has modified the content of this website page and deleted original claims of the training data.)", "paper_topic_and_main_contributions": "This paper introduces a practical code task: repository-level code completion. Within this context, the authors present a novel benchmark named RepoEval, including scenarios for line, API invocation, and function body completion. This benchmark contributes significantly to the field by providing a platform for in-depth exploration. Additionally, the paper introduces RepoCoder, which employs an iterative retrieval-generation framework to address the complexities of repository-level code completion. The detailed experiments  showcases its effectiveness.", "excitement": 4, "reproducibility": 5, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {"Missing_References": "Recently there are some papers on similar topics, such as repo-level benchmark [1], API invocation [2]. The authors could consider discussing them in related work to better tease out research directions.\n\n[1] Liu, Tianyang et al. “RepoBench: Benchmarking Repository-Level Code Auto-Completion Systems.” (2023)\n\n[2] Zhang, Kechi et al. “ToolCoder: Teach Code Generation Models to use API search tools.”(2023)"}}, {"review_id": "bagdpxIuzo", "reviewer": "Reviewer_rrM2", "summary": "", "questions": "1. In the API invocation dataset, is all API invocation local to the repository? Or are there global APIs (e.g., torch, numpy) too?\n2. Does the length of the sliding window mean the length of the retrieved code chunks (snippets)?\n3. What is the maximum number of tokens budgeted for retrieved code snippets in the prompt? It is mentioned that the top 10 snippets are retrieved, but they could be very long.\n4. The iterative retrieval and generation approach is not feasible to use practically due to its latency requirements. I want to know the authors' opinions on how we can leverage RepoCoder-like models in real time.", "limitations": "", "rating": 4, "confidence": 5, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank you for your kind feedback and for recognizing our work. To address the questions you raised, we have provided detailed responses as follows. \n\n**Q1: In the API invocation dataset, is all API invocation local to the repository? Or are there global APIs (e.g., torch, numpy) too?**\n\n**A:** Yes, all API invocations in our dataset are local to the repository. And there are no other global APIs in the dataset. We ensure the invoked APIs are locally defined within the repository. \n\n**Q2: Does the length of the sliding window mean the length of the retrieved code chunks (snippets)?**\n\n**A:** Yes, the length of the sliding window refers to the size of the retrieved code snippets.\n\n**Q3: What is the maximum number of tokens budgeted for retrieved code snippets in the prompt? It is mentioned that the top 10 snippets are retrieved, but they could be very long.**\n\n**A:** First, the budget for the retrieved code snippets takes half of the prompt's length (mentioned in Section 4.1 Implementation Details). Then, if the top 10 retrieved snippets surpass half of the prompt's length, we only keep the most relevant snippets that fit within the designated length limit (mentioned in Section 2.3 Code Generation).\n\n**Q4: The iterative retrieval and generation approach is not feasible to use practically due to its latency requirements. I want to know the authors' opinions on how we can leverage RepoCoder-like models in real time.**\n\n**A:** RepoCoder indeed introduces concerns about latency. However, RepoCoder's architecture is inherently flexible, allowing for optimizations that make it more suitable for real-time use cases. Here are several avenues:\n\n- Model Quantization or Distillation: Techniques like model quantization or distillation can be employed to reduce the size of the language model while retaining a satisfactory level of performance. Our experiments have indicated that RepoCoder can notably enhance the performance of smaller models, such as CodeGen-350M.\n\n- Hardware Acceleration: Harnessing the power of hardware accelerators, such as GPUs or TPUs, can substantially expedite the inference process of language models, leading to notable reductions in latency.\n\n- Caching and Preprocessing: By pre-caching frequently used code snippets or preprocessing repositories that are frequently accessed, speed can be further improved.\n\n- Adaptive Iterations: Adapting the iteration steps dynamically based on contextual needs and desired latency can yield a balanced compromise between accuracy and speed.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "- A simple yet effective approach for repository-level code completion.\n- Strong analysis to explain the source of improvements in the code completion tasks.\n- Introduced a good dataset in Python for repository-level code completion.", "reasons_to_reject": "I don't have a reason to reject the work.", "questions_to_authors": "1. In the API invocation dataset, is all API invocation local to the repository? Or are there global APIs (e.g., torch, numpy) too?\n2. Does the length of the sliding window mean the length of the retrieved code chunks (snippets)?\n3. What is the maximum number of tokens budgeted for retrieved code snippets in the prompt? It is mentioned that the top 10 snippets are retrieved, but they could be very long.\n4. The iterative retrieval and generation approach is not feasible to use practically due to its latency requirements. I want to know the authors' opinions on how we can leverage RepoCoder-like models in real time.", "paper_topic_and_main_contributions": "The paper studies repository-level code generation. The paper presents RepoCoder, an iterative retrieval and generation framework, and introduces a dataset called RepoEval that includes 1600 examples of line and API completion tasks and 373 examples of function completion tasks. While the proposed method is simple and straightforward, its effectiveness cannot be overlooked. The strongest part of the paper is the analysis section which studied different factors to explain the source of improvements. ", "excitement": 4, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Missing_References": "I didn't check carefully.", "Typos_Grammar_Style_and_Presentation_Improvements": "I didn't check for grammar issues or typos."}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper addresses the task of repository-level code completion. They introduce a new dataset – RepoEval, and a new framework RepoCoder that incorporates a similarity-based retriever and a pre-trained code language model. 

The paper is clear and well-written. The experiments are thorough. The proposed RepoCoder approach outperforms the in-file baseline. The work is solid and should be of interest to researchers working on code completion. 

The question and concern regarding tuning the hyper-parameters on the test data should be addressed in the paper. It should be clarified whether the test data is used for finetuning, and even if it’s only done at the inference stage, it’s still problematic. Adding  a comparison with commercial applications as baselines would strengthen the paper.
Decision: Accept-Main